A method and system for automatically sharing computing power for massive data processing

Abstract

The present invention relates to the field of priority scheduling technology, and specifically to a method and system for automated sharing of computing power for massive data processing, including: obtaining computing power parameters of multiple dimensions and importance scores for each computing demand and each computing power resource, obtaining the computing complexity of each computing demand in combination with the number of dimensions of each computing demand, and obtaining the carrying characteristic value of each computing power resource according to the allocated situation of each computing power resource; obtaining the allocation order characteristic value of each computing demand to be allocated according to the difference between the carrying characteristic value of each computing power resource and the computing power parameter of each computing power resource and the computing power parameter of each computing demand to be allocated, thereby obtaining the scheduling priority of each computing demand to be allocated, and performing priority scheduling on each computing demand to be allocated. The present invention realizes automated sharing of computing power for massive data by optimizing the scheduling configuration of computing power resources for different computing demands in a computing power sharing network.

Description

Technical Field

[0001] The present invention relates to the technical field of priority scheduling, and in particular to a method and system for automatically sharing computing power for massive data processing. Background Art

[0002] With the rapid development of the information industry, the amount of data that needs to be analyzed is increasing, and the demand for computing power is also expanding. Traditional computing resources are often unable to meet the needs of large-scale computing. Computing power sharing can effectively improve computing efficiency by sharing advanced computing resources, centralizing scattered computing resources, and realizing the rational allocation of resources to form a large computing cluster.

[0003] The core of computing power sharing is to optimize the configuration of computing resources. The scheduling method of the existing computing power sharing platform mainly schedules and processes computing needs through a task-level time-sharing scheduler, and queues computing needs according to the principle of "first come, first served" and task priority. However, when faced with the computing needs of massive data, due to the huge computing demand, for the computing resources in the computing power sharing network, the existing scheduling method may cause some computing resources to experience transmission blockage when obtaining computing needs, resulting in untimely transmission of computing result information, and reducing the operating efficiency of the computing power sharing network. Summary of the invention

[0004] The present invention provides a method and system for automatically sharing computing power for processing massive data, so as to solve the existing problems.

[0005] The present invention provides a method and system for automatically sharing computing power for processing massive data using the following technical solutions:

[0006] An embodiment of the present invention provides a method for automatic sharing of computing power for massive data processing, the method comprising the following steps:

[0007] Obtain computing power parameters in multiple dimensions for each computing requirement and each computing power resource;

[0008] Obtain importance scores of computing power parameters of different dimensions; obtain the computational complexity of each computational requirement based on the importance scores of computing power parameters of different dimensions and the number of dimensions of each computational requirement;

[0009] The carrying characteristic value of each computing resource is obtained according to the allocated status of each computing resource; the allocation order characteristic value of each computing demand to be allocated is obtained according to the difference between the carrying characteristic value of each computing resource and the computing power parameter of each computing resource and the computing power parameter of each computing demand to be allocated;

[0010] Obtaining the scheduling priority of each computing demand to be allocated according to the allocation order characteristic value and the submission time deviation of each computing demand to be allocated;

[0011] Priority scheduling is performed on each computing demand to be allocated according to the scheduling priority of each computing demand to be allocated.

[0012] Furthermore, the specific steps of obtaining computing power parameters of multiple dimensions of each computing requirement and each computing power resource are as follows:

[0013] Obtain the initial parameters and submission time of each computing resource and each computing demand in the current computing power sharing network; use each parameter in the initial parameters of each computing resource and each computing demand as a dimension to form multi-dimensional data, and use the range normalization algorithm to standardize the multi-dimensional data of all computing resources and computing demands to obtain computing power parameters of multiple dimensions for each computing demand and each computing resource.

[0014] Furthermore, the obtaining of importance scores of computing power parameters of different dimensions includes the following specific steps:

[0015] All computing power parameters are input into the trained neural network model to obtain the importance score value of each dimension in the computing requirements.

[0016] Furthermore, the computational complexity of each computational requirement is obtained according to the importance scores of the computational power parameters of different dimensions and the number of dimensions, and the specific steps include the following:

[0017]

[0018] In the formula, represents the computational complexity of the kth computation requirement; The number of dimensions representing the computing power parameter required for the kth operation; Indicates the importance score of the i-th dimension in the computing power parameter of the k-th computing requirement; The i-th dimension data in the computing power parameter representing the k-th computing requirement; The number of all data required for the kth operation; represents the hyperbolic tangent function; represents the linear normalization function.

[0019] Furthermore, the method of obtaining the carrying characteristic value of each computing resource according to the allocated status of each computing resource includes the following specific steps:

[0020]

[0021] In the formula, Indicates the carrying characteristic value of the jth computing resource; Indicates the number of computing processes of the jth computing resource; represents the computational complexity of the g-th computational requirement in the computational process of the j-th computing resource; Represents the Sigmoid function.

[0022] Furthermore, the specific method for obtaining the number of computing processes is as follows:

[0023] Count the number of computing requirements to be calculated and being calculated for each computing resource, and record it as the number of computing processes for each computing resource.

[0024] Furthermore, the step of obtaining the allocation order characteristic value of each computing requirement to be allocated includes the following specific steps:

[0025]

[0026] In the formula, Indicates the carrying characteristic value of the jth computing resource; represents the computational complexity of the dth computational requirement to be allocated; represents the allocation order characteristic value of the dth computing demand to be allocated; n represents the number of computing resources; f(x) is a non-negative function; Represents the data mean of all dimensions in the computing power parameters of the jth computing power resource; Represents the data mean of all dimensions in the computing power parameter of the dth computing requirement to be allocated; represents the linear normalization function.

[0027] Furthermore, the step of obtaining the scheduling priority of each computing demand to be allocated according to the allocation order characteristic value and the submission time of each computing demand to be allocated includes the following specific steps:

[0028]

[0029] represents the scheduling priority of the dth computing requirement to be allocated, represents the allocation order characteristics of the dth operation requirement to be allocated, Indicates the difference between the current time and the submission time of the dth computing requirement to be allocated. represents the linear normalization function.

[0030] Furthermore, the processing of each computing demand to be allocated according to the scheduling priority of each computing demand to be allocated includes the following specific steps:

[0031] Create several ready queues according to the multi-level feedback queue scheduling algorithm, set the number of queues and queue lengths; allocate the computing requirements to be allocated to each ready queue from high to low according to the scheduling priority of each computing requirement, use the time-sharing technology in the multi-level feedback queue scheduling algorithm to allocate time slices from short to long to each ready queue, and perform priority scheduling for different computing requirements according to the feedback queue scheduling algorithm.

[0032] The present invention also proposes an automated computing power sharing system for massive data processing, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program stored in the memory to implement the steps of the aforementioned automated computing power sharing method for massive data processing.

[0033] The beneficial effects of the technical solution of the present invention are: obtaining computing power parameters and importance score values ​​of multiple dimensions of each computing demand and each computing power resource, and obtaining the computing complexity of each computing demand with the number of dimensions of each computing demand, combining the impact of the amount of data on the computing demand; obtaining the carrying characteristic value of each computing power resource according to the allocated situation of each computing power resource; obtaining the allocation order characteristic value of each computing demand to be allocated according to the difference between the carrying characteristic value of each computing power resource and the computing power parameters of each computing power resource and the computing power parameters of each computing demand to be allocated, thereby avoiding the waste of computing power resources; thereby obtaining the scheduling priority of each computing demand to be allocated, performing priority scheduling on each computing demand to be allocated, and realizing automatic sharing of computing power for massive data processing. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0035] Figure 1 This is a flowchart of the steps of a method for automated sharing of computing power for massive data processing according to the present invention. DETAILED DESCRIPTION

[0036] In order to further explain the technical means and effects adopted by the present invention to achieve the predetermined invention purpose, the following is a detailed description of the specific implementation method, structure, features and effects of a computing power automation sharing method and system for massive data processing proposed by the present invention in combination with the accompanying drawings and preferred embodiments. In the following description, different "one embodiment" or "another embodiment" does not necessarily refer to the same embodiment. In addition, specific features, structures or characteristics in one or more embodiments may be combined in any suitable form.

[0037] Unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs.

[0038] The following is a detailed description of a specific solution of a method and system for automatically sharing computing power for massive data processing provided by the present invention in conjunction with the accompanying drawings.

[0039] See also Figure 1 , which shows a flowchart of a method for automatic sharing of computing power for massive data processing provided by an embodiment of the present invention, the method comprising the following steps:

[0040] Step S001: Obtain the computing power parameters of each computing power resource and the computing power parameters of each computing requirement.

[0041] The purpose of this embodiment is to realize the automatic sharing of computing power for massive data by optimizing the scheduling and configuration of computing power resources for different computing requirements in the computing power sharing network. Therefore, the initial parameters and submission time of each computing power resource and each computing requirement in the current computing power sharing network are obtained. The initial parameters described in this embodiment include the computing power of the CPU and GPU, as well as parameters such as storage and network bandwidth. Other embodiments may select other parameters as initial parameters, which are not specifically limited in this embodiment. Each parameter in the initial parameters of each computing power resource and each computing requirement is used as a dimension to form multidimensional data. The multidimensional data of all computing power resources and computing requirements are standardized using the extreme difference normalization algorithm to obtain computing power parameters of multiple dimensions for each computing requirement and each computing power resource.

[0042] Furthermore, the number of computing requirements to be calculated and being calculated for each computing resource is counted, which is recorded as the number of computing processes for each computing resource.

[0043] Step S002: Obtain importance scores of different computing power parameter dimensions based on the standardized multi-dimensional computing power parameters; obtain the computational complexity of each computational requirement based on the importance scores of different computing power parameter dimensions and the number of dimensions.

[0044] It should be noted that when realizing the automated sharing of computing power for massive data processing, the core lies in the configuration and scheduling of computing resources and computing needs. In order to maximize the processing efficiency and reduce the blocking of computing resources caused by excessive data volume of computing needs, in the process of scheduling and optimizing the operation order, it is necessary to analyze the priority of computing needs in combination with the computational complexity of different computing needs.

[0045] In the multidimensional data about computing power parameters, facing different computing requirements, the computing power information represented by different dimensional data in the multidimensional data has different importance to the computing requirements. Therefore, to analyze the computational complexity of different computing requirements, it is necessary to first evaluate the importance of different dimensional computing power parameters in each computing requirement to the current computing requirement, and then analyze the computational complexity of the computing requirements based on the difference between the different dimensional computing power parameters of the computing power resources and the different dimensional computing power parameters of the computing requirements, as well as the amount of data required for computing power. For the computing requirements of massive data, calculating its computational complexity only through its computing power requirements ignores the impact of the amount of data on the computing process, so it is also necessary to analyze the computational complexity of the computing requirements in combination with the amount of data.

[0046] Specifically, all computing power parameters are input into the trained neural network model to obtain the importance score value of each dimension in the computing requirements; the neural network model used in this embodiment is a DNN neural network, and the method for obtaining the data set for training the neural network is: collecting a number of historical computing power parameters, and scoring each historical computing power parameter by manual scoring, and the scoring values ​​are preset as: 0.1, 0.2, 0.3, ... 1, a total of 10 score types, corresponding to 10 importance levels; the historical computing power parameters and their corresponding score values ​​constitute a data set; the neural network is trained using the data set, and the loss function used in the training process is a cross entropy function; the specific training process is a well-known content of the neural network, and this embodiment will not repeat the specific training process.

[0047] Then the calculation method for the computational complexity of the kth computation requirement in the computing power sharing network is:

[0048]

[0049] In the formula, represents the computational complexity of the kth computation requirement; The number of dimensions representing the computing power parameter required for the kth operation; Indicates the importance score of the i-th dimension in the computing power parameter of the k-th computing requirement; The i-th dimension data in the computing power parameter representing the k-th computing requirement; The number of all data required for the kth operation; represents the hyperbolic tangent function; represents the linear normalization function.

[0050] By standardizing the importance score q of different dimensional data of computing power parameters and using it as the weight, the weighted mean of the computing power demand corresponding to the current computing demand is calculated. The larger the value, the higher the requirement of computing resources for the current computing demand, and the greater the corresponding computing complexity. It represents the product of the weighted mean of computing power requirements and the amount of data required for the current operation. The larger the value, the greater the impact of the data volume on the calculation process and the greater the calculation complexity.

[0051] Step S003: obtain the carrying characteristic value of each computing power resource according to the allocated status of each computing power resource; obtain the allocation order characteristic value of each computing demand to be allocated according to the carrying characteristic value and computing power parameters of each computing power resource, and the computing power parameters of each computing demand to be allocated.

[0052] It should be noted that when optimizing the scheduling process of the computing power sharing network and realizing automatic computing power sharing, after obtaining the computational complexity of each computing demand, the computing power resource information and transportation status in the current computing power sharing network should be combined to determine the applicability of the computing power resources to the computing demand, where the computing power parameters of the computing power resources suitable for the computing demand should not be lower than the computing power parameters proposed in the computing demand. In order to avoid resource waste, computing power resources should be allocated to computing demands that match their computing power parameters as much as possible. Therefore, it is necessary to combine the computing power resources and computing demand information in the computing power sharing network to comprehensively analyze the allocation order characteristics of different computing demands.

[0053] Specifically, the calculation method of the carrying characteristic value of the jth computing power resource is:

[0054]

[0055] The calculation method of the allocation order characteristic value of the dth operation requirement to be allocated is:

[0056]

[0057] In the formula, Indicates the carrying characteristic value of the jth computing resource; Indicates the number of computing processes of the jth computing resource; represents the computational complexity of the dth computational requirement to be allocated; represents the computational complexity of the g-th computational requirement in the computational process of the j-th computing resource; represents the allocation order characteristic value of the dth computing demand to be allocated; n represents the number of computing resources; f(x) is a non-negative function, where x is the input of the non-negative function. When the input is a non-negative number, the output is ,in represents an exponential function with a natural constant as the base, otherwise the output is 0; it should be noted that the The model only shows negative correlation and the output of the constraint model is in In the range, As the input of this model, it can be replaced by other models with the same purpose in specific implementation. The model is used as an example for description without any specific limitation. Represents the data mean of all dimensions in the computing power parameters of the jth computing power resource; Represents the data mean of all dimensions in the computing power parameter of the dth computing requirement to be allocated; represents the linear normalization function; Represents the Sigmoid function.

[0058] It represents the mean value of the computational complexity of the computational requirements allocated in the jth computing resource. The larger the mean value, the greater the carrying burden of the jth computing resource and the greater the carrying characteristic value of the jth computing resource. For a computing requirement, the more suitable computing resources there are, the more obvious the allocation order characteristics should be in order to improve the operating efficiency of the computing power sharing network. The non-negative function value representing the difference between the computing power parameter dimension data of the j-th computing power resource and the computing power parameter dimension data of the computing demand to be allocated, The smaller the value, the lower the allocation order characteristic value contributed by the jth computing resource to the dth computing demand to be allocated. The purpose is to reduce the waste of computing resources. When the jth computing resource is exactly the same as the computing power requirement of the dth computing demand to be allocated, The output is the maximum value 1; then This means that the more idle the j-th computing resource is at present, the more obvious the allocation order characteristics that the j-th computing resource can provide are.

[0059] Step S004: Obtain the scheduling priority of each computing demand to be allocated according to the allocation order characteristic value and submission time of each computing demand to be allocated.

[0060] It should be noted that the allocation order characteristics of computing requirements are scheduling characteristics obtained by combining the current computing resources and computing requirements. Finally, it is necessary to analyze the scheduling priority of computing requirements in combination with the submission time of the computing requirements to meet the "first come, first served" principle as much as possible.

[0061] Specifically, the scheduling priority of the dth computing requirement to be allocated is calculated as follows:

[0062]

[0063] represents the scheduling priority of the dth computing requirement to be allocated, represents the allocation order characteristics of the dth operation requirement to be allocated, Indicates the difference between the current time and the submission time of the dth computing requirement to be allocated. represents the linear normalization function.

[0064] The larger the value, the more obvious the allocation order characteristics of the computing power demand are and the earlier the demand is submitted, the higher the scheduling priority of the current computing power demand should be.

[0065] Similarly, obtain the scheduling priority of all computing requirements.

[0066] Step S005: Process each computing requirement to be allocated according to the scheduling priority of each computing requirement to be allocated.

[0067] It should be noted that this embodiment uses a multi-level feedback queue scheduling algorithm (MLFQ) to schedule and adjust the computing requirements in the computing power sharing network in combination with the scheduling priority of the computing requirements. The specific process is: create a number of ready queues. Since the number is related to the number of computing resources in the current computing power sharing network, the number of queues created in this embodiment is 50, and the length of each queue is 10. This is used as an example for description. Other embodiments can set other numbers of queues and queue lengths; the computing requirements to be allocated are allocated to each ready queue in order from high to low according to the scheduling priority of each computing requirement, and each ready queue is allocated a time slice from short to long using time-sharing technology. Different computing requirements are processed according to the multi-level feedback queue scheduling algorithm. After analyzing different computing requirements, different priorities are calculated to realize the automatic sharing of computing power in the face of massive data processing; it should be noted that the multi-level feedback queue scheduling algorithm and time-sharing technology used in this embodiment are well-known technologies, and this embodiment will not be repeated.

[0068] The present invention also provides an automated computing power sharing system for massive data processing, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program stored in the memory to implement the steps of the aforementioned automated computing power sharing method for massive data processing.

[0069] So far, the present invention is completed.

[0070] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for automatically sharing computing power for massive data processing, characterized in that: The method comprises the following steps: Obtain computing power parameters in multiple dimensions for each computing requirement and each computing power resource; The specific steps of obtaining the computing power parameters of multiple dimensions of each computing requirement and each computing power resource are as follows: Obtain the initial parameters and submission time of each computing resource and each computing demand in the current computing power sharing network; use each parameter in the initial parameters of each computing resource and each computing demand as a dimension to form multi-dimensional data, and use the range normalization algorithm to standardize the multi-dimensional data of all computing resources and computing demands to obtain computing power parameters of multiple dimensions for each computing demand and each computing resource; the initial parameters include the computing power of the CPU and GPU as well as storage and network bandwidth parameters; Obtain importance scores of computing power parameters of different dimensions; obtain the computational complexity of each computational requirement based on the importance scores of computing power parameters of different dimensions and the number of dimensions of each computational requirement; The specific steps of obtaining the computational complexity of each computational requirement according to the importance scores of the computational power parameters of different dimensions and the number of dimensions are as follows: In the formula, represents the computational complexity of the kth computation requirement; The number of dimensions representing the computing power parameter required for the kth operation; Indicates the importance score of the i-th dimension in the computing power parameter of the k-th computing requirement; The i-th dimension data in the computing power parameter representing the k-th computing requirement; The number of all data required for the kth operation; represents the hyperbolic tangent function; represents the linear normalization function; The carrying characteristic value of each computing resource is obtained according to the allocated status of each computing resource; the allocation order characteristic value of each computing demand to be allocated is obtained according to the difference between the carrying characteristic value of each computing resource and the computing power parameter of each computing resource and the computing power parameter of each computing demand to be allocated; The specific steps of obtaining the carrying characteristic value of each computing resource according to the allocation status of each computing resource are as follows: In the formula, Indicates the carrying characteristic value of the jth computing resource; Indicates the number of computing processes of the jth computing resource; represents the computational complexity of the g-th computational requirement in the computational process of the j-th computing resource; Represents the Sigmoid function; The specific method for obtaining the number of the computing processes is as follows: Count the number of computing requirements to be calculated and being calculated for each computing resource, and record it as the number of computing processes for each computing resource; The specific steps of obtaining the allocation order characteristic value of each computing requirement to be allocated include the following: In the formula, Indicates the carrying characteristic value of the jth computing resource; represents the computational complexity of the dth computational requirement to be allocated; represents the allocation order characteristic value of the dth computing demand to be allocated; n represents the number of computing resources; f(x) is a non-negative function; Represents the data mean of all dimensions in the computing power parameters of the jth computing power resource; Represents the data mean of all dimensions in the computing power parameter of the dth computing requirement to be allocated; represents the linear normalization function; Obtaining the scheduling priority of each computing demand to be allocated according to the allocation order characteristic value and the submission time deviation of each computing demand to be allocated; The step of obtaining the scheduling priority of each computing demand to be allocated according to the allocation order characteristic value and submission time of each computing demand to be allocated includes the following specific steps: represents the scheduling priority of the dth computing requirement to be allocated, represents the allocation order characteristics of the dth operation requirement to be allocated, Indicates the difference between the current time and the submission time of the dth computing requirement to be allocated. represents the linear normalization function; Priority scheduling is performed on each computing demand to be allocated according to the scheduling priority of each computing demand to be allocated.

2. The method for automatic sharing of computing power for massive data processing according to claim 1, characterized in that: The specific steps of obtaining the importance scores of the computing power parameters of different dimensions are as follows: All computing power parameters are input into the trained neural network model to obtain the importance score value of each dimension in the computing requirements.

3. The method for automatic sharing of computing power for massive data processing according to claim 1, characterized in that: The specific steps of processing each computing demand to be allocated according to the scheduling priority of each computing demand to be allocated are as follows: Create several ready queues according to the multi-level feedback queue scheduling algorithm, set the number of queues and queue lengths; allocate the computing requirements to be allocated to each ready queue from high to low according to the scheduling priority of each computing requirement, use the time-sharing technology in the multi-level feedback queue scheduling algorithm to allocate time slices from short to long to each ready queue, and perform priority scheduling for different computing requirements according to the feedback queue scheduling algorithm.

4. A computing power automatic sharing system for massive data processing, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the computer program is executed by a processor, the steps of an automated computing power sharing method for massive data processing as described in any one of claims 1 to 3 are implemented.

Images