Method, device and equipment for allocating computational nodes for task and storage medium

Abstract

The embodiment of the invention provides a method, device and equipment for allocating computing nodes for tasks and a storage medium, and belongs to the technical field of high-performance computing. By adopting the method to allocate the computational nodes for the tasks, hardware resources can be considered, meanwhile, the characteristic information of the operating systems in the computational nodes and the actual execution conditions of the computational nodes during task execution are also considered, the considered factors influencing the task execution efficiency are more comprehensive, and the task execution efficiency can be better improved.

Description

Technical Field

[0001] The present disclosure relates to the field of high performance computing technology, and in particular to a method, apparatus, device and storage medium for allocating computing nodes to tasks. Background Art

[0002] In high performance computing (HPC) systems, computing nodes are generally assigned to user tasks based on the hardware resource information of each computing node (such as processor resources, memory resources, network bandwidth resources, etc.) and the hardware resource demand information submitted by the user. However, in HPC systems, some factors other than hardware factors can also affect the processing performance of computing nodes for tasks. Therefore, assigning computing nodes to tasks only considering hardware resources may result in low execution efficiency of tasks. Summary of the invention

[0003] In order to solve the related technical problems, the present disclosure provides a method, apparatus, device and storage medium for allocating computing nodes to tasks. The technical solution is as follows:

[0004] In a first aspect, a method for assigning a computing node to a task is provided, which is applied to an HPC system, wherein the HPC system runs on a cluster, and the cluster includes a management node and a plurality of computing nodes. The management node obtains first execution status information, wherein the first execution status information is used to indicate the execution status of a target task when it is executed in a first computing node. The management node determines a first matching degree between the first computing node and the target task based on characteristic information of a first operating system in the first computing node, hardware resource information of the first computing node, and the first execution status information, wherein the characteristic information is used to indicate a characteristic that distinguishes the first operating system from other operating systems. The management node assigns a computing node to the target task based on the first matching degree.

[0005] The hardware resource information of the first computing node may include information such as the number of processors, memory capacity, and network bandwidth. The higher the first matching degree between the first computing node and the target task, the higher the execution efficiency of the target task in the first computing node. After the management node obtains the first matching degree between the first computing node and the target task, it can compare it with a preset matching degree threshold to determine whether the first matching degree reaches the preset matching degree threshold. If the preset matching degree threshold is reached, the target task can be allocated to the first computing node.

[0006] By using this method to allocate computing nodes to tasks, it is possible to consider not only hardware resources but also the characteristic information of the operating system in the computing nodes and the actual execution status of the computing nodes when executing tasks. This takes into account more comprehensive factors affecting the efficiency of task execution and can better improve the efficiency of task execution.

[0007] In a possible implementation manner, the first execution state information includes a parameter value of a first state parameter, where the first state parameter is a state parameter corresponding to a task type of the target task.

[0008] Task types can include computationally intensive, communication intensive, input-output intensive, high-parallelism, etc.

[0009] In this way, selecting different state parameters for different types of tasks can more accurately and effectively reflect the execution status of the target task when it is executed on the computing node, and thus more accurately match the computing node for the target task.

[0010] In a possible implementation, the first state parameter includes at least one of a noise level in the operating system, continuity of processor occupancy, a scheduling algorithm for applications in the operating system, and a memory allocation method.

[0011] When the target task is a computationally intensive task, the first state parameter may include the noise level in the operating system; when the target task is a task that requires relatively continuous memory, the first state parameter may include the memory allocation method.

[0012] In this way, based on these parameters, the execution status of the target task when it is executed on the computing node can be better reflected, thereby matching the computing node more accurately.

[0013] In a possible implementation, the first execution status information is detected when the target task is last executed in the first computing node.

[0014] In this way, the actual execution status of the target task when it is executed on the first computing node can be accurately obtained.

[0015] In a possible implementation, the first execution status information is detected when other tasks of the same task type as the target task are executed in the first computing node.

[0016] In this way, when the target task is executed for the first time, the computing nodes can be selected without blindness. For the first computing node allocation, the matching degree between the computing nodes and the target task can be guaranteed to a certain extent, which is conducive to improving the overall execution efficiency.

[0017] In a possible implementation, based on the characteristic information of the first operating system in the first computing node, the hardware resource information of the first computing node, and the first execution state information, the matching degree between the first computing node and the target task can be determined by machine learning. The management node extracts first characteristic information from the characteristic information of the first operating system in the first computing node and the hardware resource information of the first computing node based on the first characteristic extraction model; the management node extracts second characteristic information from the first execution state information based on the second characteristic extraction model; and determines the matching degree between the first computing node and the target task based on the similarity between the first characteristic information and the second characteristic information.

[0018] Among them, the first feature extraction model and the second feature extraction model can be machine learning models, the first feature information and the second feature information can be matrices or vectors, etc. The management node integrates the characteristic information, hardware resource information and the first execution status information into the form of matrices or vectors, and then inputs the characteristic information, hardware resource information and the first execution status information in the form of matrices or vectors into the first feature extraction model and the second feature extraction model respectively, and outputs the first feature information and the second feature information in the form of matrices or vectors.

[0019] There are many algorithms for calculating similarity. For example, when both the first feature information and the second feature information are vectors, the vector angle of the two vectors can be calculated as the similarity, or the Euclidean distance of the two vectors can be calculated as the similarity. For another example, the similarity of the first feature information and the second feature information can be determined by a machine learning model. The calculated similarity can represent the matching degree between the computing node and the target task.

[0020] In this way, we can utilize the high applicability of the machine learning model and, through reasonable training of the machine learning model, make the feature information extracted by it have high value for the calculation of matching degree, improve the accuracy of matching degree, and thus more accurately match computing nodes for target tasks.

[0021] In a possible implementation, the characteristic information includes instruction execution performance information and / or optimization direction information, where the optimization direction information is used to indicate the type of task optimized by the first operating system.

[0022] The instruction execution performance information may include multiple performance scores, each performance score corresponds to a non-general function instruction in the system program, and the performance score is negatively correlated with the execution time of the corresponding non-general function instruction in the computing node (it may be an inverse relationship, etc.). For another example, the instruction execution performance information may be the execution time of each non-general function instruction. The optimization direction information is used to indicate the task type optimized by the operating system. Task types may include computationally intensive, communication intensive, input-output intensive, high parallelism, etc.

[0023] In this way, the instruction execution performance information and optimization direction information can well reflect the advantages of the operating system. The matching degree is calculated based on this information, and the matching computing nodes can better optimize the target tasks and improve the execution efficiency.

[0024] In a possible implementation, when the characteristic information includes instruction execution performance information, based on the characteristic information of the first operating system in the first computing node, the hardware resource information of the first computing node, and the first execution status information, before determining the matching degree between the first computing node and the target task, the first instruction execution performance information of the first operating system can be obtained. The management node sends a first instruction execution notification to the first computing node, wherein the first instruction execution notification may carry non-universal function instructions. The first computing node executes the non-universal function instructions in the local operating system, and detects the performance information of the non-universal function instructions during the execution process to obtain the instruction execution performance information of the first operating system. The management node receives the instruction execution performance information of the first operating system sent by the first computing node.

[0025] After the first computing node executes the non-general function instruction, it can record the corresponding instruction execution performance information, and can send the identifier of the non-general function instruction and the corresponding instruction execution performance information to the management node. The management node can add this information to the instruction execution performance information table corresponding to the operating system of the first computing node.

[0026] In this way, the management node can obtain the instruction execution performance information of the operating system more accurately, and can better understand the performance of the non-general function instructions of the operating system when they are executed in the computing nodes.

[0027] In one possible implementation, the instruction execution performance information is determined based on the instruction execution duration.

[0028] In this way, using execution time can more accurately reflect performance and improve the effectiveness of instruction execution performance information.

[0029] In a possible implementation, a computing node may be assigned to the target task based on the matching degree. If the matching degree is greater than the matching degree threshold, the target task is assigned to the first computing node when the target task is executed next time; if the matching degree is less than the matching degree threshold, a second computing node that has not executed the target task is selected from computing nodes other than the first computing node, and the management node assigns the target task to the second computing node when the target task is executed next time.

[0030] In this way, by using the preset matching degree threshold, it can be determined whether the matching degree of the first computing node meets the matching degree threshold of the target task, and computing nodes can be better allocated to the target task.

[0031] In one possible implementation, after allocating the target task to the second computing node, the management node can detect second execution status information when the target task is executed in the second computing node; determine a second matching degree between the second computing node and the target task based on characteristic information of the second operating system in the second computing node, hardware resource information of the second computing node, and the second execution status information; and allocate a computing node to the target task based on the second matching degree.

[0032] In this way, each computing node can be judged one by one based on the matching degree, and a computing node with a sufficiently high matching degree can be selected for the target task, which can better improve the task execution efficiency.

[0033] In a second aspect, a device for allocating computing nodes to tasks is provided, which is used to implement the method provided by the above-mentioned first aspect and its possible implementation methods.

[0034] In a third aspect, a computer device is provided, the computer device comprising a memory and a processor, the memory being used to store computer instructions. The processor executes the computer instructions stored in the memory, so that the computer device executes the method provided in the first aspect and possible implementations thereof.

[0035] In a fourth aspect, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores a computer program code. When the computer program code is executed by a computer device, the computer device executes the method provided by the first aspect and its possible implementation methods.

[0036] In a fifth aspect, a computer program product is provided. The computer program product includes a computer program code. When the computer program code is executed by a computer device, the computer device executes the method provided by the first aspect and its possible implementation methods. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 is a schematic diagram of the structure of an HPC system provided by an embodiment of the present disclosure;

[0038] Figure 2 is a structural diagram of a management node provided by an embodiment of the present disclosure;

[0039] Figure 3 is a schematic diagram of the structure of a computing node provided by an embodiment of the present disclosure;

[0040] Figure 4 It is a flowchart of a method for allocating computing nodes to tasks provided by an embodiment of the present disclosure;

[0041] Figure 5 is a schematic diagram of a processing flow for determining a matching degree provided by an embodiment of the present disclosure;

[0042] Figure 6 It is a schematic diagram of a processing flow for obtaining instruction execution performance information of an operating system provided by an embodiment of the present disclosure;

[0043] Figure 7 It is a flowchart of a method for allocating computing nodes to tasks provided by an embodiment of the present disclosure;

[0044] Figure 8 It is a structural diagram of a computing node device for task allocation provided by an embodiment of the present disclosure. DETAILED DESCRIPTION

[0045] The following is a brief description of some terms involved in the embodiments of the present disclosure.

[0046] Operating system (OS) :

[0047] An operating system is a set of software programs that manage and control computer operations, execution, and hardware and software work.

[0048] There are various customized operating systems in the HPC system, and each customized operating system is used to optimize tasks with certain characteristics, so that such tasks can have higher operating efficiency or higher resource utilization, etc. The optimization of tasks by the operating system is generally achieved through specific instructions (i.e., the non-general function instructions mentioned below) or instruction sets. The general operating system may also have the above characteristics, that is, it can optimize tasks with certain characteristics, so the processing process of the embodiment of the present disclosure may not distinguish between customized operating systems and general operating systems.

[0049] General function instructions :

[0050] Common function instructions are instructions for functions that most system programs have, that is, instructions for some commonly used functions. It is worth mentioning that in different system programs, although they have the same functions, the corresponding instructions for the same function in different systems are different.

[0051] Non-common function instructions :

[0052] Different from general function instructions, non-general function instructions are instructions for functions possessed by individual system programs. Non-general function instructions are generally specified by technicians in various instructions of system programs. Non-general function instructions are generally related to the optimization characteristics of system programs. When selecting non-general function instructions, technicians can select instructions related to the optimization characteristics of system programs. For example, if the optimization characteristics of a system program are to optimize tasks with high parallelism, the non-general function instructions selected by the technicians can be instructions set to improve resource utilization in a multi-concurrency state of subtasks.

[0053] Operating system feature information :

[0054] The characteristic information is used to indicate the characteristics of the first operating system that are different from other operating systems. The characteristic information may include instruction execution performance information, optimization direction information, etc.

[0055] The instruction execution performance information is information used to reflect the performance of non-general function instructions during execution, and may be related to the execution time of each non-general function instruction in the system program. For example, the instruction execution performance information may include multiple performance scores, each performance score corresponds to a non-general function instruction in the system program, and the performance score is negatively correlated with the execution time of the corresponding non-general function instruction in the computing node (may be an inverse relationship, etc.). For another example, the instruction execution performance information may be the execution time of each non-general function instruction.

[0056] The optimization direction information is used to indicate the type of task optimized by the operating system. The task type may include computation-intensive, communication-intensive, input-output-intensive, high-parallelism, etc.

[0057] Load Application :

[0058] The load application of a task is the program that executes the task. For example, if the task is to execute a complex mathematical calculation, then the load application is the program that performs the data calculation.

[0059] The present disclosure provides a method for allocating computing nodes to tasks, which can be applied to HPC systems. The HPC system runs on a cluster, which can include management nodes and computing nodes. The system architecture diagram can be as follows: Figure 1 The management node can manage and allocate computing resources in the network, and can assign tasks to each computing node according to demand, and the computing node can process the received tasks. In addition, the HPC system can also include storage nodes, which are not shown in the figure.

[0060] Each computing node in the HPC system has its own system program installed. In order to better optimize different types of tasks, the system programs of each computing node may be different, which may include a variety of customized system programs and general system programs. The system programs of some computing nodes may also be the same. For example, 100 computing nodes are divided into 10 groups, each with 10 computing nodes. The operating systems of computing nodes in the same group are the same, while those of computing nodes in different groups are different.

[0061] In addition, the embodiment of the present disclosure can run corresponding applications on the management node and the computing node respectively to execute the method. Figure 1The management process shown in the figure can be used to perform all the related processing of the management node in the subsequent content. The application running on the computing node is established when it runs. Figure 1 In the daemon process shown in , the perception module is a functional module in the daemon process. The related processing of the computing nodes in the subsequent content can be executed by the daemon process, and the processing of parameter collection can be executed by the perception module.

[0062] From the perspective of hardware composition, the structure of the above management node can be as follows: Figure 2 As shown, it includes a processor 210 , a memory 220 , and a communication component 230 .

[0063] Processor 210 may be a central processing unit (CPU), a graphics processing unit (GPU) or a system on chip (SoC), etc. Processor 210 may be used to execute various instructions or data information involved in the method, for example, receiving hardware resource information and the name of the operating system input by an administrator.

[0064] The memory 220 may include various volatile memories or non-volatile memories, such as a solid state disk (SSD), a dynamic random access memory (DRAM), etc. The memory 220 may be used to store pre-stored data, intermediate data, and result data in processing, such as the name of the operating system entered by the administrator, non-general function instructions, etc.

[0065] The communication component 230 may be a wired network connector, a wireless fidelity (WiFi) module, a Bluetooth module, a cellular network communication module, etc. The communication component 230 may be used to transmit data with other devices, and the other devices may be computing nodes, user devices, administrator terminals, etc. For example, a first instruction execution notification is sent to a computing node.

[0066] like Figure 3 As shown, the computing node may include a processor 310 , a memory 320 , and a communication component 330 .

[0067] The processor 310 may be a CPU, a GPU or a SoC, etc. The processor 310 may be used to execute various instructions or data information involved in the method, such as non-general function instructions.

[0068] The memory 320 may include various volatile memories or non-volatile memories, such as SSD, DRAM memory, etc. The memory 320 may be used to store pre-stored data, intermediate data, and result data in processing, for example, execution status information, etc.

[0069] The communication component 330 may be a wired network connector, a WiFi module, a Bluetooth module, a cellular network communication module, etc. The communication component 330 may also be used to transmit data with other devices, and the other devices may be management nodes, etc. For example, receiving non-general function instructions, instruction execution notifications, etc. sent by the management node.

[0070] With the development of HPC technology, different computing nodes in HPC systems often run different operating systems, and different operating systems can be optimized for tasks with different characteristics. If the advantages of the operating system of each computing node can be taken into account and tasks are allocated, the efficiency of the HPC system can be greatly improved.

[0071] The present disclosure provides a method for allocating computing nodes to tasks in an HPC system. The corresponding processing can be as follows: Figure 4 As shown, the following steps are included:

[0072] 401. The management node obtains execution status information corresponding to the target task.

[0073] Among them, the target task is the task to be executed currently, and the management node needs to allocate a computing node for the target task. The execution status information is used to reflect the execution status of the target task when it is executed in a certain computing node. The execution status information may include the parameter value of the state parameter, and the state parameter is the state parameter corresponding to the task type of the target task. The state parameter may include at least one of the noise size in the operating system, the continuity of the processor occupancy, the memory allocation method, and the scheduling algorithm of the operating system for the load application, wherein the noise is a factor that affects the execution of the target task by the computing node (interrupting the execution of the target task, etc.), the memory allocation method refers to whether the memory that can be allocated to the target task in the computing node is continuous or scattered, and the scheduling algorithm is an algorithm in the operating system that can affect the operating mode of the load application. For example, when the task type of the target task is a computing-intensive task, the state parameter may include the noise size in the operating system, and when the target task belongs to a task type that requires relatively continuous memory, the state parameter may include the memory allocation method. With the diversification of operating systems, state parameters may increase.

[0074] The user device sends an execution request of the target task to the management node, and can upload the load application of the target task and the hardware resource requirement information of the target task at the same time. The management node can first select a computing node (which can be called the first computing node) from each computing node, and can specifically select randomly from all computing nodes that meet the hardware resource requirement information, or select by polling, or select the computing node that is finally assigned to other tasks of the same task type as the target task during the historical execution process.

[0075] There are many possible ways to obtain execution status information. Two possible ways are given below:

[0076] In the first approach, the execution status information is detected when the target task was last executed in the first computing node.

[0077] Each task may be executed multiple times. For example, a weather forecast task can be executed once a day or once an hour. When any task is executed on any computing node, the computing node can determine the task type of the task, and then find at least one state parameter corresponding to the task type of the task in the pre-stored correspondence between the task type and the state parameter. Later, during the execution of the task, the computing node can detect the parameter value of the at least one state parameter, which is the execution state information. The execution state information corresponds to the task and the computing node. The computing node can send the execution state information, the identifier of the task, and the identifier of the computing node to the management node. The management node can add this information to the corresponding relationship table of the execution state information, the task, and the computing node.

[0078] Each computing node processes in the above manner when executing any task, and the above correspondence table can be enriched in the management node, and the correspondence table can also be called an execution status information library.

[0079] After determining the first computing node, the management node may allocate the target task to the first computing node for execution for a specified number of times (eg, once), so that the management node can obtain and record corresponding execution status information.

[0080] In the second method, the execution status information is obtained by detecting other tasks of the same task type as the target task when they are executed in the computing node.

[0081] In the second method, the execution status information library can also be established in a similar way to the first method. When the computing node sends the execution status information to the management node, the corresponding task type can also be sent and added to the execution status information library. In this way, the execution status information library records the corresponding relationship table of the execution status information, tasks, task types, and computing nodes.

[0082] After determining the first computing node, the management node may search for execution status information of other tasks of the same type in the execution status information library based on the task type of the target task and the identifier of the first computing node as the execution status information corresponding to the target task.

[0083] 402. The management node determines a matching degree between the first computing node and the target task based on characteristic information of the operating system in the first computing node, hardware resource information and execution status information of the first computing node.

[0084] The characteristic information can refer to the glossary at the beginning of the embodiment. The hardware resource information of the computing node may include the number of processors, memory capacity, network bandwidth, etc. The higher the matching degree between the first computing node and the target task, the higher the execution efficiency of the target task in the first computing node.

[0085] After obtaining parameters such as characteristic information, hardware resource information, and execution status information, the management node can determine the matching degree between the first computing node and the target task through a machine learning model. The corresponding processing will be described in detail later.

[0086] 403, the management node allocates computing nodes to the target task based on the matching degree.

[0087] After the management node obtains the matching degree between the computing node and the target task, it can compare it with the preset matching degree threshold to determine whether the matching degree reaches the preset matching degree threshold. If the matching degree is greater than the matching degree threshold, the target task will be assigned to the first computing node when the target task is executed next time; if the matching degree is less than the matching degree threshold, other computing nodes that have not executed the target task are selected from computing nodes other than the first computing node, and the target task will be assigned to other computing nodes when the target task is executed next time. Among them, other computing nodes that have not executed the target task can also be considered as computing nodes that have not been selected by the management node to perform matching degree calculation with the target task.

[0088] By using this method to allocate computing nodes to tasks, it is possible to consider not only hardware resources but also the characteristic information of the operating system in the computing nodes and the actual execution status of the computing nodes when executing tasks. This takes into account more comprehensive factors affecting the efficiency of task execution and can better improve the efficiency of task execution.

[0089] In the embodiment of the present disclosure, the matching degree between the computing node and the target task can be determined by machine learning, and the corresponding processing can be as follows: Figure 5 As shown, the following steps are included:

[0090] 501. The management node extracts first feature information from characteristic information of an operating system in a computing node and hardware resource information of the computing node based on a first feature extraction model.

[0091] The first feature extraction model may be a machine learning model, such as a neural network model, etc. The first feature information may be a matrix or a vector, etc.

[0092] The management node integrates the characteristic information and the hardware resource information into a matrix or vector form, and then inputs the integrated matrix or vector into the first feature extraction model. The first feature extraction model can then output the first feature information in a matrix or vector form.

[0093] 502. The management node extracts second feature information from the execution status information corresponding to the target task based on the second feature extraction model.

[0094] The second feature extraction model may be a machine learning model, such as a neural network model, etc. The second feature information may be a matrix or a vector, etc.

[0095] The management node sets the execution state information in the form of a matrix or a vector and then inputs the execution state information in the form of a matrix or a vector into the second feature extraction model. The second feature extraction model can then output the second feature information in the form of an output matrix or a vector.

[0096] 503. The management node determines a matching degree between the computing node and the target task based on a similarity between the first feature information and the second feature information.

[0097] After determining the first feature information and the second feature information, the management node can calculate the similarity of the first feature information and the second feature information. There can be many algorithms for calculating the similarity. For example, when the first feature information and the second feature information are both vectors, the vector angle of the two vectors can be calculated as the similarity, or the Euclidean distance of the two vectors can be calculated as the similarity. For another example, the similarity of the first feature information and the second feature information can be determined by a machine learning model. The calculated similarity can represent the matching degree between the computing node and the target task.

[0098] In order to make the similarity between the first feature information extracted by the first feature extraction model and the second feature information extracted by the second feature extraction model accurately reflect the matching degree between the computing node and the target task, the first feature extraction model and the second feature extraction model can be trained based on a large number of samples. The training process can be as follows:

[0099] For a sample task, the execution status information of the sample task when it is executed on a computing node and the characteristic information and hardware resource information of the operating system in the computing node are obtained as sample execution status information, sample characteristic information, and sample hardware resource information. A technician can perform manual calibration based on this information to determine the matching degree between the computing node and the sample task as a benchmark matching degree. The matching degree can be determined based on information such as sample execution status information, sample characteristic information, sample hardware resource information, task type, etc., combined with empirical judgment, or the sample task can be executed on computing nodes of multiple different operating systems to obtain execution status information, and then determined after comparison. The sample execution status information, sample characteristic information, sample hardware resource information, and benchmark matching degree can be determined as a group of samples. In a similar manner, multiple groups of samples can be obtained.

[0100] A set of samples is obtained, and the sample characteristic information and sample hardware resource information are input into the first feature extraction model to be trained, and the first sample feature information is output. The sample execution state information is input into the second feature extraction model to be trained, and the second sample feature information is output. The predicted matching degree is calculated based on the first sample feature information and the second sample feature information, and the loss value is determined based on the predicted matching degree and the benchmark matching degree of the sample task, and the first feature extraction model and the second feature extraction model to be trained are adjusted based on the loss value. The above training and parameter adjustment process is performed using multiple sets of samples until the training end condition is met, and the first feature extraction model and the second feature extraction model after parameter adjustment are determined as the first feature extraction model and the second feature extraction model that have completed training. The training end condition can be reaching a specified number of training times (which can be set by a technician based on experience or based on the total number of samples), or the loss value is less than a preset threshold value (which can be set by a technician based on experience) in N consecutive training times (N is a preset value, which can be set by a technician based on experience or based on the total number of samples), and so on.

[0101] In the embodiment of the present disclosure, a processing method for obtaining the instruction execution performance information of the operating system is provided. The corresponding processing can be as follows: Figure 6 As shown, the following steps are included:

[0102] 601. The management node sends a command execution notification to each computing node.

[0103] The administrator can upload a configuration file to the management node, which includes the non-universal function instructions, optimization direction information, and identification information of the operating system that are specific to each computing node's operating system. After obtaining this information, the management node can generate a perception script for each operating system based on the non-universal function instructions of the operating system. The content of the perception script is: instruct the computing node to execute each non-universal function instruction of the operating system one by one, and detect the performance information of each non-universal function instruction, obtain the instruction execution performance information, and send it to the management node. Then, the management node can carry the perception script corresponding to each operating system in the instruction execution notification and send it to the computing node where the operating system is located.

[0104] 602. The computing node executes the non-general function instruction in the local operating system, and detects the performance information of the non-general function instruction during the execution process to obtain the instruction execution performance information of the operating system.

[0105] After receiving the instruction execution notification, the computing node can execute non-general instructions in the local operating system based on the perception script, and detect the performance information of the non-general function instructions during the execution process to obtain the instruction execution performance information of the operating system.

[0106] 603. The computing node sends the instruction execution performance information of the operating system to the management node.

[0107] After the computing node executes the non-general function instruction, it can record the corresponding instruction execution performance information, and can send the identification of the non-general function instruction and the corresponding instruction execution performance information to the management node. The management node can add this information to the instruction execution performance information table corresponding to the computing node's operating system.

[0108] In the embodiment of the present disclosure, computing nodes can be selected cyclically to perform matching judgment with the target task, and finally computing nodes with sufficient matching degree are selected and assigned to the target task. The corresponding method of assigning computing nodes can be as follows: Figure 7 As shown, the following steps are included:

[0109] 701. The management node sets the first computing node as a computing node to be matched.

[0110] The user device sends a request to execute the target task to the management node. The management node may first select a computing node from each computing node as the first computing node. The selection of the first computing node may refer to the selection method in the above step 401. The management node may set the first computing node as the computing node to be matched to prepare for subsequent steps. The computing node to be matched is the computing node that is currently to be determined whether it matches the target task.

[0111] 702 , the management node assigns the target task to the to-be-matched computing node for execution once, and obtains the execution status information detected by the to-be-matched computing node.

[0112] After determining the computing node to be matched, the management node assigns the target task to the computing node to be matched for execution once, and obtains the execution status information detected by the computing node to be matched, wherein the acquisition of the execution status information can refer to the method in the above step 401.

[0113] 703 , the management node determines the matching degree between the computing node to be matched and the target task based on the characteristic information of the operating system in the computing node to be matched, the hardware resource information and the execution state information of the computing node to be matched.

[0114] The characteristic information can refer to the glossary at the beginning of the embodiment, and the method for calculating the matching degree can refer to the above Figure 5 The processing flow.

[0115] 704, the management node compares the matching degree with a preset matching degree threshold. If the matching degree is greater than the matching degree threshold, step 705 is executed; if the matching degree is less than the matching degree threshold, step 706 is executed.

[0116] When the matching degree is equal to the matching degree threshold, step 705 may be executed, or step 706 may be executed, which may be set by the administrator according to actual conditions.

[0117] 705 , the management node allocates the target task to the computing node to be matched each time the target task is subsequently executed.

[0118] That is to say, the current to-be-matched computing node is determined as the execution computing node of the target task, and in the subsequent execution process, the target task will always be assigned to the execution computing node.

[0119] 706 , the management node selects other computing nodes that have not executed the target task from the computing nodes other than the computing nodes to be matched, sets them as the computing nodes to be matched, and goes to step 702 .

[0120] If the matching degree is less than the matching degree threshold, it means that the current computing node to be matched is not suitable for executing the target task. At this time, another computing node can be selected as the computing node to be matched, and the processing of steps 702-706 can be re-executed. If it is still not suitable, the next computing node can be selected until the matching degree between the selected computing node and the target task is greater than the matching degree threshold. Then, the computing node will be set as the execution computing node of the target task. In the subsequent execution process, the target task will always be assigned to the execution computing node.

[0121] By using this method to allocate computing nodes to tasks, it is possible to consider not only hardware resources but also the characteristic information of the operating system in the computing nodes and the actual execution status of the computing nodes when executing tasks. This takes into account more comprehensive factors affecting the efficiency of task execution and can better improve the efficiency of task execution.

[0122] Based on the same technical concept, an embodiment of the present disclosure provides a device for allocating computing nodes to tasks, the device is applied to an HPC system, the HPC system runs on a cluster, the cluster includes a management node and multiple computing nodes, the device is applied to the management node, such as Figure 8 As shown, the device comprises:

[0123] The acquisition module 810 is used to acquire first execution status information, wherein the first execution status information is used to indicate the execution status of the target task when it is executed in the first computing node. Specifically, the processing functions of the above steps 401 and 702 and other implicit steps can be implemented.

[0124] The determination module 820 is used to determine the first matching degree between the first computing node and the target task based on the characteristic information of the first operating system in the first computing node, the hardware resource information of the first computing node, and the first execution state information, wherein the characteristic information is used to indicate the characteristics of the first operating system that are different from other operating systems. Specifically, the processing functions of the above steps 402, 503, 703, and other implicit steps can be implemented.

[0125] The allocation module 830 is used to allocate computing nodes to the target task based on the first matching degree, and can specifically implement the processing functions of the above steps 403, 705, 706, and other implicit steps.

[0126] In a possible implementation manner, the first execution state information includes a parameter value of a first state parameter, where the first state parameter is a state parameter corresponding to a task type of the target task.

[0127] In a possible implementation, the first state parameter includes at least one of a noise level in the operating system, continuity of processor occupancy, memory allocation method, and a scheduling algorithm used by the operating system for load applications.

[0128] In a possible implementation, the first execution status information is detected when the target task is last executed in the first computing node.

[0129] In a possible implementation, the first execution status information is detected when other tasks of the same task type as the target task are executed in the first computing node.

[0130] In a possible implementation, the determination module 820 is used to: extract first feature information from the characteristic information of the first operating system in the first computing node and the hardware resource information of the first computing node based on the first feature extraction model; extract second feature information from the first execution state information based on the second feature extraction model; and determine the matching degree between the first computing node and the target task based on the similarity between the first feature information and the second feature information. Specifically, the processing functions of the above steps 501, 502, and 503, as well as other implicit steps, can be implemented.

[0131] In a possible implementation, the characteristic information includes instruction execution performance information and / or optimization direction information, where the optimization direction information is used to indicate the type of task optimized by the first operating system.

[0132] In a possible implementation, the acquisition module 810 is further used to: send a first instruction execution notification to the first computing node, wherein the first instruction execution notification is used to instruct the first computing node to execute a non-general function instruction in the first operating system, and detect the performance information of the non-general function instruction during the execution process to obtain the instruction execution performance information of the first operating system; receive the instruction execution performance information of the first operating system sent by the first computing node. Specifically, the processing functions of the above steps 601, 602, 603, and 703, as well as other implicit steps, can be implemented.

[0133] In one possible implementation, the instruction execution performance information is determined based on the instruction execution duration.

[0134] In a possible implementation, the allocation module 830 is used to: if the matching degree is greater than the matching degree threshold, the target task is allocated to the first computing node when the target task is executed next time; if the matching degree is less than the matching degree threshold, a second computing node that has not executed the target task is selected from computing nodes other than the first computing node, and the target task is allocated to the second computing node when the target task is executed next time. Specifically, the processing functions of the above steps 704, 705, and 706, as well as other implicit steps, can be implemented.

[0135] In a possible implementation, the allocation module 830 is further used to: detect the second execution state information when the target task is executed in the second computing node; determine the second matching degree between the second computing node and the target task based on the characteristic information of the second operating system in the second computing node, the hardware resource information of the second computing node, and the second execution state information; and allocate a computing node to the target task based on the second matching degree. Specifically, the processing functions of the above steps 704, 705, and 706, as well as other implicit steps, can be implemented.

[0136] The acquisition module 810, determination module 820, and allocation module 830 may be implemented by a processor, or by a processor in conjunction with a memory and a communication component.

[0137] By using this method to allocate computing nodes to tasks, it is possible to consider not only hardware resources but also the characteristic information of the operating system in the computing nodes and the actual execution status of the computing nodes when executing tasks. This takes into account more comprehensive factors affecting the efficiency of task execution and can better improve the efficiency of task execution.

[0138] The above embodiments provide a device for allocating computing nodes for tasks. When allocating computing nodes, only the division of the above functional modules is used as an example. In practical applications, the above functional allocation can be completed by different functional modules as needed, that is, the internal structure of the user equipment is divided into different functional modules to complete all or part of the functions described above. In addition, the device for allocating computing nodes for tasks provided by the above embodiments and the method embodiment for allocating computing nodes for tasks belong to the same concept. The specific implementation process is detailed in the method embodiment, which will not be repeated here.

[0139] The embodiments of the present disclosure also provide a computer-readable storage medium. The computer-readable storage medium may be any available medium that can be stored by a computing device or a data storage device such as a data center that includes one or more available media. The available medium may be a magnetic medium (e.g., a floppy disk, a hard disk, a magnetic tape), an optical medium (e.g., a digital video disk (DVD)), or a semiconductor medium (e.g., a solid-state hard disk). The computer-readable storage medium includes instructions that instruct a computing device to perform a business processing method, or instructs a computing device to perform a business processing method.

[0140] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present disclosure, rather than to limit them. Although the present disclosure has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the protection scope of the technical solutions of the embodiments of the present disclosure.

Claims

1. A method for allocating computing nodes to tasks, characterized in that: The method is applied to a high performance computing (HPC) system, the HPC system runs on a cluster, the cluster includes a management node and multiple computing nodes, the method is executed by the management node, and the method includes: Acquire first execution status information, wherein the first execution status information is used to indicate an execution status of the target task when it is executed in the first computing node; Determining a first matching degree between the first computing node and the target task based on characteristic information of a first operating system in the first computing node, hardware resource information of the first computing node, and the first execution state information, wherein the characteristic information is used to indicate a characteristic of the first operating system that is different from other operating systems; Based on the first matching degree, a computing node is allocated to the target task.

2. The method according to claim 1, characterized in that The first execution status information includes a parameter value of a first status parameter, where the first status parameter is a status parameter corresponding to the task type of the target task.

3. The method according to claim 1 or 2, characterized in that: The first state parameter includes at least one of the noise level in the operating system, the continuity of processor occupancy, the memory allocation method, and the scheduling algorithm of the operating system for load application.

4. The method according to any one of claims 1 to 3, characterized in that: The first execution status information is detected when the target task was last executed in the first computing node.

5. The method according to any one of claims 1 to 3, characterized in that: The first execution status information is detected when other tasks of the same task type as the target task are executed in the first computing node.

6. The method according to any one of claims 1 to 5, characterized in that: The determining, based on the characteristic information of the first operating system in the first computing node, the hardware resource information of the first computing node, and the first execution state information, a matching degree between the first computing node and the target task includes: Extracting first feature information from characteristic information of a first operating system in the first computing node and hardware resource information of the first computing node based on a first feature extraction model; extracting second feature information from the first execution state information based on a second feature extraction model; Based on the similarity between the first feature information and the second feature information, a matching degree between the first computing node and the target task is determined.

7. The method according to any one of claims 1 to 6, characterized in that: The characteristic information includes instruction execution performance information and / or optimization direction information, and the optimization direction information is used to indicate the task type optimized by the first operating system.

8. The method according to any one of claims 1 to 7, characterized in that: When the characteristic information includes instruction execution performance information, before determining the matching degree between the first computing node and the target task based on the characteristic information of the first operating system in the first computing node, the hardware resource information of the first computing node, and the first execution state information, the method further includes: Sending a first instruction execution notification to the first computing node, wherein the first instruction execution notification is used to instruct the first computing node to execute a non-general function instruction in the first operating system, and to detect performance information of the non-general function instruction during execution, so as to obtain instruction execution performance information of the first operating system; Receive instruction execution performance information of the first operating system sent by the first computing node.

9. The method according to claim 8, characterized in that The instruction execution performance information is determined based on the instruction execution duration.

10. The method according to any one of claims 1 to 9, characterized in that: The allocating a computing node to the target task based on the matching degree includes: If the matching degree is greater than the matching degree threshold, the target task is allocated to the first computing node when the target task is executed next time; If the matching degree is less than the matching degree threshold, a second computing node that has not executed the target task is selected from computing nodes other than the first computing node, and the target task is allocated to the second computing node when the target task is executed next time.

11. The method according to claim 10, characterized in that After allocating the target task to the second computing node, the method further includes: Detecting second execution state information of the target task when it is executed in the second computing node; Determining a second matching degree between the second computing node and the target task based on characteristic information of the second operating system in the second computing node, hardware resource information of the second computing node, and the second execution state information; Based on the second matching degree, a computing node is allocated to the target task.

12. A device for allocating computing nodes to tasks, characterized in that: The device is applied to an HPC system, the HPC system runs on a cluster, the cluster includes a management node and multiple computing nodes, the device is applied to the management node, and the device includes: An acquisition module, used to acquire first execution status information, wherein the first execution status information is used to indicate an execution status of a target task when it is executed in a first computing node; a determination module, configured to determine a first matching degree between the first computing node and the target task based on characteristic information of a first operating system in the first computing node, hardware resource information of the first computing node, and the first execution state information, wherein the characteristic information is used to indicate a characteristic of the first operating system that is different from other operating systems; An allocation module is used to allocate a computing node to the target task based on the first matching degree.

13. The device according to claim 12, characterized in that The first execution status information includes a parameter value of a first status parameter, where the first status parameter is a status parameter corresponding to the task type of the target task.

14. The device according to claim 12 or 13, characterized in that The first state parameter includes at least one of the noise level in the operating system, the continuity of processor occupancy, the memory allocation method, and the scheduling algorithm of the operating system for load application.

15. The device according to any one of claims 12 to 14, characterized in that: The first execution status information is detected when the target task was last executed in the first computing node.

16. The device according to any one of claims 12 to 14, characterized in that: The first execution status information is detected when other tasks of the same task type as the target task are executed in the first computing node.

17. The device according to any one of claims 12 to 16, characterized in that: The determining module is used to: Extracting first feature information from characteristic information of a first operating system in the first computing node and hardware resource information of the first computing node based on a first feature extraction model; extracting second feature information from the first execution state information based on a second feature extraction model; Based on the similarity between the first feature information and the second feature information, a matching degree between the first computing node and the target task is determined.

18. The device according to any one of claims 12 to 17, characterized in that: The characteristic information includes instruction execution performance information and / or optimization direction information, and the optimization direction information is used to indicate the task type optimized by the first operating system.

19. The device according to any one of claims 12 to 18, characterized in that: The acquisition module is further used for: Sending a first instruction execution notification to the first computing node, wherein the first instruction execution notification is used to instruct the first computing node to execute a non-general function instruction in the first operating system, and to detect performance information of the non-general function instruction during execution, so as to obtain instruction execution performance information of the first operating system; Receive instruction execution performance information of the first operating system sent by the first computing node.

20. The device according to claim 19, characterized in that The instruction execution performance information is determined based on the instruction execution duration.

21. The device according to any one of claims 12 to 20, characterized in that: The allocation module is used to: If the matching degree is greater than the matching degree threshold, the target task is allocated to the first computing node when the target task is executed next time; If the matching degree is less than the matching degree threshold, a second computing node that has not executed the target task is selected from computing nodes other than the first computing node, and the target task is allocated to the second computing node when the target task is executed next time.

22. The device according to claim 21, characterized in that The allocation module is further used for: Detecting second execution state information of the target task when it is executed in the second computing node; Determining a second matching degree between the second computing node and the target task based on characteristic information of the second operating system in the second computing node, hardware resource information of the second computing node, and the second execution state information; Based on the second matching degree, a computing node is allocated to the target task.

23. A computer device, characterized in that: The computer device comprises a memory and a processor, wherein the memory is used to store computer instructions; The processor executes the computer instructions stored in the memory to enable the computer device to perform the method according to any one of claims 1 to 11.

24. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program code. When the computer program code is executed by a computer device, the computer device executes the method according to any one of claims 1 to 11.

Images