Method and computer device for determining computing nodes based on processes

Abstract

The application discloses a method and computer equipment for determining a computing node based on a process, relates to the technical field of high-performance computing, and comprises the following steps: firstly, determining a candidate node array combination according to a first target number of the computing node; judging the proportion value of the computing node and the process in different directions in the candidate node array combination; determining a target node array combination by judging whether the proportion value is an integer, and determining a target computing node. The method can solve the problem that the one-dimensional sequential mapping mode is used to determine the computing node to which the MPI process belongs, a large amount of cross-node communication exists between the computing nodes, the performance of the WRF mode is affected, and the like. The method can reduce the cross-node communication between different computing nodes, more uses the internal communication of the node, thereby reducing the running time of the WRF mode, improving the computing efficiency, and reducing the energy consumption required by the WRF mode calculation.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of high-performance computing, and in particular to a method for determining a computing node based on a process and a computer device. BACKGROUND

[0002] WRF (Weather Research and Forecasting Model) is one of the most commonly used numerical models in the field of meteorology and oceanography. Currently, the parallel division scheme adopted by the WRF model is to divide the forecast area into N sub-regions, and each MPI (Message Passing Interface) process is responsible for a sub-region. Assuming that there are Nx processes in the x direction and Ny processes in the y direction, the number of processes N is equal to the product of Nx and Ny. After determining the number of processes in the x direction and the y direction, the mapping relationship between the MPI process and the computing node needs to be determined, and through the mapping relationship, it can be determined which MPI processes are used by the computing node.

[0003] Currently, a one-dimensional sequential mapping method is used to determine the mapping relationship between the MPI process and the computing node. After determining the number of processes used by each computing node, the computing nodes are arranged in sequence. Using the one-dimensional sequential mapping method to determine the computing nodes and the MPI processes contained in the computing nodes will result in a large amount of inter-node communication demand between the computing nodes, affecting the performance of the WRF model.

[0004] Therefore, the related technology has the problem that the one-dimensional sequential mapping method is used to determine the computing node to which the MPI process belongs, resulting in a large amount of cross-node communication between the computing nodes, affecting the performance of the WRF model. SUMMARY

[0005] Therefore, the present application provides a method for determining a computing node based on a process and a computer device to solve the problem that the one-dimensional sequential mapping method is used to determine the computing node to which the MPI process belongs, resulting in a large amount of cross-node communication between the computing nodes, affecting the performance of the WRF model.

[0006] In a first aspect, the present application provides a method for determining a computing node based on a process, which comprises:

[0007] After determining the first number of processes in the first direction and the second number of processes in the second direction, a preset number of candidate node combinations are obtained according to the first target number of computing nodes, wherein the candidate node combination contains the first number of nodes in the first direction and the second number of nodes in the second direction, and the product of the first number and the second number is the first target number;

[0008] determine a first ratio value of the first node number and the first process number, and determine a second ratio value of the second node number and the second process number;

[0009] If it is determined that the target node combination exists in the candidate node combination, the target computing node is determined according to the target node combination, wherein the first ratio value and the second ratio value corresponding to the target node combination are integers.

[0010] In a second aspect, the present application provides a device for determining a computing node based on a process, the device comprising:

[0011] The combination determining module is configured to, in a case where the first process number in the first direction and the second process number in the second direction are determined, obtain a preset number of candidate node combinations according to the first target number of the computing nodes, wherein the candidate node combinations contain a first node number of the computing nodes in the first direction and a second node number of the computing nodes in the second direction, and the product of the first node number and the second node number is the first target number.

[0012] The ratio value determining module is configured to determine a first ratio value of the first node number and the first process number, and determine a second ratio value of the second node number and the second process number.

[0013] The node determining module is configured to, if it is determined that the target node combination exists in the candidate node combination, determine the target computing node according to the target node combination, wherein the first ratio value and the second ratio value corresponding to the target node combination are integers.

[0014] In a third aspect, the present application provides a computer device, comprising a memory and a processor, the memory and the processor are communicatively connected with each other, the memory stores computer instructions, and the processor executes the computer instructions to perform the process determining method of the computing node based on the process according to the first aspect or any one of the corresponding embodiments thereof.

[0015] In a fourth aspect, the present application provides a computer readable storage medium, the computer readable storage medium stores computer instructions, and the computer instructions are used to make the computer execute the process determining method of the computing node based on the process according to the first aspect or any one of the corresponding embodiments thereof.

[0016] In a fifth aspect, the present application provides a computer program product, comprising computer instructions, and the computer instructions are used to make the computer execute the process determining method of the computing node based on the process according to the first aspect or any one of the corresponding embodiments thereof.

[0017] According to the application, the method first determines a candidate node array combination according to the first target number of computing nodes, judges the ratio value of the computing nodes and the processes in different directions in the candidate node array combination, determines the target node array combination by judging whether the ratio value is an integer, and determines the target computing node. The problem that the one-dimensional sequential mapping method for determining the computing node to which the MPI process belongs causes a large amount of cross-node communication between the computing nodes, affects the performance of the WRF model, and the like can be solved. The method can reduce the cross-node communication between different computing nodes by arranging the computing nodes in two dimensions, more use internal communication of the nodes, thereby reducing the running time of the WRF model, improving the computing efficiency, and reducing the energy consumption required by the WRF model. BRIEF DESCRIPTION OF DRAWINGS

[0018] In order to more clearly illustrate the technical solutions in the specific embodiments or related art of the present application, the drawings needed to be used in the specific embodiments or related art description will be briefly introduced. Obviously, the drawings in the following description are some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor on the basis of these drawings.

[0019] Figure 1 is a flow diagram of a method for determining a computing node based on a process according to an embodiment of the present application;

[0020] Figure 2 is a schematic diagram of determining a computing node by using a two-dimensional arrangement method according to an embodiment of the present application;

[0021] Figure 3 is a schematic diagram of determining a computing node by using a one-dimensional sequential mapping method according to an embodiment of the present application;

[0022] Figure 4 is a flowchart of an MPI process and computing node mapping method for a WRF model according to an embodiment of the present application;

[0023] Figure 5 is a structural block diagram of an apparatus for determining a computing node based on a process according to an embodiment of the present application;

[0024] Figure 6 is a hardware structure schematic diagram of a computer device according to an embodiment of the present application. DETAILED DESCRIPTION

[0025] With reference to the drawings, the technical solutions in the embodiments of the present application will be clearly and completely described below. Obviously, the described embodiments are only a part of the embodiments of the present application, but not all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort should fall within the protection scope of the present application.

[0026] It should be noted that in the description of the present application, the terms "comprising", "containing" or any other variants thereof are intended to cover non-exclusive inclusion, so that the process, method, article or equipment including a series of elements not only includes those elements, but also includes other elements not explicitly listed or inherent to such process, method, article or equipment. The terms "first", "second" and the like in the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence.

[0027] In order for those skilled in the art to better understand the present application, the present application will be further described in detail below with reference to the drawings and specific embodiments.

[0028] WRF model is a commonly used numerical weather and climate prediction model in meteorology, and is one of the numerical weather prediction models mainly used in meteorological business prediction. Each meteorological business center runs WRF model on a high-performance computer to output weather prediction results such as temperature and precipitation by using multi-node parallel mode. The model can run in the form of pure MPI process, pure OpenMP (open multi-processing, shared memory parallel programming) thread or MPI process + OpenMP thread combination. The most commonly used running mode is MPI process + OpenMP thread, which can support WRF cross-node parallel and communication using shared memory of the node. When WRF model runs in MPI process + OpenMP thread mode, it needs to determine the parallel division scheme adopted by WRF, for example: the prediction area is divided into N sub-areas by using the number of processes N, and each process is responsible for a sub-area. Assuming that there are Nx processes in the x direction and Ny processes in the y direction, N is equal to the product of Nx and Ny. For the values of Nx and Ny, the currently default automatic parallel division scheme of WRF is Nx=Ny=N1 / 2. After determining the number of processes in the x direction and the y direction, the mapping of MPI process and computing node needs to be determined. The commonly used one-dimensional sequential mapping method is to arrange the computing nodes in sequence after determining the number of processes of each computing node.

[0029] Based on the above, the embodiment of the application provides a method for determining a computing node based on a process, and the application proposes an improved MPI process and computing node mapping method, that is, the computing nodes are arranged in two dimensions, so that the number of computing nodes Kx in the x direction and the number of computing nodes Ky in the y direction are as consistent as possible, and the difference between the number of computing nodes Ky in the y direction and the number of computing nodes Kx in the x direction is as small as possible. Compared with the general one-dimensional sequential mapping method, the mapping method can significantly reduce the cross-node communication between different computing nodes, and more use of internal node communication, so as to reduce the running time of the WRF model, improve the efficiency, and then reduce the energy consumption required for a single weather forecast and save electricity.

[0030] According to the embodiment of the application, an embodiment of determining a computing node based on a process is provided. It should be noted that the steps shown in the flowchart of the drawings can be executed in a computer system such as a group of computer executable instructions, for example: computer, server, etc., and although the logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from here.

[0031] In the embodiment, a method for determining a computing node based on a process is provided, Figure 1 is a flowchart of the method for determining a computing node based on a process according to the embodiment of the application, as Figure 1 shown, the flow includes the following steps:

[0032] Step S101, in the case of determining the first process number in the first direction and the second process number in the second direction, according to the first target number of computing nodes, a preset number of candidate node array combinations are obtained, wherein the candidate node array combination contains the first node number of computing nodes in the first direction and the second node number of computing nodes in the second direction, and the product of the first node number and the second node number is the first target number.

[0033] Specifically, as Figure 2 shown, the first direction is, for example: x direction, and the second direction is, for example: y direction. The number of processes N is used to divide the forecast area into N sub-regions, and each MPI process is responsible for a sub-region. The number of MPI processes N used to run the WRF model is determined, and the first target number K of computing nodes needs to satisfy that N can be divided by K. Taking the case that 90 MPI processes are specified to be used, that is, N is equal to 90, and 15 computing nodes are specified to be used, and each computing node uses 6 processes as an example. According to the parallel division scheme automatically executed by the WRF model by default, the number of MPI processes Nx in the x direction, that is, the first process number in the first direction, and the number of MPI processes Ny in the y direction, that is, the second process number in the second direction, are determined. For example, the number of MPI processes Nx in the x direction is determined to be equal to 9, and the number of MPI processes Ny in the y direction is determined to be equal to 10.

[0034] In the case of determining the first process number in the first direction and the second process number in the second direction, according to the first target number of the computing nodes, a preset number of candidate node combinations (Kx, Ky) is obtained, and the candidate node combination contains the first node number Kx of the computing nodes in the first direction and the second node number Ky of the computing nodes in the second direction. The preset number represents a plurality, which is not limited in quantity here. For example, the first target number is 15, and the candidate node combinations include (3, 5) and (1, 15). In (3, 5), the first node number is 3, the second node number is 5, and the product of the two is 15, which is the same as the first target number.

[0035] In step S102, the first ratio value of the first node number and the first process number is determined, and the second ratio value of the second node number and the second process number is determined.

[0036] Specifically, each candidate node combination (Kx, Ky) is traversed in turn to see whether the MPI process number Nx in the first direction can be evenly divided by the first node number Kx of the computing nodes in the first direction and whether the MPI process number Ny in the second direction can be evenly divided by the second node number Ky of the computing nodes in the second direction. The first ratio value of the first node number and the first process number can be determined first, and the second ratio value of the second node number and the second process number can be determined. It is determined whether the first ratio value and the second ratio value are integers to determine whether they can be evenly divided. For example, Nx = 9, Ny = 10, and the candidate node combination is (3, 5). Kx = 3, Ky = 5, the first ratio value is 3, and the second ratio value is 2. Therefore, Nx can be evenly divided by Kx, and Ny can be evenly divided by Ky.

[0037] In step S103, if it is determined that there is a target node combination in the candidate node combination, the target computing node is determined according to the target node combination, wherein the first ratio value and the second ratio value corresponding to the target node combination are both integers.

[0038] Specifically, if there is a candidate node combination whose first ratio value and second ratio value are both integers, it is taken as the target node combination. If the number of target node combinations is not 1, the difference between the first node number and the second node number in the target node combination is determined, and the target computing node is determined by selecting the target node combination with the smallest difference. For example, the candidate node combination (3, 5) corresponds to the first ratio value of 3 and the second ratio value of 2, which are both integers. The candidate node combination (3, 5) is taken as the target node combination. Kx and Ky in the target node combination (3, 5) are respectively assigned to and Ky*, that is, = 3, Ky* = 5, indicating that there are 3 target computing nodes in the x direction and 5 target computing nodes in the y direction.

[0039] The target computing node is as followsFigure 2 As shown, Figure 2 The processes framed by the black lines together constitute a computing node, Figure 2 The black bidirectional arrows in the figure represent the cross-node communication relationship between the computing nodes. For example, 90 processes are specified, that is, N is equal to 90, and 15 computing nodes are specified, each of which uses 6 processes. The 90 processes correspond to the numbers 0-89, and each rectangular frame is a target computing node. The first target computing node cu01 corresponds to process numbers 0-2 and 9-11; the second target computing node cu02 corresponds to process numbers 3-5 and 12-14; the third target computing node cu03 corresponds to process numbers 6-8 and 15-17; the fourth target computing node cu04 corresponds to process numbers 18-20 and 27-27; and so on. Each process will communicate with the processes around it. The processes in the same target computing node perform intra-node communication, and the processes not in the same target computing node perform cross-node communication. The black arrows in the figure represent communication between different target computing nodes. For example, the first target computing node communicates with the second target computing node and the fourth target computing node. The improved method only requires 3 target computing nodes to communicate with 4 adjacent target computing nodes, while the general method requires 11 target computing nodes to communicate with 4 adjacent target computing nodes. Therefore, it can be seen that the improved method greatly reduces the inter-node communication.

[0040] It should be noted that the method is not only applicable to the WRF model, but also applicable to other models that use structured grids and perform neighbor communication.

[0041] The method for determining a computing node based on a process provided by the embodiment first determines a candidate node array combination according to a first target number of computing nodes; judges the ratio value of the computing nodes and the processes in different directions in the candidate node array combination; determines a target node array combination and determines a target computing node by judging whether the ratio value is an integer. This method can reduce cross-node communication between different computing nodes by arranging the computing nodes in two dimensions, and more intra-node communication is used, thereby reducing the running time of the WRF model, improving the computing efficiency, and reducing the energy consumption required for WRF model calculation. The problem of using a one-dimensional sequential mapping method to determine the computing node to which an MPI process belongs, resulting in a large amount of cross-node communication between computing nodes and affecting the performance of the WRF model, is solved.

[0042] As an optional embodiment, the method further comprises:

[0043] If it is determined that there is no target node array combination in the candidate node array combination, the process number of the process is determined according to the first direction, the second direction, the first process number, and the second process number;

[0044] acquiring a second target number of processes, determining a third number of processes contained in the computing nodes according to the first target number and the second target number;

[0045] determining the target computing nodes according to the process number and the third number of processes.

[0046] Specifically, if it is determined that there is no target node combination in the candidate node combination, the first direction, the second direction, the first number of processes and the second number of processes are determined first, for example, the first direction is the x direction, the second direction is the y direction, the first number of processes in the first direction is Nx, and the second number of processes in the second direction is Ny.

[0047] determining the process number of processes according to the first direction, the second direction, the first number of processes and the second number of processes, for example, as shown in Figure 3 , Nx is 9 and Ny is 10, and it is determined that the process number includes 0-89, a total of 90 MPI processes.

[0048] acquiring a second target number of processes, for example, 90 processes are specified to be used, that is, the second target number N is equal to 90, and 15 computing nodes are specified to be used, that is, the first target number is 15. According to the first target number and the second target number, the third number of processes contained in the computing nodes is determined, for example, the third number of processes = the second target number ÷ the first target number, which represents how many processes are contained in each computing node, for example, 90 ÷ 15 = 6, that is, 6 processes are used in each computing node.

[0049] determining the target computing nodes according to the process number and the third number of processes, for example, processes with process numbers 0-5 are divided into a first target computing node, processes with process numbers 6-11 are divided into a second target computing node, processes with process numbers 12-17 are divided into a third target computing node, and so on. The determined target computing nodes are as shown in Figure 3 . Figure 3 In Figure 3 , the numbers are process numbers, that is, 90 processes correspond to numbers 0-89, Figure 3The black bidirectional arrows in the figure represent the cross-node communication relationship between the computing nodes. The first target computing node corresponds to process numbers 0-5; the second target computing node corresponds to process numbers 6-11; the third target computing node corresponds to process numbers 12-17, and so on. Each process communicates with its surrounding processes, the processes in the same target computing node communicate internally, and the processes not in the same target computing node communicate across nodes. The black arrows in the figure represent the communication between different target computing nodes, such as the first target computing node communicating with the second target computing node and the third target computing node. As can be seen from the figure, 11 of the target computing nodes need to communicate with 4 adjacent target computing nodes (corresponding to 4 black arrows), and the remaining 4 target computing nodes need to communicate with 2-3 adjacent nodes.

[0050] However, the target computing nodes determined by the above method will result in a large number of inter-node communications, thereby affecting the performance of the WRF model. Therefore, the target computing nodes are preferentially determined by the method of step S103, and only when no target node combination is determined, the embodiment is used.

[0051] In the embodiment of the present application, if it is determined that there is no target node combination in the candidate node combination, the target computing node is determined according to the process number and the third process number contained in the computing node. This enables the target computing node to be successfully determined in different situations, thereby improving the scope of application of the method.

[0052] As an optional embodiment, after the target computing node is determined, the method further includes:

[0053] determining the node number of the target computing node;

[0054] determining a fourth process number of processes contained in the target computing node in a first direction according to the first process number and a first node number of the target node combination;

[0055] determining a fifth process number of processes contained in the target computing node in a second direction according to the second process number and a second node number of the target node combination;

[0056] determining a mapping relationship between the processes and the target computing node according to the fourth process number, the fifth process number, the node number, the first node number, and the process number, wherein the mapping relationship is used to run the target model.

[0057] Specifically, the node number of the target computing node is determined, for example: the node number Num is the name of the target computing node, such as cu01, cu02, …, cu15.

[0058] According to the first process number and the first node number of the target node array combination, a fourth process number of processes contained by the target computing node in a first direction is determined, for example, the first process number is Nx, the first node number of the target node array combination is , and the fourth process number may be obtained by dividing Nx by . According to the second process number and the second node number of the target node array combination, a fifth process number of processes contained by the target computing node in a second direction is determined, for example, the second process number is Ny, the second node number of the target node array combination is Ky*, and the fifth process number may be obtained by dividing Ny by Ky*, then and are the number of processes in the x and y directions in each computing node, such as = 3, = 2.

[0059] According to the fourth process number , the fifth process number , the node number Num, the first node number , and the process number Ni, the mapping relationship between the process and the target computing node is determined, for example, for any process number Ni (starting from 0), according to the process number, the column number of the process in the x direction and the row number in the y direction are determined, that is, using Ni divided by Nx, the quotient is and the remainder is , if the division is exact, the remainder is equal to 0. Then according to and , the computing node name number Num (starting from 0) corresponding to the process number is obtained, so as to determine the mapping relationship of the computing node name corresponding to the process, such as formula (1).

[0060] (1)

[0061] Where / represents integer division.

[0062] The process number such as 0-89 can be substituted into formula (1) to obtain the node number Num of the target computing node corresponding to each process. The mapping relationship is used to run the target mode, and the mapping relationship is, for example, a two-dimensional mapping relationship. The target mode is, for example, the WRF mode.

[0063] In this embodiment, the mapping relationship between processes and target computing nodes is determined based on the fourth process number, the fifth process number, the node number, the first node number, and the program number, and the target mode is run through the mapping relationship. Furthermore, the node number of the target computing node corresponding to the process can be quickly determined based on the mapping relationship and the program number.

[0064] As an optional embodiment, after determining the mapping relationship between the process and the target computing node, the method further includes:

[0065] If a process to be mapped exists, determine the reference process number of the process to be mapped;

[0066] Based on the reference process number, determine the first number of the process to be mapped in the first direction and the second number in the second direction;

[0067] The first data item is obtained based on the first number, the fourth process number, and the first node number;

[0068] The second data item is obtained based on the second number and the fifth process number;

[0069] Based on the first data item, the second data item, and the mapping relationship, determine the target computing node corresponding to the process to be mapped.

[0070] Specifically, if a process to be mapped exists, the reference process number of the process to be mapped is determined, for example, the reference process number is any number between 0 and 89.

[0071] Based on the reference process number, determine the first number of the process to be mapped in the first direction and the second number in the second direction. For example, if the reference process number is 87, the corresponding first number is... It is 9, the second number. It is 6. According to the first number Fourth process number and the number of first nodes The first data item is obtained. / )× .

[0072] Based on the second number and the fifth process number, the second data item is obtained ( / ).

[0073] The mapping relationship is as follows: Formula (1), which maps the first data item ( / )× Second data item ( / Substitute into formula (1) to determine the target computing node Num corresponding to the process to be mapped.

[0074] As an optional embodiment, before obtaining the preset number of candidate node combinations according to the first target number of computing nodes, the method further comprises:

[0075] obtaining a second target number of processes;

[0076] determining a first candidate number of processes in a first direction and a second candidate number of processes in a second direction according to the second target number, wherein the product of the first candidate number and the corresponding second candidate number is the second target number;

[0077] determining a first difference between the first candidate number and the corresponding second candidate number;

[0078] determining a first process number and a second process number from the first candidate number and the second candidate number according to the first difference.

[0079] Specifically, the second target number of processes is obtained, for example: 90.

[0080] determining a first candidate number of processes in a first direction and a second candidate number of processes in a second direction according to the second target number, for example: the first direction is x and the second direction is y. For the values of Nx and Ny, the currently default automatic parallel division scheme of WRF is Nx=Ny=N1 / 2, if N cannot be an integer square root, Nx×Ny=N and the difference between Ny and Nx is the minimum combination, for example, if N is specified to be equal to 81, WRF automatically performs parallel division to make Nx=9 and Ny=9; if N is specified to be equal to 90, the first candidate number includes 1, 2, 3, 5, 6, 9, 10…, and the corresponding second candidate number includes 90, 45, 30, 18, 15, 10, 9…

[0081] determining a first difference between the first candidate number and the corresponding second candidate number, for example: 89, 43, 27, 13, 9, 1…, and determining a first process number and a second process number from the first candidate number and the second candidate number according to the first difference, for example: selecting a pair of first candidate number and second candidate number with the smallest first difference as the first process number and the second process number, for example: Nx=9 and Ny=10.

[0082] In the embodiments of the present application, the pair of first candidate number and second candidate number with the smallest difference is selected as the first process number and the second process number, which can reduce the cross-node communication between target computing nodes when mapping the processes to the computing nodes subsequently.

[0083] As an optional embodiment, obtaining the preset number of candidate node combinations according to the first target number of computing nodes comprises:

[0084] According to the first target number, a third candidate number of the computing nodes in the first direction and a fourth candidate number of the computing nodes in the second direction are determined, wherein the third candidate number is less than the fourth candidate number, and the product of the third candidate number and the fourth candidate number is the first target number.

[0085] According to the third candidate number and the fourth candidate number, a preset number of candidate node array combinations are obtained.

[0086] Specifically, the first target number is, for example, 15.

[0087] The number Kx of the computing nodes in the x direction and the number Ky of the computing nodes in the y direction are calculated, and the product of Kx and Ky needs to be equal to K, and Kx needs to be less than Ky, so as to obtain different (Kx, Ky) combinations. Therefore, according to the first target number, a third candidate number of the computing nodes in the first direction and a fourth candidate number of the computing nodes in the second direction are determined, for example, the third candidate number in the x direction includes 1 and 3, and the fourth candidate number in the y direction includes 15 and 5.

[0088] According to the third candidate number and the fourth candidate number, a preset number of candidate node array combinations (Kx, Ky) are obtained, for example, (3, 5) and (1, 15).

[0089] As an optional embodiment, a first ratio value of the first number of nodes and the first number of processes is determined, and a second ratio value of the second number of nodes and the second number of processes is determined, including:

[0090] A second difference value of the first number of nodes and the second number of nodes in the candidate node array combination is determined.

[0091] According to the second difference value, the candidate node array combination is sorted to obtain a combination sequence.

[0092] According to the arrangement order of the candidate node array combination in the combination sequence, the first ratio value of the first number of nodes and the first number of processes in the candidate node array combination is determined, and the second ratio value of the second number of nodes and the second number of processes is determined.

[0093] It is judged whether the first ratio value and the second ratio value are integers.

[0094] Specifically, the second difference value of the first number of nodes and the second number of nodes in the candidate node array combination is determined, for example, the candidate node array combination includes (3, 5) and (1, 15), and the second difference value is 2 and 14 respectively.

[0095] According to the second difference value, the candidate node array combination is sorted to obtain a combination sequence, for example, the candidate node array combination is arranged from small to large according to the second difference value, and the combination sequence is (3, 5) and (1, 15).

[0096] According to the arrangement order of the candidate node array combination in the combination sequence, a first ratio value of the first node number and the first process number in the candidate node array combination is determined, and a second ratio value of the second node number and the second process number is determined, for example: first calculate the first candidate node array combination in the combination sequence, Kx=3, Ky=5, Nx=9, Ny=10, then the first ratio value is 3 and the second ratio value is 2. If Nx can be divided by Kx and Ny can be divided by Ky, (3, 5) can be directly set as the target node array combination.

[0097] If the first ratio value and the second ratio value are integers, for example: the first ratio value is 3 and the second ratio value is 2, if Nx can be divided by Kx and Ny can be divided by Ky, (3, 5) can be directly set as the target node array combination.

[0098] In the embodiment of the application, the candidate node array combinations are sorted according to the second difference value, the first ratio value and the second ratio value are determined according to the arrangement order of the candidate node array combination in the combination sequence, whether the node number in each candidate node array combination meets the requirement of being divided by the corresponding process number is judged in turn, the judgment process is ended after the target node array combination is determined, the efficiency of determining the target node array combination is improved, and the calculation amount is reduced.

[0099] As an optional embodiment, the target computing node corresponding to the to-be-mapped process is determined according to the first data item, the second data item and the mapping relationship, including:

[0100] The first data item and the second data item are substituted into the mapping relationship to obtain the target computing node number corresponding to the to-be-mapped process;

[0101] The mapping relationship is obtained by the following formula: , wherein, represents the second number, represents the first number, represents the fifth process number, represents the fourth process number, represents the first node number of the target node array combination, represents the first data item, represents the second data item, represents the target computing node number, and / represents integer division.

[0102] The target computing node corresponding to the to-be-mapped process is determined according to the target computing node number corresponding to the to-be-mapped process.

[0103] Specifically, the mapping relationship is determined by the formula The first data item and the second data item Substitute the mapping relationship, get the target computing node number corresponding to the process to be mapped . The target computing node number is, for example, cu01, cu02, …, cu15.

[0104] According to the target computing node number corresponding to the process to be mapped, determine the target computing node corresponding to the process to be mapped, for example, if the target computing node number is cu01, the target computing node corresponding to the process to be mapped is Figure 2 The computing node in the lower left corner of the figure, containing processes with serial numbers 0, 1, 2, 9, 10, 11.

[0105] In the embodiments of the present application, the mapping relationship is determined by the formula, and the target computing node corresponding to the process is quickly determined by the mapping relationship, the first data item and the second data item.

[0106] As an optional embodiment, after determining the mapping relationship between the process and the target computing node, the method further comprises:

[0107] According to the mapping relationship, determine the target process contained in the target computing node;

[0108] Obtain the thread parameters of the data processing thread in the target computing node;

[0109] According to the thread parameters, create a data transmission thread;

[0110] According to the target process, obtain the data to be transmitted by the data processing thread and the target thread corresponding to the data to be transmitted;

[0111] Use the data transmission thread to transmit the data to be transmitted to the target thread.

[0112] Specifically, according to the mapping relationship, determine the target process contained in the target computing node, for example, the target computing node cu01 contains processes with serial numbers 0, 1, 2, 9, 10, 11, which are target processes of the target computing node cu01. The target process is an MPI process.

[0113] In addition to containing target processes, the target computing node will also create a number of OpenMP threads, which will be used as data processing threads. MPI processes and OpenMP threads can run in a serial-parallel mode, which can ensure that a group of OpenMP threads can implement data transmission with other groups of OpenMP threads through MPI processes. Data processing threads can run in parallel and are used to implement data processing functions.

[0114] When the data processing thread needs to transmit data to other threads, the data processing thread can send the to-be-transmitted data to be sent to a target process, and the data processing thread enters a waiting state, and the target process sends the to-be-transmitted data to a target thread. It can be seen that when the target process transmits data, the data processing thread is in a waiting state, and the resources on the computing node are occupied, which causes resource waste.

[0115] Based on the above, in order to improve the resource utilization rate in the target computing node, in the case that the data processing thread is in a waiting state, a data transmission thread is created to occupy the resources occupied by the data processing thread for data transmission.

[0116] Thread parameters of the data processing thread in the target computing node are obtained, for example, the number of threads of the data processing thread, processor information occupied by the data processing thread, and the like. The processor information is used to identify a processor. For example, the resource affinity of the data processing thread is used to represent the processor occupied by the data processing thread.

[0117] According to the thread parameters, a data transmission thread is created, for example, OpenMP threads are created in the same number as the data processing threads, and the OpenMP threads are used as data transmission threads, so that the data transmission threads correspond to the data processing threads one by one, and the corresponding data transmission threads and data processing threads occupy the same processor. In addition, when the data processing thread is an OpenMP thread, the OpenMP thread can be set to a passive mode, and the OpenMP thread in the passive mode allows other threads to occupy the processor of the OpenMP thread. Similarly, the data transmission thread can also be configured to support processor sharing, so as to give up the processor and allow other threads to use it. The data transmission threads can run in parallel and transmit data at the same time.

[0118] According to the target process, the to-be-transmitted data of the data processing thread and the target thread corresponding to the to-be-transmitted data are obtained. The target thread can be another group of data processing threads in the same target computing node, or data processing threads in another computing node.

[0119] The to-be-transmitted data is transmitted to the target thread by using the data transmission thread, for example, the target thread divides the to-be-transmitted data into a plurality of sub-data, the number of sub-data is the same as the number of data transmission threads, and the data transmission threads run in parallel to transmit the sub-data to the target thread. In addition, when data transmission is not needed, the target process can control the data processing thread to enter a sleep state.

[0120] In the embodiment of the present application, the data transmission thread can occupy the resources of the data processing thread to realize data transmission, so that additional resources do not need to be allocated for the data transmission thread, and the resource utilization rate of the computing node is effectively improved. In addition, multiple data transmission threads can simultaneously transmit data, and compared with a single running MPI process, the data transmission efficiency is higher, and the data transmission can be completed in a shorter time.

[0121] As an optional embodiment, another method for determining the mapping relationship between the processes and the computing nodes and binding the processes and the computing nodes, the specific process includes steps A1 to A4.

[0122] In step A1, the different processes communicate with each other using the message passing interface (MPI), collect the communication affinity information between the MPI processes of the parallel simulation program in the WRF model, and record the information into a log file. The communication affinity between the MPI processes is sorted in the log file.

[0123] In step A2, a communication overhead model Z of the parallel simulation program is defined, as shown in formula (2). For the communication affinity of the processes and the processes , the communication distance between the computing units where the processes and the processes are located is represented, and the total communication overhead Z of the parallel simulation program under the process mapping is calculated.

[0124] (2)

[0125] In step A3, the hybrid parallel genetic algorithm is used to solve the optimal process mapping strategy according to the communication affinity and the communication distance between the computing units. The process mapping scheme of the parallel simulation program in the WRF model is defined as an individual, and the individuals in the population are selected, crossed, mutated, and simulated annealing in an iterative manner to generate a process mapping scheme that minimizes the communication overhead Z.

[0126] In step A4, the mapping relationship between the MPI processes and the computing nodes is determined according to the process mapping scheme obtained in step A3, the MPI processes are statically bound to the specified computing nodes, and the parallel simulation program in the WRF model is re-run.

[0127] In the embodiment of the present application, the solving quality and solving speed of the mapping algorithm for solving the process mapping strategy are optimized, a high-quality process mapping scheme can be solved in a short time, the communication overhead of the parallel simulation program is effectively reduced, and the effect of communication optimization is realized.

[0128] As an optional embodiment,​​Figure 4 is a flow chart of an MPI process and computing node mapping method for a WRF model, as shown in Figure 4 includes the following steps:

[0129] determining the number of MPI processes N and the number of computing nodes K used for running the WRF model, which need to satisfy that N is divisible by K and a name list of the computing nodes to be used; determining the number of processes Nx in the x direction and the number of processes Ny in the y direction; calculating the number of computing nodes Kx in the x direction and the number of computing nodes Ky in the y direction, which need to satisfy that the product of Kx and Ky is equal to K and Kx is smaller than Ky, so as to obtain different combinations; arranging the different combinations in ascending order according to the difference between Ky and Kx; traversing each (Kx, Ky) combination in turn to check whether the number of processes Nx in the x direction is divisible by the number of computing nodes Kx in the x direction and the number of processes Ny in the y direction is divisible by the number of computing nodes Ky in the y direction. If both are divisible, then Kx and Ky in the current combination are respectively assigned to Kx* and Ky*; the number of processes Nx in the x direction and the number of processes Ny in the y direction, the number of computing nodes Kx* in the x direction and the number of computing nodes Ky* in the y direction, and the name list of the computing nodes to be used are all determined, then the number Mi of processes in the x direction within each computing node and the number Mj of processes in the y direction within each computing node are determined; for any process number Ni, first determine its column and row numbers Nii and Njj, i.e., divide Ni by Nx to obtain quotient Njj and remainder Nii, if divisible, the remainder Nii is equal to 0; then obtain the computing node name number Num corresponding to the process number according to Nii and Njj, so as to determine the mapping relationship of the process corresponding to the computing node name; obtain the two-dimensional mapping of the MPI process and the computing node, and run the WRF model according to the two-dimensional mapping relationship. If both are not divisible, determine that there is no combination that satisfies the condition; if there is no combination that satisfies the condition, then use one-dimensional sequential MPI process and computing node mapping to run the WRF model.

[0130] In the embodiments of the present application, the computing nodes are arranged in two dimensions, so that the number of computing nodes Kx in the x direction and the number of computing nodes Ky in the y direction are as consistent as possible, and the difference between the number of computing nodes Ky in the y direction and the number of computing nodes Kx in the x direction is as small as possible. Compared with the general one-dimensional sequential mapping method, this mapping method can significantly reduce the cross-node communication between different computing nodes, and more use of node internal communication, thereby reducing the running time of the WRF model, improving the efficiency, and further reducing the energy consumption required for a single weather forecast, saving electricity, etc.

[0131] An apparatus for determining computing nodes based on processes is also provided in the embodiments, which is configured to implement the above-mentioned embodiments and preferred implementation manners, and details of which have been described above. As used below, the term "module" can be a combination of software and / or hardware that implements a predetermined function. Although the apparatus described in the following embodiments is preferably implemented in software, implementation in hardware, or a combination of software and hardware, is also possible and contemplated.

[0132] The embodiments provide an apparatus for determining computing nodes based on processes, which comprises: Figure 5 as shown in the figure, comprising:

[0133] The combination determining module 501 is configured to, in a case where the first number of processes in the first direction and the second number of processes in the second direction are determined, obtain a preset number of candidate node array combinations according to the first target number of computing nodes, wherein the candidate node array combination contains the first number of nodes in the first direction and the second number of nodes in the second direction, and the product of the first number of nodes and the second number of nodes is the first target number;

[0134] The ratio value determining module 502 is configured to determine a first ratio value of the first number of nodes and the first number of processes, and determine a second ratio value of the second number of nodes and the second number of processes;

[0135] The node determining module 503 is configured to, if it is determined that there is a target node array combination in the candidate node array combination, determine the target computing node according to the target node array combination, wherein the first ratio value and the second ratio value corresponding to the target node array combination are both integers.

[0136] In some optional implementation manners, the apparatus further comprises:

[0137] The first determining module is configured to, if it is determined that there is no target node array combination in the candidate node array combination, determine a process number of the processes according to the first direction, the second direction, the first number of processes and the second number of processes;

[0138] The first obtaining module is configured to obtain a second target number of processes, and determine a third number of processes contained in the computing nodes according to the first target number and the second target number;

[0139] The second determining module is configured to determine the target computing node according to the process number and the third number of processes.

[0140] In some optional implementation manners, the apparatus further comprises:

[0141] The third determining module is configured to determine a node number of the target computing node;

[0142] a fourth determining module, configured to determine a fourth number of processes included in the target computing node in the first direction according to the first number of processes and a first number of nodes in the target node array;

[0143] a fifth determining module, configured to determine a fifth number of processes included in the target computing node in the second direction according to the second number of processes and a second number of nodes in the target node array;

[0144] a sixth determining module, configured to determine a mapping relationship between the processes and the target computing node according to the fourth number of processes, the fifth number of processes, the node number, the first number of nodes and the process number, wherein the mapping relationship is used to run the target mode.

[0145] In some optional embodiments, the apparatus further includes:

[0146] a seventh determining module, configured to determine a reference process number of a process to be mapped in a case where the process to be mapped exists;

[0147] an eighth determining module, configured to determine a first number in the first direction and a second number in the second direction of the process to be mapped according to the reference process number;

[0148] a first calculating module, configured to obtain a first data item according to the first number, the fourth number of processes and the first number of nodes;

[0149] a second calculating module, configured to obtain a second data item according to the second number and the fifth number of processes;

[0150] a ninth determining module, configured to determine a target computing node corresponding to the process to be mapped according to the first data item, the second data item and the mapping relationship.

[0151] In some optional embodiments, the apparatus further includes:

[0152] a second obtaining module, configured to obtain a second target number of processes;

[0153] a tenth determining module, configured to determine a first candidate number of processes in the first direction and a second candidate number of processes in the second direction according to the second target number, wherein a product of the first candidate number and the corresponding second candidate number is the second target number;

[0154] an eleventh determining module, configured to determine a first difference value between the first candidate number and the corresponding second candidate number;

[0155] a twelfth determining module, configured to determine the first number of processes and the second number of processes from the first candidate number and the second candidate number according to the first difference value.

[0156] In some optional embodiments, the combination determining module 501 includes:

[0157] The first determining unit is configured to determine, according to the first target number, a third candidate number of the computing nodes in the first direction and a fourth candidate number of the computing nodes in the second direction, wherein the third candidate number is less than the fourth candidate number, and the product of the third candidate number and the fourth candidate number is the first target number.

[0158] The computing unit is configured to obtain, according to the third candidate number and the fourth candidate number, a preset number of candidate node array combinations.

[0159] In some optional embodiments, the proportion value determining module 502 includes:

[0160] The second determining unit is configured to determine a second difference value of the first node number and the second node number in the candidate node array combination.

[0161] The sorting unit is configured to sort the candidate node array combination according to the second difference value, to obtain a combination sequence.

[0162] The third determining unit is configured to determine, according to the arrangement order of the candidate node array combination in the combination sequence, a first proportion value of the first node number and the first process number in the candidate node array combination, and determine a second proportion value of the second node number and the second process number.

[0163] The judging unit is configured to judge whether the first proportion value and the second proportion value are integers.

[0164] In some optional embodiments, the ninth determining module includes:

[0165] The data processing unit is configured to substitute the first data item and the second data item into a mapping relationship to obtain a target computing node number corresponding to the to-be-mapped process, wherein the mapping relationship is obtained by the following formula: wherein, represents the second number, represents the first number, represents the fifth process number, represents the fourth process number, represents the first node number of the target node array combination, represents the first data item, represents the second data item, represents the target computing node number, and / represents integer division.

[0166] The fourth determining unit is configured to determine, according to the target computing node number corresponding to the to-be-mapped process, a target computing node corresponding to the to-be-mapped process.

[0167] In some optional embodiments, the apparatus further includes:

[0168] The thirteenth determination module is used to determine the target process contained in the target computing node based on the mapping relationship;

[0169] The third acquisition module is used to acquire the thread parameters of the data processing thread in the target computing node;

[0170] Create a module to create a data transmission thread based on thread parameters;

[0171] The fourth acquisition module is used to acquire the data to be transmitted from the data processing thread and the target thread corresponding to the data to be transmitted, based on the target process.

[0172] The data transmission module is used to transmit data to the target thread using the data transmission thread.

[0173] Further functional descriptions of the above modules and units are the same as those in the corresponding embodiments described above, and will not be repeated here.

[0174] In this embodiment, the device for determining computing nodes based on processes is presented in the form of functional units. Here, a unit refers to an ASIC (Application Specific Integrated Circuit) circuit, a processor and memory that execute one or more software or fixed programs, and / or other devices that can provide the above functions.

[0175] This application also provides a computer device having the above-described features. Figure 5 The apparatus shown is for determining computing nodes based on processes.

[0176] Please see Figure 6 , Figure 6 This is a schematic diagram of the structure of a computer device provided in an optional embodiment of this application, such as... Figure 6 As shown, the computer device includes one or more processors 10, memory 20, and interfaces for connecting the components, including high-speed interfaces and low-speed interfaces. The components communicate with each other via different buses and can be mounted on a common motherboard or otherwise installed as needed. The processors can process instructions executed within the computer device, including instructions stored in or on memory to display graphical information of a GUI on external input / output devices (such as display devices coupled to the interfaces). In some alternative implementations, multiple processors and / or multiple buses can be used with multiple memories and multiple memory modules, if desired. Similarly, multiple computer devices can be connected, each providing some of the necessary operations (e.g., as a server array, a group of blade servers, or a multiprocessor system). Figure 6 Take a processor 10 as an example.

[0177] The processor 10 can be a central processor, a network processor, or a combination thereof. The processor 10 can further include an integrated circuit, a programmable logic device, or a combination thereof. The programmable logic device can be a complex programmable logic device, a field programmable logic gate array, a programmable array logic, or any combination thereof.

[0178] The memory 20 stores instructions executable by the at least one processor 10 to cause the at least one processor 10 to perform the methods illustrated by the above embodiments.

[0179] The memory 20 can include a program storage area and a data storage area. The program storage area can store an operating system, application programs required by at least one function, and the like. The data storage area can store data created according to the use of the computer device, and the like. In addition, the memory 20 can include a high-speed random access memory, and can further include a non-transitory memory such as at least one disk storage device, a flash memory device, or other non-transitory solid-state memory device. In some alternative embodiments, the memory 20 can optionally include a memory disposed remotely from the processor 10, and these remote memories can be connected to the computer device through a network. Examples of the network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and a combination thereof.

[0180] The memory 20 can include a volatile memory such as a random access memory, and can also include a non-volatile memory such as a flash memory, a hard disk, or a solid state disk. The memory 20 can further include a combination of the above-mentioned types of memories.

[0181] The computer device further includes a communication interface 30 for communication of the computer device with other devices or communication networks.

[0182] The embodiments of the present application also provide a computer readable storage medium. The above-mentioned methods according to the embodiments of the present application can be implemented in hardware, firmware, or recorded in a storage medium, or stored in a remote storage medium or a non-transitory machine readable storage medium by original computer code downloaded through a network and stored in a local storage medium, so that the methods described herein can be processed by such software on a storage medium using a general-purpose computer, a special-purpose processor, or programmable or special-purpose hardware. The storage medium can be a magnetic disk, an optical disk, a read-only memory, a random access memory, a flash memory, a hard disk, or a solid state disk, etc. Further, the storage medium can further include a combination of the above-mentioned types of memories. It can be understood that the computer, the processor, the microprocessor controller, or the programmable hardware includes a storage component that can store or receive software or computer code, which, when accessed and executed by the computer, the processor, or the hardware, implements the methods illustrated by the above embodiments.

[0183] Part of the present application can be applied as a computer program product, for example, computer program instructions, when executed by a computer, through the operation of the computer, the method and / or technical solutions according to the present application can be called or provided. Those skilled in the art should understand that the form of computer program instructions in computer readable medium includes but is not limited to source file, executable file, installation package file and the like, and accordingly, the way of computer program instructions executed by computer includes but is not limited to: the computer directly executes the instructions, or the computer compiles the instructions and then executes the corresponding compiled program, or the computer reads and executes the instructions, or the computer reads and installs the instructions and then executes the corresponding installed program. Here, the computer readable medium can be any available computer readable storage medium or communication medium accessible to the computer.

[0184] Although the embodiments of the present application are described in conjunction with the drawings, various modifications and changes can be made by those skilled in the art without departing from the spirit and scope of the present application, and such modifications and changes fall within the scope defined by the present application.

Claims

1. A method for determining a computing node based on a process, characterized in that, The method comprises: In the case of determining the first process number in the first direction and the second process number in the second direction, obtaining a preset number of candidate node combinations according to the first target number of the computing nodes, wherein the candidate node combination contains the first node number of the computing nodes in the first direction and the second node number of the computing nodes in the second direction, and the product of the first node number and the second node number is the first target number; determining the first ratio value of the first node number and the first process number, and determining the second ratio value of the second node number and the second process number; If it is determined that there is a target node combination in the candidate node combination, determine the target computing node according to the target node combination, wherein the first ratio value and the second ratio value corresponding to the target node combination are integers; determine the node number of the target computing node; determine the fourth process number of the process contained in the target computing node in the first direction according to the first process number and the first node number of the target node combination; determine the fifth process number of the process contained in the target computing node in the second direction according to the second process number and the second node number of the target node combination; determine the mapping relationship between the process and the target computing node according to the fourth process number, the fifth process number, the node number, the first node number and the process number, wherein the mapping relationship is used to run the target mode; According to the mapping relationship, determine the target process contained in the target computing node; obtain the thread parameter of the data processing thread in the target computing node; create a data transmission thread according to the thread parameter; according to the target process, obtain the to-be-transmitted data of the data processing thread and the target thread corresponding to the to-be-transmitted data; use the data transmission thread to transmit the to-be-transmitted data to the target thread.

2. The method of claim 1, wherein, The method further comprises: If it is determined that there is no target node combination in the candidate node combination, determine the process number according to the first direction, the second direction, the first process number and the second process number; obtain the second target number of the process, and determine the third process number of the process contained in the computing node according to the first target number and the second target number; determine the target computing node according to the process number and the third process number.

3. The method of claim 1, wherein, After determining the mapping relationship between the process and the target computing node, the method further comprises: In the case of existing to-be-mapped processes, determine the reference process number of the to-be-mapped process; determine the first number in the first direction and the second number in the second direction of the to-be-mapped process according to the reference process number; obtain the first data item according to the first number, the fourth process number and the first node number; obtain the second data item according to the second number and the fifth process number; According to the first data item, the second data item and the mapping relationship, a target computing node corresponding to the process to be mapped is determined.

4. The method of claim 1, wherein, Before the obtaining of the preset number of candidate node combinations according to the first target number of computing nodes, the method further comprises: obtaining a second target number of processes; determining a first candidate number of the processes in the first direction and a second candidate number of the processes in the second direction according to the second target number, wherein the product of the first candidate number and the corresponding second candidate number is the second target number; determining a first difference value between the first candidate number and the corresponding second candidate number; determining the first process number and the second process number from the first candidate number and the second candidate number according to the first difference value.

5. The method of claim 1, wherein, The obtaining of the preset number of candidate node combinations according to the first target number of computing nodes comprises: determining a third candidate number of the computing nodes in the first direction and a fourth candidate number of the computing nodes in the second direction according to the first target number, wherein the third candidate number is less than the fourth candidate number, and the product of the third candidate number and the fourth candidate number is the first target number; obtaining the preset number of candidate node combinations according to the third candidate number and the fourth candidate number.

6. The method according to claim 1 or 5, characterized in that, The determining of the first ratio value of the first node number and the first process number and the determining of the second ratio value of the second node number and the second process number comprise: determining a second difference value between the first node number and the second node number in the candidate node combination; sorting the candidate node combinations according to the second difference value to obtain a combination sequence; determining the first ratio value of the first node number and the first process number and the second ratio value of the second node number and the second process number according to the arrangement order of the candidate node combinations in the combination sequence; judging whether the first ratio value and the second ratio value are integers.

7. The method of claim 3, wherein, The determining of the target computing node corresponding to the process to be mapped according to the first data item, the second data item and the mapping relationship comprises: substituting the first data item and the second data item into the mapping relationship to obtain a target computing node number corresponding to the process to be mapped; The mapping relationship is obtained by the following formula: wherein, denotes the second number, denotes the first number, denotes the fifth process number, denotes the fourth process number, denotes the first node number of the target node array combination, denotes the first data item, denotes the second data item, denotes the target computing node number, and / denotes integer division; determining the target computing node corresponding to the process to be mapped according to the target computing node number corresponding to the process to be mapped.

8. A computer device, comprising: comprise: a memory and a processor, which are communicatively connected, the memory stores computer instructions, and the processor executes the computer instructions to perform the method for determining a computing node based on a process according to any one of claims 1 to 7.

Images