Method for determining computing node based on process and computer equipment

Abstract

The invention discloses a process-based computing node determining method and computer equipment, and relates to the technical field of high-performance computing, and the process-based computing node determining method comprises the following steps: determining a candidate node number combination according to a first target number of computing nodes; judging the proportion values of the computational nodes and the processes in different directions in the candidate node number combination; and determining a target node number combination by judging whether the proportion value is an integer or not, and determining a target computing node. The problem that a large amount of cross-node communication exists between computing nodes and the performance of a WRF mode is affected due to the fact that the computing nodes to which the MPI process belongs are determined in a one-dimensional sequential mapping mode can be solved. According to the method, the computing nodes are arranged in a two-dimensional manner, so that the cross-node communication among different computing nodes can be reduced, and the internal communication of the nodes can be more used, thereby reducing the running time of the WRF mode, improving the computing efficiency and reducing the energy consumption required by the calculation of the WRF mode.

Description

Technical Field

[0001] This invention relates to the field of high-performance computing technology, and more specifically to a method and computer device for determining computing nodes based on processes. Background Technology

[0002] WRF (Weather Research and Forecasting Model) is one of the most commonly used numerical models in the meteorological and oceanographic fields. Currently, the parallel partitioning scheme used in WRF models divides the forecast area into N sub-regions, with each MPI (Message Passing Interface) process responsible for one sub-region. Assuming there are Nx processes in the x-direction and Ny processes in the y-direction, the number of processes N is equal to the product of Nx and Ny. After determining the number of processes in the x and y directions, it is necessary to determine the mapping relationship between MPI processes and computing nodes. This mapping relationship determines which MPI processes each computing node uses.

[0003] Currently, a one-dimensional sequential mapping method is used to determine the mapping relationship between MPI processes and compute nodes. After determining the number of processes used by each compute node, the compute nodes are arranged sequentially. Using a one-dimensional sequential mapping method to determine the compute nodes and the MPI processes contained in the compute nodes will result in a large number of inter-node communication requirements between compute nodes, which will affect the performance of WRF mode.

[0004] Therefore, the related technologies use a one-dimensional sequential mapping method to determine the computing node to which the MPI process belongs, resulting in a large amount of cross-node communication between computing nodes, which affects the performance of WRF mode. Summary of the Invention

[0005] In view of this, this application provides a method and computer device for determining computing nodes based on processes, in order to solve the problem that using a one-dimensional sequential mapping method to determine the computing node to which an MPI process belongs results in a large amount of cross-node communication between computing nodes, which affects the performance of WRF mode.

[0006] In a first aspect, this application provides a method for determining computing nodes based on processes, the method comprising: Given the number of first processes in the first direction and the number of second processes in the second direction, a preset number of candidate node number combinations are obtained based on the first target number of computing nodes. The candidate node number combination includes the number of first nodes in the first direction and the number of second nodes in the second direction, and the product of the number of first nodes and the number of second nodes is the first target number. Determine a first ratio of the number of first nodes to the number of first processes, and determine a second ratio of the number of second nodes to the number of second processes; If a target node combination is found among the candidate node combinations, the target computing node is determined based on the target node combination, wherein the first and second ratio values ​​corresponding to the target node combination are both integers.

[0007] Secondly, this application provides an apparatus for determining computing nodes based on processes, the apparatus comprising: The combination determination module is used to obtain a preset number of candidate node number combinations based on the first target number of computing nodes, given a first process number in the first direction and a second process number in the second direction. The candidate node number combination includes the first node number of computing nodes in the first direction and the second node number of computing nodes in the second direction, and the product of the first node number and the second node number is the first target number. The ratio determination module is used to determine a first ratio value between the number of first nodes and the number of first processes, and to determine a second ratio value between the number of second nodes and the number of second processes; The node determination module is used to determine the target computing node based on the target node combination if a target node combination exists among the candidate node number combinations, wherein the first ratio value and the second ratio value corresponding to the target node number combination are both integers.

[0008] Thirdly, this application provides a computer device, including: a memory and a processor, which are communicatively connected to each other. The memory stores computer instructions, and the processor executes the computer instructions to perform the process-based method for determining computing nodes described in the first aspect or any corresponding embodiment.

[0009] Fourthly, this application provides a computer-readable storage medium storing computer instructions for causing a computer to execute the process-based method for determining computing nodes described in the first aspect or any corresponding embodiment.

[0010] Fifthly, this application provides a computer program product, including computer instructions for causing a computer to execute the process-based method for determining computing nodes described in the first aspect or any corresponding embodiment.

[0011] This application addresses the problem that using a one-dimensional sequential mapping method to determine the computing node to which an MPI process belongs leads to extensive cross-node communication, impacting WRF mode performance. This method, by arranging computing nodes in a two-dimensional manner, reduces cross-node communication and utilizes more intra-node communication, thereby reducing WRF mode runtime, improving computational efficiency, and lowering energy consumption. The method first determines candidate node combinations based on the initial target number of computing nodes; then, it determines the ratio of computing nodes to processes in different directions within each candidate node combination; finally, it determines the target node combination and the target computing node by checking if the ratio is an integer. Attached Figure Description

[0012] To more clearly illustrate the technical solutions in the specific embodiments or related technologies of this application, the drawings used in the description of the specific embodiments or related technologies will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0013] Figure 1 This is a flowchart illustrating a method for determining computing nodes based on processes according to an embodiment of this application; Figure 2 This is a schematic diagram illustrating the determination of computing nodes using a two-dimensional arrangement according to an embodiment of this application; Figure 3 This is a schematic diagram illustrating the determination of computing nodes using a one-dimensional sequential mapping method according to an embodiment of this application; Figure 4 This is a flowchart of an MPI process and compute node mapping method for WRF mode according to an embodiment of this application; Figure 5 This is a structural block diagram of an apparatus for determining computing nodes based on processes according to an embodiment of this application; Figure 6 This is a schematic diagram of the hardware structure of a computer device according to an embodiment of this application. Detailed Implementation

[0014] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the protection scope of this application.

[0015] It should be noted that, in the description of this application, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. The terms "first," "second," etc., in this application are used to distinguish similar objects and are not used to describe a specific order or sequence.

[0016] To enable those skilled in the art to better understand the present application, the present application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0017] WRF (Weather Forecasting) is a commonly used numerical weather and climate forecasting model in meteorology, and one of the main numerical weather forecasting models used in operational meteorological forecasting. Each meteorological operational center runs the WRF model on high-performance computers using a multi-node parallel approach to output weather forecast results, such as temperature and precipitation. This model can run using pure MPI processes, pure OpenMP (open multi-processing, shared memory parallel programming) threads, or a combination of MPI processes and OpenMP threads. The most common approach is MPI processes + OpenMP threads, which supports cross-node parallelism and allows communication using shared memory. When using the MPI processes + OpenMP threads approach, the parallel partitioning scheme for WRF needs to be determined. For example, the forecast area can be divided into N sub-regions using the number of processes N, with each process responsible for one sub-region. Assume there are Nx processes in the x-direction and Ny processes in the y-direction, where N is equal to the product of Nx and Ny. Currently, the default parallel partitioning scheme automatically executed by WRF for the values ​​of Nx and Ny is Nx = Ny = N1 / 2. After determining the number of processes in the x and y directions, it is necessary to determine the mapping between MPI processes and compute nodes. A commonly used one-dimensional sequential mapping method is to determine the number of processes on each compute node and then arrange the compute nodes sequentially.

[0018] Based on the above, this application provides a method for determining computing nodes based on processes. This invention proposes an improved MPI process-to-computing node mapping method, which arranges computing nodes in a two-dimensional arrangement such that the number of computing nodes Kx and Ky in the x and y directions are as consistent as possible, and the difference between the number of computing nodes Ky in the y direction and the number of computing nodes Kx in the x direction is minimized. Compared with the commonly used one-dimensional sequential mapping method, this mapping method can significantly reduce cross-node communication between different computing nodes and utilize more intra-node communication, thereby reducing the runtime of WRF mode, improving efficiency, and ultimately reducing the energy consumption required for a single weather forecast, thus saving electricity.

[0019] According to the embodiments of this application, an embodiment of determining computing nodes based on processes is provided. It should be noted that the steps shown in the flowcharts of the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, for example, a computer, a server, etc. Furthermore, although a logical order is shown in the flowcharts, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0020] This embodiment provides a method for determining computing nodes based on processes. Figure 1 This is a flowchart of a method for determining computing nodes based on processes according to an embodiment of this application, such as... Figure 1 As shown, the process includes the following steps: Step S101: Given the first number of processes in the first direction and the second number of processes in the second direction, a preset number of candidate node number combinations are obtained based on the first target number of computing nodes. The candidate node number combination includes the first number of computing nodes in the first direction and the second number of computing nodes in the second direction. The product of the first number of nodes and the second number of nodes is the first target number.

[0021] Specifically, such as Figure 2 As shown, the first direction is, for example, the x-direction, and the second direction is, for example, the y-direction. The forecast area is divided into N sub-regions using the number of processes N, with each MPI process responsible for one sub-region. The number of MPI processes N and the first target number K of computing nodes used in WRF mode must be determined, ensuring that N is divisible by K. For example, specifying 90 MPI processes (N equals 90) and 15 computing nodes, each using 6 processes. Based on the default parallel partitioning scheme automatically executed by WRF mode, determine the number of MPI processes Nx in the x-direction (the first process in the first direction) and the number of MPI processes Ny in the y-direction (the second process in the second direction). For example, determine that the number of MPI processes Nx in the x-direction is 9 and the number of MPI processes Ny in the y-direction is 10.

[0022] Given a determined number of first processes in the first direction and a determined number of second processes in the second direction, a preset number of candidate node combinations (Kx, Ky) are obtained based on the first target number of computing nodes. Each candidate node combination includes the first node number Kx in the first direction and the second node number Ky in the second direction. The preset number represents multiple combinations; no specific limit is imposed here. For example, if the first target number is 15, the candidate node combinations include (3, 5) and (1, 15). In (3, 5), the first node number is 3, and the second node number is 5; their product is 15, which is the same as the first target number.

[0023] Step S102: Determine the first ratio of the number of first nodes to the number of first processes, and determine the second ratio of the number of second nodes to the number of second processes.

[0024] Specifically, iterate through each candidate node combination (Kx, Ky) sequentially, checking if the number of MPI processes Nx in the first direction is divisible by the number of the first node Kx in the first direction, and simultaneously if the number of MPI processes Ny in the second direction is divisible by the number of the second node Ky in the second direction. We can first determine the first ratio of the number of first nodes to the number of first processes, and then determine the second ratio of the number of second nodes to the number of second processes. We then check if the first and second ratios are integers to determine if they are divisible. For example: Nx=9, Ny=10, and the candidate node combination is (3, 5), then Kx=3, Ky=5, the first ratio is 3, and the second ratio is 2. Therefore, Nx is divisible by Kx, and Ny is divisible by Ky.

[0025] Step S103: If a target node combination is found among the candidate node combination combinations, the target computing node is determined according to the target node combination, wherein the first ratio value and the second ratio value corresponding to the target node combination are both integers.

[0026] Specifically, if there exists a candidate node combination where both the first and second ratio values ​​are integers, this combination is used as the target node combination. If the number of target node combinations is not 1, the difference between the first and second node numbers in the target node combinations is determined, and the target node combination with the smallest difference is selected as the target computing node. For example, if the candidate node combination (3,5) has a first ratio value of 3 and a second ratio value of 2, both of which are integers, then the candidate node combination (3,5) is used as the target node combination. The values ​​of Kx and Ky in the target node combination (3,5) are then assigned to... and Ky*, i.e. =3, Ky*=5, indicating that there are 3 target computation nodes in the x-direction and 5 target computation nodes in the y-direction.

[0027] Target computing nodes such as Figure 2As shown, Figure 2 The processes outlined in the black box together constitute a computing node. Figure 2 The black double-headed arrows in the diagram represent cross-node communication relationships between compute nodes. Taking the example of using 90 processes (N = 90) and 15 compute nodes, with each node using 6 processes, as an example: The 90 processes correspond to numbers 0-89, and each rectangle represents a target compute node. The first target compute node cu01 corresponds to process numbers 0-2 and 9-11; the second target compute node cu02 corresponds to process numbers 3-5 and 12-14; the third target compute node cu03 corresponds to process numbers 6-8 and 15-17; the fourth target compute node cu04 corresponds to process numbers 18-20 and 27-27, and so on. Each process communicates with its surrounding processes. Processes within the same target compute node communicate intra-node, while processes outside the same target compute node communicate across nodes. The black arrows in the diagram represent communication between different target compute nodes; for example, the first target compute node communicates with the second and fourth target compute nodes. The improved method requires only 3 target computing nodes to communicate with their 4 neighboring target computing nodes, while the commonly used method requires 11 target computing nodes to communicate with their 4 neighboring target computing nodes. This shows that the improved method significantly reduces inter-node communication.

[0028] It should be noted that this method is applicable not only to WRF mode, but also to other modes that use structured grids and perform neighbor communication.

[0029] The process-based method for determining computing nodes provided in this embodiment first determines candidate node combinations based on a first target number of computing nodes; then, it determines the ratio of computing nodes to processes in different directions within the candidate node combinations; finally, by determining whether the ratio is an integer, it determines the target node combination and the target computing node. This method, by arranging computing nodes in a two-dimensional manner, reduces cross-node communication between different computing nodes and utilizes more intra-node communication, thereby reducing the runtime of WRF mode, improving computational efficiency, and reducing the energy consumption required for WRF mode computation. It solves the problem that using a one-dimensional sequential mapping method to determine the computing node to which an MPI process belongs leads to a large amount of cross-node communication between computing nodes, affecting the performance of WRF mode.

[0030] As an optional embodiment, the method further includes: If it is determined that there is no target node combination among the candidate node combinations, the process number is determined based on the first direction, the second direction, the first process number, and the second process number. Obtain the second target number of processes, and based on the first and second target numbers, determine the third number of processes contained in the computing node; The target computing node is determined based on the program number and the number of third processes.

[0031] Specifically, if it is determined that there is no target node combination among the candidate node combinations, the first direction, the second direction, the first process number, and the second process number are determined first. For example, the first direction is the x direction, the second direction is the y direction, the first process number in the first direction is Nx, and the second process number in the second direction is Ny.

[0032] Based on the first direction, the second direction, the number of the first process, and the number of the second process, determine the process number, for example: Figure 3 As shown, Nx is 9 and Ny is 10, which determines that the entry and exit process numbers include 0-89, totaling 90 MPI processes.

[0033] Obtain the second target number of processes. For example, specifying 90 processes (second target number N equals 90) and 15 compute nodes (first target number is 15). Based on the first and second target numbers, determine the third number of processes contained in each compute node. For example, the third number of processes = second target number ÷ first target number, indicating how many processes are contained in each compute node. For instance, 90 ÷ 15 = 6, meaning each compute node uses 6 processes.

[0034] Based on the process entry number and the number of third processes, the target computing nodes are determined. For example, processes with process entry numbers 0-5 are assigned to the first target computing node, processes with process entry numbers 6-11 are assigned to the second target computing node, processes with process entry numbers 12-17 are assigned to the third target computing node, and so on. The determined target settlement nodes are as follows: Figure 3 As shown. In Figure 3 In the diagram, the numbers represent process IDs; that is, 90 processes correspond to IDs from 0 to 89. Figure 3 The processes outlined in the black box together constitute a target computing node. Figure 3The black double-headed arrows in the diagram represent cross-node communication relationships between computing nodes. The first target computing node corresponds to process numbers 0-5; the second target computing node corresponds to process numbers 6-11; the third target computing node corresponds to process numbers 12-17, and so on. Each process communicates with its surrounding processes. Processes within the same target computing node communicate intra-node, while processes outside the same target computing node communicate across nodes. The black arrows in the diagram represent communication between different target computing nodes; for example, the first target computing node communicates with the second and third target computing nodes. As can be seen from the diagram, 11 target computing nodes need to communicate with their four adjacent target computing nodes (corresponding to the four black arrows), and the remaining four target computing nodes need to communicate with their two to three adjacent nodes.

[0035] However, the target computing nodes determined by the above method will result in a large amount of inter-node communication, thus affecting the performance of the WRF mode. Therefore, the method of step S103 is preferred to determine the target computing nodes. This embodiment is only used if it is determined that there is no target node combination.

[0036] In this embodiment, if it is determined that the target node combination does not exist among the candidate node combinations, the target computing node is determined based on the process number and the number of third processes contained in the computing node. This enables successful determination of the target computing node in different scenarios, improving the applicability of the method.

[0037] As an optional embodiment, after determining the target computing node, the method further includes: Determine the node number of the target computation node; Based on the first node number, which is a combination of the first process number and the target node number, determine the fourth process number of the processes contained in the target computing node in the first direction; Based on the second node number, which is a combination of the second process number and the target node number, determine the fifth process number of the processes contained in the target computing node in the second direction; Based on the number of fourth processes, the number of fifth processes, the node number, the number of first nodes, and the program number, the mapping relationship between processes and target computing nodes is determined, whereby the mapping relationship is used to run the target mode.

[0038] Specifically, determine the node number of the target computing node. For example, the node number Num is the name of the target computing node, such as cu01, cu02, ..., cu15.

[0039] Based on the first node number, which is the combination of the first process number and the target node number, determine the fourth process number of the processes included in the target computing node in the first direction. For example: if the first process number is Nx, and the first node number is the combination of the target node number... Fourth process number You can divide Nx by Obtained. Based on the second node number resulting from the combination of the second process number and the target node number, determine the fifth process number of the processes contained in the target computing node in the second direction. For example: if the second process number is Ny, the second node number resulting from the combination of the target node number is Ky*, and the fifth process number... This can be obtained by dividing Ny by Ky*. and These represent the number of processes in each compute node in the x and y directions, respectively. =3, =2.

[0040] According to the fourth process number Fifth process number Node number Num, first node number And the process number, which determines the mapping relationship between the process and the target computing node. For example, for any process number Ni (starting from 0), the column number of the process in the x-direction is determined based on the process number. and row number in the y direction That is, dividing Ni by Nx yields the quotient as The remainder is If it is divisible, then the remainder is... It equals 0. Then according to and Obtain the computing node name number Num (starting from 0) corresponding to the process number, and thus determine the mapping relationship of the computing node name corresponding to the process, as shown in formula (1).

[0041] (1) Where / indicates divisibility.

[0042] The process numbers, such as 0-89, can be substituted into formula (1) to obtain the node number Num of the target computing node corresponding to each process. The mapping relationship is used to run the target mode. For example, the mapping relationship is a two-dimensional mapping relationship. The target mode is, for example, the WRF mode.

[0043] In this embodiment, the mapping relationship between processes and target computing nodes is determined based on the fourth process number, the fifth process number, the node number, the first node number, and the program number, and the target mode is run through the mapping relationship. Furthermore, the node number of the target computing node corresponding to the process can be quickly determined based on the mapping relationship and the program number.

[0044] As an optional embodiment, after determining the mapping relationship between the process and the target computing node, the method further includes: If a process to be mapped exists, determine the reference process number of the process to be mapped; Based on the reference process number, determine the first number of the process to be mapped in the first direction and the second number in the second direction; The first data item is obtained based on the first number, the fourth process number, and the first node number; The second data item is obtained based on the second number and the fifth process number; Based on the first data item, the second data item, and the mapping relationship, determine the target computing node corresponding to the process to be mapped.

[0045] Specifically, if a process to be mapped exists, the reference process number of the process to be mapped is determined, for example, the reference process number is any number between 0 and 89.

[0046] Based on the reference process number, determine the first number of the process to be mapped in the first direction and the second number in the second direction. For example, if the reference process number is 87, the corresponding first number is... It is 9, the second number. It is 6. According to the first number Fourth process number and the number of first nodes The first data item is obtained. / )× .

[0047] Based on the second number and the fifth process number, the second data item is obtained ( / ).

[0048] The mapping relationship is as follows: Formula (1), which maps the first data item ( / )× Second data item ( / Substitute into formula (1) to determine the target computing node Num corresponding to the process to be mapped.

[0049] As an optional embodiment, before obtaining a preset number of candidate node combinations based on the first target number of computing nodes, the method further includes: Obtain the second target number of processes; Based on the second target quantity, determine the first candidate quantity of the process in the first direction and the second candidate quantity of the process in the second direction, wherein the product of the first candidate quantity and the corresponding second candidate quantity is the second target quantity; Determine the first difference between the number of first candidates and the corresponding number of second candidates; Based on the first difference, the number of first processes and the number of second processes are determined from the number of first candidates and the number of second candidates.

[0050] Specifically, obtain the second target number of the process, for example: 90.

[0051] Based on the second target quantity, determine the first candidate quantity of processes in the first direction and the second candidate quantity of processes in the second direction. For example, the first direction is x and the second direction is y. For the values ​​of Nx and Ny, the default parallel partitioning scheme automatically executed by WRF is Nx=Ny=N1 / 2. If N cannot be calculated as an integer square root, then the combination where Nx×Ny=N and the difference between Ny and Nx is minimized is taken. For example, if N is specified as 81, WRF automatically performs parallel partitioning with Nx=9 and Ny=9. If N is specified as 90, the first candidate quantity includes 1, 2, 3, 5, 6, 9, 10... and the corresponding second candidate quantity includes 90, 45, 30, 18, 15, 10, 9...

[0052] Determine the first difference between the first candidate quantity and the corresponding second candidate quantity, for example: 89, 43, 27, 13, 9, 1… Based on the first difference, determine the first process number and the second process number from the first candidate quantity and the second candidate quantity. For example, select the pair of first candidate quantities and second candidate quantities with the smallest first difference as the first process number and the second process number, such as Nx=9, Ny=10.

[0053] In this embodiment of the application, the first candidate quantity and the second candidate quantity with the smallest difference are taken as the first process quantity and the second process quantity. When mapping processes to computing nodes in the future, cross-node communication between target computing nodes can be reduced.

[0054] As an optional embodiment, based on the first target number of computing nodes, a preset number of candidate node combinations are obtained, including: Based on the first target quantity, determine the third candidate quantity of computing nodes in the first direction and the fourth candidate quantity of computing nodes in the second direction, wherein the third candidate quantity is less than the fourth candidate quantity, and the product of the third candidate quantity and the fourth candidate quantity is the first target quantity. Based on the number of third and fourth candidates, a preset number of candidate node combinations are obtained.

[0055] Specifically, the first target quantity is, for example, 15.

[0056] To calculate the number of computing nodes Kx in the x-direction and the number of computing nodes Ky in the y-direction, the product of Kx and Ky must equal K, and Kx must be less than Ky, thus yielding different (Kx, Ky) combinations. Therefore, based on the first target number, the third candidate number of computing nodes in the first direction and the fourth candidate number of computing nodes in the second direction are determined. For example, the sum of the third candidate numbers in the x-direction includes 1 and 3, and the sum of the fourth candidate numbers in the y-direction includes 15 and 5.

[0057] Based on the number of third and fourth candidates, a preset number of candidate node combinations (Kx, Ky) are obtained, for example: (3, 5), (1, 15).

[0058] As an optional embodiment, determining a first ratio of the number of first nodes to the number of first processes, and determining a second ratio of the number of second nodes to the number of second processes, includes: Determine the second difference between the first and second number of nodes in the candidate node combination; Based on the second difference, the candidate node combinations are sorted to obtain the combination sequence; Based on the order of candidate node number combinations in the combination sequence, determine the first ratio of the first node number to the first process number in the candidate node number combination, and determine the second ratio of the second node number to the second process number. Determine whether the first ratio value and the second ratio value are integers.

[0059] Specifically, determine the second difference between the first and second number of nodes in the candidate node number combination. For example, the candidate node number combinations include (3, 5) and (1, 15), and the second differences are 2 and 14, respectively.

[0060] Based on the second difference, the candidate node number combinations are sorted to obtain a combination sequence. For example, the candidate node number combinations are arranged in ascending order according to the second difference, and the combination sequence is (3, 5) and (1, 15).

[0061] Based on the order of candidate node combinations in the combination sequence, determine the first ratio of the number of nodes to the number of processes in the first candidate node combination, and determine the second ratio of the number of nodes to the number of processes. For example, first calculate the first candidate node combination in the combination sequence: Kx=3, Ky=5, Nx=9, Ny=10. Then the first ratio is 3, and the second ratio is 2. It is found that Nx is divisible by Kx, and Ny is divisible by Ky. At this time, (3, 5) can be directly set as the target node combination.

[0062] Determine whether the first ratio value and the second ratio value are integers. For example, if the first ratio value is 3 and the second ratio value is 2, we find that Nx can divide Kx and Ny can divide Ky. In this case, we can directly set (3, 5) as the target number of nodes.

[0063] In this embodiment, the candidate node number combinations are sorted according to the second difference. Based on the order of the candidate node number combinations in the combination sequence, the first ratio value and the second ratio value are determined. It is then determined whether the number of nodes in each candidate node number combination meets the requirement of being divisible by the corresponding number of processes. The judgment process ends after the target node number combination is determined, which improves the efficiency of determining the target node number combination and reduces the amount of computation.

[0064] As an optional embodiment, the target computing node corresponding to the process to be mapped is determined based on the first data item, the second data item, and the mapping relationship, including: Substitute the first and second data items into the mapping relationship to obtain the target computing node number corresponding to the process to be mapped. The mapping relationship is obtained through the following formula: ,in, Indicates the second number. Indicates the first number, Indicates the number of the fifth process. Indicates the fourth process number. This represents the first node number of the target node combination. Indicates the first data item. Indicates the second data item. This indicates the target computation node number, and / indicates integer division. The target computing node corresponding to the process to be mapped is determined based on the target computing node number corresponding to the process to be mapped.

[0065] Specifically, through the formula Determine the mapping relationship and assign the first data item. Second data item Substituting the mapping relationship, we obtain the target computing node number corresponding to the process to be mapped. The target computing node numbers are, for example: cu01, cu02, ..., cu15.

[0066] The target compute node corresponding to the process to be mapped is determined based on the target compute node number. For example, if the target compute node number is cu01, then the target compute node corresponding to the process to be mapped is... Figure 2 The compute node in the lower left corner contains processes with serial numbers 0, 1, 2, 9, 10, and 11.

[0067] In this embodiment of the application, the mapping relationship is determined by a formula, and the target computing node corresponding to the process is quickly determined by the mapping relationship, the first data item, and the second data item.

[0068] As an optional embodiment, after determining the mapping relationship between the process and the target computing node, the method further includes: Based on the mapping relationship, determine the target processes contained in the target computing node; Obtain the thread parameters of the data processing thread in the target compute node; Create a data transmission thread based on the thread parameters; Based on the target process, obtain the data to be transmitted from the data processing thread and the target thread corresponding to the data to be transmitted; The data transmission thread is used to transmit the data to the target thread.

[0069] Specifically, based on the mapping relationship, the target processes contained in the target computing node are determined. For example, the target computing node cu01 contains processes with sequence numbers 0, 1, 2, 9, 10, and 11. These processes are the target processes of the target computing node cu01. The target processes are MPI processes.

[0070] In addition to the target process, the target compute node also creates several OpenMP threads, which serve as data processing threads. The MPI process and OpenMP threads can operate in a serial-parallel mode, ensuring that one group of OpenMP threads can transmit data with other groups of OpenMP threads through the MPI process. The data processing threads can run in parallel to perform data processing functions.

[0071] When a data processing thread needs to transfer data with other threads, it can send the data to the target process and then enter a waiting state, at which point the target process sends the data back to the target thread. It is evident that while the target process is transferring data, the data processing thread is in a waiting state, occupying resources on the computing node and resulting in resource waste.

[0072] Based on the above, in order to improve the resource utilization of the target computing node, a data transmission thread is created to use the resources occupied by the data processing thread for data transmission when the data processing thread is in a waiting state.

[0073] Obtain the thread parameters of the data processing threads in the target compute node. These parameters include, for example, the number of data processing threads and the processor information used by each thread. Processor information is used to identify the processor. For example, the resource affinity of the data processing thread is used to characterize the processor used by that data processing thread.

[0074] Based on the thread parameters, create data transfer threads. For example, create the same number of OpenMP threads as the data processing threads, and use these OpenMP threads as data transfer threads, ensuring a one-to-one correspondence between them, with each data transfer thread occupying the same processor. Additionally, when a data processing thread is an OpenMP thread, it can be set to passive mode, allowing other threads to use its processor. Similarly, data transfer threads can be configured to support processor sharing, relinquishing the processor for other threads. Data transfer threads can run in parallel, transferring data simultaneously.

[0075] Based on the target process, obtain the data to be transmitted from the data processing thread and the target thread corresponding to the data to be transmitted. The target thread can be another set of data processing threads in the same target computing node, or it can be a data processing thread in another computing node.

[0076] Data transfer threads are used to transmit data to the target thread. For example, the target thread divides the data to be transmitted into multiple sub-data sets, the number of which is the same as the number of data transfer threads. These threads run in parallel, transmitting the sub-data to the target thread. Additionally, when data transmission is not required, the target process can control the data processing threads to enter a sleep state.

[0077] In this embodiment, the data transmission thread can utilize the resources of the data processing thread to achieve data transmission. This eliminates the need to allocate additional resources to the data transmission thread, effectively improving the resource utilization of the computing node. Furthermore, multiple data transmission threads can transmit data simultaneously, resulting in higher data transmission efficiency and faster completion of data transmission compared to a single running MPI process.

[0078] As an optional embodiment, another method for determining the mapping relationship between a process and a computing node and binding the process to the computing node includes steps A1 to A4.

[0079] Step A1: Different processes communicate using the Message Passing Interface (MPI). Collect communication affinity information between each MPI process in the parallel simulation program under WRF mode, record it in the log file, and organize the communication affinity between each MPI process in the log file.

[0080] Step A2, define the communication overhead model Z of the parallel simulation program, as shown in formula (2). For process and process Communication affinity, Represents a process and process The communication distance between the computing units is calculated to obtain the process mapping of the parallel simulation program. Total communication overhead .

[0081] (2) Step A3: Use a hybrid parallel genetic algorithm to solve for the optimal process mapping strategy based on communication affinity and communication distance between computing units; define the process mapping scheme of the parallel simulation program in WRF mode as an individual, and use iterative methods to select, crossover, mutate, and simulate anneal the individuals in the population to generate the process mapping scheme that minimizes communication overhead Z.

[0082] Step A4: Based on the process mapping scheme obtained in Step A3, determine the mapping relationship between the MPI process and the computing node, statically bind the MPI process to the specified computing node, and rerun the parallel simulation program in WRF mode.

[0083] In this embodiment, the solution quality and speed of the mapping algorithm for solving process mapping strategies are optimized, enabling the solution of high-quality process mapping schemes to be obtained in a short time, effectively reducing the communication overhead of parallel simulation programs, and achieving the effect of communication optimization.

[0084] As an optional embodiment, Figure 4 This is a flowchart of the MPI process and compute node mapping method for WRF mode, such as... Figure 4 As shown, the method includes: Determine the number of MPI processes N and the number of compute nodes K used in running WRF mode, ensuring that N is divisible by K, and a list of compute node names to be used; determine the number of processes Nx in the x-direction and the number of processes Ny in the y-direction; calculate the number of compute nodes Kx in the x-direction and the number of compute nodes Ky in the y-direction, ensuring that the product of Kx and Ky equals K, and that Kx is less than Ky, thus obtaining different combinations; arrange the different combinations in ascending order of the difference between Ky and Kx; iterate through each (Kx, Ky) combination, checking whether the number of processes Nx in the x-direction divides the number of compute nodes Kx in the x-direction, and simultaneously whether the number of processes Ny in the y-direction divides the number of compute nodes Ky in the y-direction. If all are divisible, assign Kx and Ky in the current combination to Kx* and Ky* respectively. The number of processes Nx in the x-direction, the number of processes Ny in the y-direction, the number of compute nodes Kx* in the x-direction, the number of compute nodes Ky* in the y-direction, and the list of compute node names to be used are all determined. Then determine the number of processes Mi in the x-direction and the number of processes Mj in the y-direction within each compute node. For any process number Ni, first determine its column and row numbers Nii and Njj, i.e., divide Ni by Nx to obtain the quotient Njj and the remainder Nii. If divisible, the remainder Nii equals 0. Then, based on Nii and Njj, obtain the compute node name number Num corresponding to the process number, thus determining the mapping relationship for the compute node name corresponding to the process. This results in a two-dimensional mapping between MPI processes and compute nodes, and the WRF mode is run based on this two-dimensional mapping. If all are not divisible, it is determined that there is no combination that meets the conditions. If there is no combination that meets the conditions, a one-dimensional sequential mapping of MPI processes and compute nodes is used to run the WRF mode.

[0085] In this embodiment, the computing nodes are arranged in a two-dimensional pattern, such that the number of computing nodes Kx and Ky in the x and y directions are as consistent as possible, and the difference between the number of computing nodes Ky in the y direction and the number of computing nodes Kx in the x direction is minimized. Compared with the commonly used one-dimensional sequential mapping method, this mapping method can significantly reduce cross-node communication between different computing nodes and make more use of intra-node communication, thereby reducing the running time of WRF mode, improving efficiency, and further reducing the energy consumption required for a single weather forecast and saving electricity.

[0086] This embodiment also provides a device for determining computing nodes based on processes. This device is used to implement the above embodiments and preferred embodiments, and details already described will not be repeated. As used below, the term "module" can be a combination of software and / or hardware that implements a predetermined function. Although the device described in the following embodiments is preferably implemented in software, hardware implementation, or a combination of software and hardware, is also possible and contemplated.

[0087] This embodiment provides a device for determining computing nodes based on processes, such as... Figure 5 As shown, it includes: The combination determination module 501 is used to obtain a preset number of candidate node number combinations based on the first target number of computing nodes, given the first number of processes in the first direction and the second number of processes in the second direction. The candidate node number combination includes the first number of computing nodes in the first direction and the second number of computing nodes in the second direction, and the product of the first number of nodes and the second number of nodes is the first target number. The ratio determination module 502 is used to determine a first ratio value between the number of first nodes and the number of first processes, and to determine a second ratio value between the number of second nodes and the number of second processes; The node determination module 503 is used to determine the target computing node based on the target node combination if a target node combination exists among the candidate node number combinations, wherein the first ratio value and the second ratio value corresponding to the target node number combination are both integers.

[0088] In some alternative embodiments, the device further includes: The first determining module is used to determine the process number based on the first direction, the second direction, the first process number, and the second process number if the target node number combination is not found in the determined candidate node number combination. The first acquisition module is used to acquire the second target number of processes, and determine the third number of processes contained in the computing node based on the first target number and the second target number. The second determination module is used to determine the target computing node based on the program number and the third process number.

[0089] In some alternative embodiments, the device further includes: The third determination module is used to determine the node number of the target computing node; The fourth determining module is used to determine the fourth number of processes contained in the target computing node in the first direction based on the first number of nodes, which is a combination of the first number of processes and the target number of nodes. The fifth determining module is used to determine the fifth process number of the processes contained in the target computing node in the second direction based on the second node number, which is a combination of the second process number and the target node number; The sixth determination module is used to determine the mapping relationship between processes and target computing nodes based on the fourth process number, the fifth process number, the node number, the first node number, and the program number. The mapping relationship is used to run the target mode.

[0090] In some alternative embodiments, the device further includes: The seventh determination module is used to determine the reference process number of the process to be mapped when there is a process to be mapped. The eighth determining module is used to determine the first number of the process to be mapped in the first direction and the second number in the second direction based on the reference process number; The first calculation module is used to obtain the first data item based on the first number, the fourth process number, and the first node number; The second calculation module is used to obtain the second data item based on the second number and the fifth process number; The ninth determining module is used to determine the target computing node corresponding to the process to be mapped based on the first data item, the second data item, and the mapping relationship.

[0091] In some alternative embodiments, the device further includes: The second acquisition module is used to acquire the second target number of processes; The tenth determining module is used to determine the first candidate number of the process in the first direction and the second candidate number of the process in the second direction based on the second target number, wherein the product of the first candidate number and the corresponding second candidate number is the second target number; The eleventh determining module is used to determine the first difference between the number of first candidates and the corresponding number of second candidates; The twelfth determining module is used to determine the first process number and the second process number from the first candidate number and the second candidate number based on the first difference.

[0092] In some alternative implementations, the combination determination module 501 includes: The first determining unit is used to determine the third candidate number of computing nodes in the first direction and the fourth candidate number of computing nodes in the second direction based on the first target number, wherein the third candidate number is less than the fourth candidate number, and the product of the third candidate number and the fourth candidate number is the first target number. The calculation unit is used to obtain a preset number of candidate node combinations based on the number of third and fourth candidates.

[0093] In some optional implementations, the ratio determination module 502 includes: The second determining unit is used to determine the second difference between the first number of nodes and the second number of nodes in the candidate node number combination; The sorting unit is used to sort the combinations of candidate node numbers according to the second difference to obtain a combination sequence; The third determining unit is used to determine the first ratio of the number of first nodes to the number of first processes in the candidate node number combination according to the arrangement order of the candidate node number combination in the combination sequence, and to determine the second ratio of the number of second nodes to the number of second processes. The judgment unit is used to determine whether the first ratio value and the second ratio value are integers.

[0094] In some optional implementations, the ninth determining module includes: The data processing unit is used to substitute the first data item and the second data item into the mapping relationship to obtain the target computing node number corresponding to the process to be mapped. The mapping relationship is obtained by the following formula: ,in, Indicates the second number. Indicates the first number, Indicates the number of the fifth process. Indicates the fourth process number. This represents the first node number of the target node combination. Indicates the first data item. Indicates the second data item. This indicates the target computation node number, and / indicates integer division. The fourth determining unit is used to determine the target computing node corresponding to the process to be mapped based on the target computing node number corresponding to the process to be mapped.

[0095] In some alternative embodiments, the device further includes: The thirteenth determination module is used to determine the target process contained in the target computing node based on the mapping relationship; The third acquisition module is used to acquire the thread parameters of the data processing thread in the target computing node; Create a module to create a data transmission thread based on thread parameters; The fourth acquisition module is used to acquire the data to be transmitted from the data processing thread and the target thread corresponding to the data to be transmitted, based on the target process. The data transmission module is used to transmit data to the target thread using the data transmission thread.

[0096] Further functional descriptions of the above modules and units are the same as those in the corresponding embodiments described above, and will not be repeated here.

[0097] In this embodiment, the device for determining computing nodes based on processes is presented in the form of functional units. Here, a unit refers to an ASIC (Application Specific Integrated Circuit) circuit, a processor and memory that execute one or more software or fixed programs, and / or other devices that can provide the above functions.

[0098] This application also provides a computer device having the above-described features. Figure 5 The apparatus shown is for determining computing nodes based on processes.

[0099] Please see Figure 6 , Figure 6 This is a schematic diagram of the structure of a computer device provided in an optional embodiment of this application, such as... Figure 6 As shown, the computer device includes one or more processors 10, memory 20, and interfaces for connecting the components, including high-speed interfaces and low-speed interfaces. The components communicate with each other via different buses and can be mounted on a common motherboard or otherwise installed as needed. The processors can process instructions executed within the computer device, including instructions stored in or on memory to display graphical information of a GUI on external input / output devices (such as display devices coupled to the interfaces). In some alternative implementations, multiple processors and / or multiple buses can be used with multiple memories and multiple memory modules, if desired. Similarly, multiple computer devices can be connected, each providing some of the necessary operations (e.g., as a server array, a group of blade servers, or a multiprocessor system). Figure 6 Take a processor 10 as an example.

[0100] Processor 10 may be a central processing unit, a network processor, or a combination thereof. Processor 10 may further include an integrated circuit, a programmable logic device, or a combination thereof. The programmable logic device may be a complex programmable logic device, a field-programmable gate array (FPGA), a general-purpose array logic (GPA), or any combination thereof.

[0101] The memory 20 stores instructions executable by at least one processor 10 to cause at least one processor 10 to perform the method shown in the above embodiments.

[0102] The memory 20 may include a program storage area and a data storage area. The program storage area may store the operating system and applications required for at least one function; the data storage area may store data created based on the use of the computer device. Furthermore, the memory 20 may include high-speed random access memory and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some alternative embodiments, the memory 20 may optionally include memory remotely located relative to the processor 10, and these remote memories may be connected to the computer device via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0103] The memory 20 may include volatile memory, such as random access memory; the memory may also include non-volatile memory, such as flash memory, hard disk or solid-state drive; the memory 20 may also include a combination of the above types of memory.

[0104] The computer device also includes a communication interface 30 for communicating with other devices or communication networks.

[0105] This application also provides a computer-readable storage medium. The methods described in this application can be implemented in hardware or firmware, or implemented as recordable on a storage medium, or implemented as computer code downloaded over a network and originally stored on a remote storage medium or a non-transitory machine-readable storage medium and subsequently stored on a local storage medium. Thus, the methods described herein can be processed by software stored on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. The storage medium can be a magnetic disk, optical disk, read-only memory, random access memory, flash memory, hard disk, or solid-state drive, etc.; further, the storage medium can also include combinations of the above types of memory. It is understood that computers, processors, microprocessor controllers, or programmable hardware include storage components capable of storing or receiving software or computer code. When the software or computer code is accessed and executed by the computer, processor, or hardware, the methods shown in the above embodiments are implemented.

[0106] A portion of this application can be applied as a computer program product, such as computer program instructions, which, when executed by a computer, can invoke or provide the methods and / or technical solutions according to this application through the operation of the computer. Those skilled in the art will understand that the forms in which computer program instructions exist in a computer-readable medium include, but are not limited to, source files, executable files, installation package files, etc. Correspondingly, the ways in which computer program instructions are executed by a computer include, but are not limited to: the computer directly executing the instructions, or the computer compiling the instructions and then executing the corresponding compiled program, or the computer reading and executing the instructions, or the computer reading and installing the instructions and then executing the corresponding installed program. Here, the computer-readable medium can be any available computer-readable storage medium or communication medium accessible to a computer.

[0107] Although embodiments of this application have been described in conjunction with the accompanying drawings, those skilled in the art can make various modifications and variations without departing from the spirit and scope of this application, and such modifications and variations all fall within the scope defined by this application.

Claims

1. A method for determining computing nodes based on processes, characterized in that, The method includes: Given a first process number in the first direction and a second process number in the second direction, a preset number of candidate node number combinations are obtained based on a first target number of computing nodes. The candidate node number combination includes the first node number of the computing nodes in the first direction and the second node number of the computing nodes in the second direction, and the product of the first node number and the second node number is the first target number. Determine a first ratio of the number of first nodes to the number of first processes, and determine a second ratio of the number of second nodes to the number of second processes; If it is determined that a target node combination exists among the candidate node combinations, a target computing node is determined based on the target node combination, wherein the first ratio value and the second ratio value corresponding to the target node combination are both integers.

2. The method according to claim 1, characterized in that, The method further includes: If it is determined that the target node combination does not exist in the candidate node combination, the process number is determined based on the first direction, the second direction, the first process number, and the second process number. Obtain the second target number of the processes, and determine the third number of processes contained in the computing node based on the first target number and the second target number; The target computing node is determined based on the process number and the number of third processes.

3. The method according to claim 2, characterized in that, After determining the target computing node, the method further includes: Determine the node number of the target computing node; Based on the first number of nodes, which is a combination of the first number of processes and the target number of nodes, determine the fourth number of processes contained in the target computing node in the first direction; Based on the second number of nodes, which is a combination of the second number of processes and the target number of nodes, determine the fifth number of processes contained in the target computing node in the second direction; Based on the fourth process number, the fifth process number, the node number, the first node number, and the process number, a mapping relationship between the process and the target computing node is determined, wherein the mapping relationship is used to run the target mode.

4. The method according to claim 3, characterized in that, After determining the mapping relationship between the process and the target computing node, the method further includes: If a process to be mapped exists, determine the reference process number of the process to be mapped; Based on the reference process number, determine the first number of the process to be mapped in the first direction and the second number in the second direction; The first data item is obtained based on the first number, the fourth process number, and the first node number; The second data item is obtained based on the second number and the fifth process number; Based on the first data item, the second data item, and the mapping relationship, the target computing node corresponding to the process to be mapped is determined.

5. The method according to claim 1, characterized in that, Before obtaining a preset number of candidate node combinations based on the first target number of computing nodes, the method further includes: Obtain the second target number of processes; Based on the second target quantity, a first candidate quantity of the process in the first direction and a second candidate quantity of the process in the second direction are determined, wherein the product of the first candidate quantity and the corresponding second candidate quantity is the second target quantity; Determine the first difference between the first number of candidates and the corresponding second number of candidates; Based on the first difference, the first process number and the second process number are determined from the first candidate number and the second candidate number.

6. The method according to claim 1, characterized in that, The step of obtaining a preset number of candidate node combinations based on the first target number of computing nodes includes: Based on the first target quantity, determine the third candidate quantity of the computing node in the first direction and the fourth candidate quantity of the computing node in the second direction, wherein the third candidate quantity is less than the fourth candidate quantity, and the product of the third candidate quantity and the fourth candidate quantity is the first target quantity; Based on the third candidate number and the fourth candidate number, a preset number of candidate node combinations are obtained.

7. The method according to claim 1 or 6, characterized in that, Determining the first ratio of the number of first nodes to the number of first processes, and determining the second ratio of the number of second nodes to the number of second processes, includes: Determine a second difference between the first number of nodes and the second number of nodes in the candidate node combination; Based on the second difference, the combinations of candidate node numbers are sorted to obtain a combination sequence; Based on the arrangement order of the candidate node number combinations in the combination sequence, determine a first ratio of the first node number to the first process number in the candidate node number combination, and determine a second ratio of the second node number to the second process number. Determine whether the first ratio value and the second ratio value are integers.

8. The method according to claim 4, characterized in that, The step of determining the target computing node corresponding to the process to be mapped based on the first data item, the second data item, and the mapping relationship includes: Substituting the first data item and the second data item into the mapping relationship, the target computing node number corresponding to the process to be mapped is obtained; The mapping relationship is obtained through the following formula: ,in, Indicates the second number, This indicates the first number. This indicates the number of the fifth process. This indicates the number of the fourth process. This represents the first node number in the target node combination. This represents the first data item. This represents the second data item. This represents the target computing node number, and / indicates integer division; The target computing node corresponding to the process to be mapped is determined based on the target computing node number corresponding to the process to be mapped.

9. The method according to claim 3, characterized in that, After determining the mapping relationship between the process and the target computing node, the method further includes: Based on the mapping relationship, the target processes contained in the target computing node are determined; Obtain the thread parameters of the data processing thread in the target computing node; Based on the thread parameters, a data transmission thread is created; Based on the target process, obtain the data to be transmitted from the data processing thread and the target thread corresponding to the data to be transmitted; The data transmission thread is used to transmit the data to the target thread.

10. A computer device, characterized in that, include: A memory and a processor are communicatively connected, the memory storing computer instructions, and the processor executing the computer instructions to perform the process-based method for determining computing nodes as described in any one of claims 1 to 9.

Images