A regional ocean mode load balancing test method based on many-core processors

Abstract

The present invention relates to a regional ocean mode load balancing test method based on a many-core processor, which belongs to the field of load balancing test technology, including obtaining the code of the regional ocean mode system, transplanting it to the many-core processor platform, allocating the calculation task of the hotspot function in the regional ocean mode system to the slave core, and performing many-core acceleration; configuring the load balancing statistical library in the configuration file of the regional ocean mode system, calling the statistical function of the load balancing statistical library by calling the interface of the load balancing statistical library; integrating the statistical function with the timer function at the entrance and exit of the hotspot function, executing the statistical function and the timer function, and performing statistics on the time consumption and load balancing of the hotspot function; recording and outputting the statistical situation. By integrating the load balancing statistical tool with the timer function, a lot of repeated work can be saved, and the load balancing situation of the current parallel strategy can be quickly analyzed, providing an optimization direction for the next step of acceleration.

Description

Technical Field

[0001] The present invention relates to the technical field of many-core processor load balancing, and in particular to a regional ocean mode load balancing test method based on many-core processors. Background Art

[0002] When accelerating the regional ocean model with multiple cores, the regional ocean model simulation software was first transplanted to the domestic supercomputer platform to achieve normal operation of the master core version and consistency of simulation results. On this basis, for hot functions, that is, functions with high time consumption, 8*8 slave cores were fully utilized, and the calculation tasks were distributed to 64 slave cores through algorithms to achieve parallel calculations, thereby improving calculation efficiency.

[0003] Among them, load balancing is one of the key issues affecting the efficiency of parallel computing. Ensuring load balancing can avoid the situation where some processors are overloaded while other processors are idle in the multi-core processor, thereby avoiding resource waste and improving parallel efficiency. Therefore, it is very important to confirm whether there is a load imbalance problem in the multi-core acceleration process of the regional ocean model, which is very important for improving parallel efficiency and speeding up the numerical simulation process.

[0004] The existing regional ocean model does not integrate load balancing analysis tools. Users need to add load balancing test tools before and after specific modules or code segments as needed, which requires a lot of repetitive work and consumes manpower. Summary of the invention

[0005] In view of the shortcomings existing in the related art, the purpose of the present invention is to provide a regional ocean mode load balancing test method based on a many-core processor to solve the technical problem that the existing regional ocean mode does not integrate the load balancing analysis tool, and the user needs to add the load balancing test tool before and after the specific module or code segment as needed, which requires a lot of repetitive work and consumes manpower.

[0006] The present invention provides a regional ocean mode load balancing test method based on a many-core processor, comprising the following steps:

[0007] S1. Obtain the code of the regional ocean model system, transplant the code to the many-core processor platform, assign the computing tasks of the hotspot functions in the regional ocean model system to the slave cores, and perform many-core acceleration;

[0008] S2, configuring a load balancing statistical library in a configuration file of the regional ocean model system, and calling a statistical function of the load balancing statistical library by calling an interface of the load balancing statistical library;

[0009] S3, calling the statistical function of the load balancing statistical library, integrating the statistical function with the timer functions at the entrance and exit of the hotspot function, executing the statistical function and the timer function, and performing statistics on the time consumption and load balancing of the hotspot function;

[0010] S4. Record and output the time consumption and load balancing statistics of the hotspot function.

[0011] In the embodiment of the present invention, by integrating the load balancing statistics library tool with the timer function and applying them to the hot spot function, there is no need to add the load balancing test tool before and after the specific module or code segment, which can save a lot of repetitive work. The load balancing statistics library tool and the timer function are combined to quickly analyze the load balancing situation of the current parallel strategy and provide optimization direction for the next step of acceleration.

[0012] In some embodiments of the present invention, the step S2 further includes: querying the MPI initialization position in the code, adding the initialization function of the load balancing statistics library to the initialization position of the MPI, and calling the initialization function of the load balancing statistics library after the MPI is initialized.

[0013] The embodiment of the present invention can ensure that the statistical function of the load balancing statistical library runs in a multi-process environment by adding the initialization function of the load balancing statistical library to the MPI initialization.

[0014] In some embodiments of the present invention, step S2 further comprises:

[0015] Compiling the load balancing statistics library into a static link library through a preset compiler, and configuring the static link library in a makefile file of the regional ocean model system;

[0016] The compiler compiles the interface file of the load balancing statistics library into an interface module, which is used to call the interface of the load balancing statistics library by calling the interface module.

[0017] The embodiment of the present invention realizes transplanting the load balancing statistics library to the regional ocean mode for counting the load balancing situation between processes by compiling and configuring the link library of the load balancing statistics library and the interface file of the load balancing statistics library.

[0018] In some embodiments of the present invention, step S3 further comprises:

[0019] Step S31, configuring a mapping table for recording the coding of the hotspot function and the functional description of the hotspot function;

[0020] Step S32: when the execution of the timer function at the entrance of the hotspot function begins, the statistical function of the load balancing statistical library is called and executed, the code of the hotspot function is read, and the time consumption and load balancing of the hotspot function are counted. When the execution of the timer function at the exit of the hotspot function ends, the call of the statistical function of the load balancing statistical library ends;

[0021] After step S32 is completed, the read code of the hotspot function is converted into a corresponding functional description of the hotspot function according to the mapping table.

[0022] The embodiment of the present invention can utilize the existing code logic to quickly implement code modification and realize inter-process load balancing statistics of hot functions by merging the statistical function of the load balancing statistical library with the timer function; through the defined mapping table, each function code and its corresponding function function description can be clearly defined to ensure that the code is converted into the corresponding function function description when the statistics are output, which is convenient for users to understand and find the corresponding function module.

[0023] In some embodiments of the present invention, step S4 further comprises:

[0024] Record the time consumption of the process with the highest time consumption in the hotspot function in a log file, and output the time consumption in the log file;

[0025] Record the load balancing statistics of the hotspot function in a statistical file, and output the functional description of the hotspot function, the time consumption of the hotspot function, the time consumption of the process with the highest time consumption in the hotspot function, the time consumption of the process with the lowest time consumption in the hotspot function, and the time consumption variance between the processes in the statistical file;

[0026] When the time difference between the process with the highest time consumption and the process with the lowest time consumption recorded in the statistical file reaches a preset value, there is a load imbalance in the codes corresponding to the process with the highest time consumption and the process with the lowest time consumption. By modifying the codes corresponding to the processes, the time variance between processes can be reduced.

[0027] The embodiment of the present invention obtains the large difference in time consumption between processes recorded in the statistical file through the statistical function of the load balancing statistical library, that is, the load imbalance situation, and can further refine the load balancing situation of each part of the code in the hot spot function, adjust the algorithm in a targeted manner, and solve the load imbalance problem. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the specific embodiments of the present invention will be described in detail below in conjunction with the accompanying drawings. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0029] Figure 1 A flow chart of a regional ocean mode load balancing test method based on a many-core processor provided in an embodiment of the present invention;

[0030] Figure 2 An operation flow chart of a statistical function and a timer function of a load balancing statistical library provided by an embodiment of the present invention;

[0031] Figure 3 An application flow chart of a regional ocean mode load balancing test method based on a many-core processor provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0032] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application is described and illustrated below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not intended to limit the present application. Based on the embodiments provided in the present application, all other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of the present application.

[0033] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit the exemplary embodiments according to the present application. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should be understood that the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0034] In the absence of conflict, the embodiments of the present invention and the features of the embodiments may be combined with each other.

[0035] The technical solution of the present invention is described in detail below in conjunction with specific embodiments and the accompanying drawings.

[0036] As attached Figure 1 As shown, in an illustrative embodiment of a regional ocean mode load balancing test method based on a multi-core processor of the present invention, it includes:

[0037] S1. Obtain the code of the regional ocean model system, transplant the code to the many-core processor platform, and assign the computing tasks of the hotspot functions in the regional ocean model system to the slave cores for many-core acceleration; optionally, the many-core processor platform is based on the "Shenwei*Wenhai No. 1" supercomputer platform, equipped with the domestic SW26010pro heterogeneous many-core processor, which uses the 64-bit independent Shenwei instruction set and adopts a master-slave core architecture. One processor includes 6 core groups, each of which includes 1 master core, 8*8 slave cores and a memory controller; transplant the regional ocean model simulation software to the supercomputer platform to achieve normal operation of the master core version and consistency of simulation results, configure the Shenwei compiler to compile according to the compilation requirements of the supercomputer platform, and implement the transplantation of the master core version of the regional ocean model. Under the same circumstances, compare the calculation results with the x86 platform to ensure data consistency, assign the computing tasks of the hotspot function to 8*8 slave cores, and ensure data consistency, implement computing parallelism, and thus improve computing efficiency;

[0038] S2, configuring the load balancing statistics library in the configuration file of the regional ocean model system, and calling the statistics function of the load balancing statistics library by calling the interface of the load balancing statistics library;

[0039] S3, calling the statistical function of the load balancing statistical library, integrating the statistical function with the timer function at the entrance and exit of the hotspot function, executing the statistical function and the timer function, and performing statistics on the time consumption and load balancing of the hotspot function; it should be noted that the hotspot function is a function with a high time consumption, and different test cases have different hotspot functions;

[0040] S4. Record and output the time consumption and load balancing statistics of the hotspot function.

[0041] Based on the above method, by integrating the load balancing statistics library tool with the timer function and applying it to the hot function, there is no need to add the load balancing test tool before and after a specific module or code segment, which can save a lot of repetitive work. The timer function combined with the load balancing statistics library tool can quickly analyze the load balancing situation of the current parallel strategy, and output the load balancing situation between all key function processes after each simulation calculation, providing optimization direction for the next step of acceleration.

[0042] In order to count the load balancing situation between processes, the load balancing statistics library is transplanted to the regional ocean mode based on the Shenwei multi-core processor. The MPI initialization position in the code is queried, and the initialization function of the load balancing statistics library is added to the initialization position of the MPI. The initialization function of the load balancing statistics library is called after the MPI is initialized. At the entrance of the regional ocean mode, the load balancing statistics library initialization function is called after the MPI is initialized. Before closing all MPI processes, the load balancing statistics library performance test is ended and the load balancing statistics results are printed out.

[0043] Compile the load balancing statistics library into a static link library *.a file through a preset compiler, and configure the static link library *.a file in a makefile file of the regional ocean model system; optionally, the compiler is a Shenwei compiler;

[0044] The interface file of the load balancing statistics library is compiled into an interface module by a compiler, which is used to call the interface of the load balancing statistics library by calling the interface module; optionally, the compiler is the Shenwei compiler.

[0045] It should be noted that the statistical functions of the load balancing statistics library need to run in a multi-process environment, that is, they need to run in an MPI environment. Therefore, the modules outside the MPI environment are timed and the statistics of the load balancing statistics library are not enabled.

[0046] In order to ensure that users can understand and find corresponding function modules easily when outputting load balancing statistics, a mapping table is configured to record the encoding of hotspot functions and the functional description of hotspot functions. According to the mapping table, the encoding of the read hotspot function can be converted into the functional description of the corresponding hotspot function.

[0047] Combined with reference Figure 2 As shown, the native timer function wclock code is modified, and the statistical function of the load balancing statistical library is integrated into the timer function. When the timer function at the entrance of the hotspot function starts to execute, that is, wclock_on, the statistical function of the load balancing statistical library is called and executed, the code of the hotspot function is read, and the time consumption and load balancing of the hotspot function are counted. When the timer function at the exit of the hotspot function ends, that is, wclok_off, the execution of the statistical function of the load balancing statistical library ends.

[0048] It should be noted that, in conjunction with the reference Figure 3 As shown, if there is no module with a timer function in the native code, it is still necessary to add the encoding of the hotspot function and the functional description of the hotspot function in the mapping table, and also to add the function code after merging the statistical function of the load balancing statistics library and the timer function to the entry and exit of the hotspot function to complete the hotspot function timing and load balancing statistics.

[0049] After the time consumption and load balancing of the hot function are counted, the time consumption of the process with the highest time consumption in the hot function is recorded in the log file, and the time consumption in the log file is output;

[0050] Record the load balancing statistics of the hotspot function in a statistical file, and output the function description of the hotspot function, the time consumption of the hotspot function, the time consumption of the process with the highest time consumption in the hotspot function, the time consumption of the process with the lowest time consumption in the hotspot function, and the time consumption variance between processes in the statistical file;

[0051] When the time difference between the process with the highest time consumption and the process with the lowest time consumption recorded in the statistical file reaches a preset value, there is a load imbalance in the codes corresponding to the process with the highest time consumption and the process with the lowest time consumption. By modifying the code corresponding to the process, the time variance between processes can be reduced.

[0052] It should be noted that the above is a reference method for a regional ocean mode load balancing test method based on a multi-core processor, and the present invention is not limited thereto.

[0053] The embodiments of the present invention realize the integration of the load balancing statistical library tool and the timer function, and apply them to the hot spot function, which can save a lot of repetitive work. The timer function combined with the load balancing statistical library tool can quickly analyze the load balancing situation of the current parallel strategy, and output the load balancing situation between all key function processes after each simulation calculation, providing optimization direction for the next acceleration. It solves the technical problem that the prior art does not integrate the load balancing analysis tool, and needs to add the load balancing test tool before and after the specific module or code segment, which is time-consuming and manpower-consuming.

[0054] Finally, it should be noted that: the various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0055] The above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, ordinary technicians in the field should understand that the specific implementation methods of the present invention can still be modified or some technical features can be replaced by equivalents without departing from the spirit of the technical solution of the present invention, which should be included in the scope of the technical solution for protection of the present invention.

Claims

1. A regional ocean mode load balancing test method based on many-core processors, characterized in that: The steps include: S1. Obtain the code of the regional ocean model system, transplant the code to the many-core processor platform, assign the computing tasks of the hotspot functions in the regional ocean model system to the slave cores, and perform many-core acceleration; S2, configuring a load balancing statistical library in a configuration file of the regional ocean model system, and calling a statistical function of the load balancing statistical library by calling an interface of the load balancing statistical library; S3, calling the statistical function of the load balancing statistical library, integrating the statistical function with the timer functions at the entrance and exit of the hotspot function, executing the statistical function and the timer function, and performing statistics on the time consumption and load balancing of the hotspot function; S4, recording and outputting the time consumption and load balancing statistics of the hotspot function; Wherein, the step S3 further comprises: Step S31, configuring a mapping table for recording the coding of the hotspot function and the functional description of the hotspot function; Step S32: when the execution of the timer function at the entrance of the hotspot function begins, the statistical function of the load balancing statistical library is called and executed, the code of the hotspot function is read, and the time consumption and load balancing of the hotspot function are counted. When the execution of the timer function at the exit of the hotspot function ends, the execution of the statistical function of the load balancing statistical library ends; Among them, after the step S32 is completed, the read code of the hotspot function is converted into the corresponding functional description of the hotspot function according to the mapping table.

2. The regional ocean mode load balancing test method based on many-core processors according to claim 1 is characterized in that: The step S2 further includes: querying the MPI initialization position in the code, adding the initialization function of the load balancing statistics library to the initialization position of the MPI, and calling the initialization function of the load balancing statistics library after the MPI is initialized.

3. The regional ocean mode load balancing test method based on many-core processors according to claim 1 is characterized in that: The step S2 further comprises: compiling the load balancing statistics library into a static link library by a preset compiler, and configuring the static link library in the makefile file of the regional ocean model system.

4. The regional ocean mode load balancing test method based on many-core processors according to claim 3 is characterized in that: The step S2 further includes: compiling the interface file of the load balancing statistics library into an interface module by the compiler, so as to call the interface of the load balancing statistics library by calling the interface module.

5. The regional ocean mode load balancing test method based on many-core processors according to claim 1 is characterized in that: The step S4 further includes: recording the time consumption of the process with the highest time consumption in the hotspot function in a log file, and outputting the time consumption in the log file.

6. The regional ocean mode load balancing test method based on many-core processors according to claim 5 is characterized in that: The step S4 further includes: recording the load balancing statistics of the hotspot function in a statistical file, and outputting the functional description of the hotspot function, the time consumption of the hotspot function, the time consumption of the process with the highest time consumption in the hotspot function, the time consumption of the process with the lowest time consumption in the hotspot function, and the time consumption variance between the processes in the statistical file.

7. The regional ocean mode load balancing test method based on many-core processors according to claim 6 is characterized in that: The step S4 further includes: when the time consumption difference between the process with the highest time consumption and the process with the lowest time consumption recorded in the statistical file reaches a preset value, there is a load imbalance in the codes corresponding to the process with the highest time consumption and the process with the lowest time consumption, and the time consumption variance between processes is reduced by modifying the codes corresponding to the processes.

Images