Computing resource limitation and communication redundancy optimization method based on new-generation SW processor

Abstract

The invention relates to a computing resource limitation and communication redundancy optimization method based on a new generation SW processor, and belongs to the technical field of electronic information. Comprising the following steps: firstly, analyzing a hotspot calculation part, and judging the number of cycles of the hotspot calculation part and the number of core groups required by calculation amount; wherein if the code running time of the calculation part accounts for more than half of the total code running time, the calculation part is judged to be a hotspot calculation part; then, applying and numbering the multi-core group, and enabling the multi-core group to carry out cooperative calculation on the hotspot calculation part; thirdly, after the cooperative calculation is completed, grouping the slave cores in the core groups, optimizing data collection and transmission, and performing data collection and transmission among the core groups; and finally, the final result is transmitted back to the main memory once. According to the method, by applying for multi-core group collaborative optimization, slave core optimization can utilize more computing resources, and the execution efficiency of the program is greatly improved.

Description

Technical Field

[0001] The invention relates to a method for optimizing limited computing resources and communication redundancy based on a new generation of Shenwei processors, and belongs to the field of electronic information technology. Background Art

[0002] At present, my country's new generation of Sunway supercomputer "Sunway Blue Light II" has been put into use. Its Sunway multi-core processor SW26010pro is a representative of China's self-developed processors. The development of the Sunway series of supercomputers has provided strong support for data simulation and scientific research in various fields.

[0003] The SW26010pro chip includes 6 core groups (CG), each of which includes a management processing element (MPE), also known as the master core. An 8 × 8 computing processing element (CPE), also known as a slave core array. Each core group has its own memory controller (MC), which is 16 GB of DDR4 memory and is connected to the memory controller through a direct memory access (DMA) with a bandwidth of 51.2 GB / s. Data is transferred between two slave cores in the same slave core array through remote memory access (RMA). Each slave core has a 256KB local data memory (LDM). Each SW26010pro processor consists of 390 processing units. A single SW26010pro processor can provide about 14 TB / s of floating-point operations and about 307.2 GB / s of memory access bandwidth performance through DMA. The hardware architecture of the SW26010pro processor is as follows Figure 1 shown.

[0004] When optimizing programs using the new generation of Sunway supercomputers, a combination of process-level optimization and thread-level optimization is usually used, more specifically, MPI and Athread technologies are used. Process-level optimization means running processes on the master core of the core group to complete the application's computing tasks in parallel, and further improving program execution efficiency by reducing communications and other means. Thread-level optimization means running threads on each slave core to complete the acceleration task of process startup in parallel. A process can use the Athread interface to start the slave core array of the core group, and then further subdivide the task to each slave core for execution.

[0005] When optimizing and accelerating a program, for programs with thousands or even millions of loops and a large amount of computation within the loop, more computing resources are often needed to improve program execution efficiency. The general parallel optimization method is a combination of task parallelism and thread parallelism. When optimizing a program based on the new generation of Sunway multi-core processors, the commonly used parallel optimization method is MPI + Athread.

[0006] Task parallelism mainly refers to process-level parallelism between core groups, that is, the computing tasks of the application are completed in parallel by running processes on the master core of the core group. According to the interface provided by the parallel programming framework of the application development, different processes can be distinguished by using the MPI process number. Thread parallelism mainly refers to the acceleration task of process startup by running threads on each slave core in parallel. The process can use the Athread interface to start the slave core array of a single core group, and further subdivide the tasks to each slave core for execution, thereby greatly improving the efficiency of program execution.

[0007] However, although some programs can be optimized at the process level by adding MPI, the acceleration effect may not be obvious due to the long communication time. For such programs, it is usually not recommended to use process-level optimization, but it is more suitable to improve the execution speed by using better thread-level optimization. Since only thread-level optimization is performed, the available computing resources will be reduced. Therefore, when faced with programs with thousands or even millions of loops and large amounts of calculations within the loop, the acceleration effect may still be limited. At the same time, after the slave core calculation acceleration is completed, it is usually necessary to reduce the calculation results of each slave core or transfer them back to the main memory through DMA, but these operations often increase the overall execution time of the program and reduce efficiency. Summary of the invention

[0008] In view of the shortcomings of the prior art, the present invention provides a method for optimizing computing resource constraints and communication redundancy based on a new generation of Shenwei processors; When optimizing a program on a Shenwei multi-core processor, if the program joins MPI and the optimization effect is not obvious, it can only rely on better slave core optimization. At this time, the lack of computing resources becomes the main bottleneck. Especially when the number of cycles of the hot spot calculation part reaches thousands, tens of thousands or even millions of times, and the amount of calculation in each cycle is large, computing resources are particularly important. When computing resources are sufficient, the collection of the final results from the core and the data transmission with the main memory take a lot of time, affecting the execution efficiency of the program. For programs that join MPI for process-level optimization and have an unobvious acceleration effect, the method of the present invention proposes to apply for a multi-core group to allow the multi-core group to collaborate in computing, and optimize the data collection and transmission within and between the core groups, thereby reducing the number of master-slave core transmissions, thereby improving the execution efficiency and performance of the program. At the same time, the present invention also designs an automatic interface MCGO (Multi-Core Group Optimization), the master core interface is used to determine the number of core groups, and the slave core interface is used to perform multi-level aggregation and transmission of data. It is convenient for programmers to call and use directly.

[0009] The technical solution of the present invention is: Based on the new generation of Shenwei processors, the computing resource constraints and communication redundancy optimization methods include: First, the hotspot calculation part is analyzed to determine the number of hotspot calculation part loops and the number of core groups required for the calculation amount; if the code running time of the calculation part accounts for more than half of the total code running time, then the calculation part is determined to be a hotspot calculation part; Then, the multi-core group applies for and numbers the hotspot computing part, allowing the multi-core group to perform collaborative computing; Thirdly, after the collaborative computing is completed, the slave cores in the core group are grouped to optimize the data collection and transmission, and the data collection and transmission are then performed between the core groups; Finally, the final result is transferred back to main memory in a single pass.

[0010] Preferably, the multi-core group application and numbering according to the present invention includes: For a single-process program, when the number of cycles of the hotspot calculation part reaches more than one million times, apply for a multi-core group and start running, number the slave cores of the multi-core group in sequence, and then assign the hotspot calculation part to the numbered slave cores for calculation; For a program running in multiple processes, first, divide the tasks and assign them to multiple processes to run. Each process carries its own part of the task execution program. Then, each process assigns its own hot spot computing part to its own slave core for execution. Finally, perform the reduction calculation.

[0011] Further preferably, when applying for a multi-core group variable space, variables required for calculation are defined in each core group, and private LDM variables are defined.

[0012] Preferably, according to the present invention, determining the number of core groups required for the number of hot spot calculation part cycles and the amount of calculation is: predicting the number of core groups by training a linear regression model; including: 1) Extract data from three types of programs as training data; the three types of programs are , , Program, n represents the number of loop executions; O() is an asymptotic upper bound representation used to describe the changing trend of the running time or space requirements of the algorithm as the input size grows; It means that the running time of the algorithm is proportional to the input size; It means that the running time of the algorithm is proportional to the square of the input size; It indicates that the running time of the algorithm is proportional to the cube of the input size; the training data includes multiple samples, and the data of the three types of programs include features and target values, and the features include the number of loops and the computational complexity; the target value is the number of core groups required for each combination obtained from the actual test; by modifying the number of loops of the three types of programs, the number of loops and the computational complexity are arranged and combined, and the optimal number of core groups for each combination is tested, and multiple samples of training data are obtained; 2) Use the above training data to train a linear regression model. The formula of the linear regression model is as follows: ; in, is the predicted value, is the deviation term, is the weight of each feature, is the value of each feature; The training process includes: initializing the linear regression model according to the number of features, including weights and bias terms, with the initial values ​​of weights and bias terms set to 0; using the training data to train the linear regression model, adjusting the weights and bias terms through multiple iterations so that the linear regression model fits the training data; using the mean square error as the loss function, continuously optimizing the parameters of the linear regression model through the gradient descent method to obtain a trained linear regression model; 3) Input the new data into the trained linear regression model for prediction and output the predicted number of core groups.

[0013] Preferably, according to the present invention, the multi-core group performs collaborative computing on the hot spot computing part; including: When selecting 2-core group collaborative computing, the thread number range is 0-127, and the core group number range is 0-1. Each thread has its own corresponding core group number. According to the characteristics of the hot spot computing part, the computing tasks are allocated to each slave core. Specifically, the total task amount is divided by the total number of threads to obtain the specific task amount that each slave core needs to operate, and each slave core performs the calculation of its own part of the task. Each slave core obtains the required data from the master core through DMA or direct access to the main memory. Each slave core independently completes its own part of the task. The slave core calculations between the core groups do not interfere with each other. The calculation part is reduced and RMA transmission operations are performed within the core group, and the results are transmitted back to the master core. When selecting 3-core group collaborative computing, the thread number range is 0-191, and the core group number range is 0-2. Each thread has its own corresponding core group number. According to the characteristics of the hot spot computing part, the computing tasks are allocated to each slave core. Specifically, the total task amount is divided by the total number of threads to obtain the specific task amount that each slave core needs to operate, and each slave core performs the calculation of its own part of the task. Each slave core obtains the required data from the master core through DMA or direct access to the main memory. Each slave core independently completes its own part of the task. The slave core calculations between the core groups do not interfere with each other. The calculation part is reduced and RMA transmission operations are performed within the core group, and the results are transmitted back to the master core. When selecting 6-core group collaborative computing, the thread number range is 0-383, the core group number range is 0-5, and each thread has its own corresponding core group number; according to the characteristics of the hot spot computing part, the computing tasks are assigned to each slave core, specifically: the total task amount is divided by the total number of threads to obtain the specific task amount that each slave core needs to operate, and each slave core performs the calculation of its own part of the task; each slave core obtains the required data from the main core through DMA or direct access to the main memory. Each slave core independently completes its own part of the task, and the slave core calculations between the core groups do not interfere with each other. The calculation part is reduced and RMA transmission operations are performed within the core group, and the results are transmitted back to the main core.

[0014] Preferably, according to the present invention, when three core groups are used for collaborative optimization, the slave cores in the core group are grouped to optimize data collection and transmission, and data collection and transmission are performed between the core groups; including: First, when executing a multi-slave core task, the cores are divided into groups. Every four cores form a core group. The logical number of each core is 0, 1, 2, 3, and LDM continuous shared space is allocated to each core in the core group. After each core is calculated, the result is placed in the LDM continuous shared space. Each core group designates the core with the logical number 0 as the third-level summary core, which is responsible for collecting the calculation results of other cores in the group. Then, in each core group, slave core No. 0 is designated as the second-level summary slave core, which is responsible for summarizing the results of each third-level summary slave core in the core group; the specific operation is: each third-level summary slave core sends the result to the second-level summary slave core through the RMA operation; Finally, slave core No. 0 of core group number 0 is set as the first-level summary slave core, which is responsible for summarizing the results of the second-level summary slave cores of core groups numbered 1 and 2 to obtain the final calculation result, and then the first-level summary slave core transfers the final calculation result back to the main memory through DMA.

[0015] Preferably, according to the present invention, when the collaborative optimization is performed with two core groups, the slave cores in the core group are grouped to optimize the collection and transmission of data, and then the data is collected and transmitted between the core groups; including: First, when executing a multi-slave core task, the cores are divided into groups. Every four cores form a core group. The logical number of each core is 0, 1, 2, 3, and LDM continuous shared space is allocated to each core in the core group. After each core is calculated, the result is placed in the LDM continuous shared space. Each core group designates the core with the logical number 0 as the third-level summary core, which is responsible for collecting the calculation results of other cores in the group. Then, in each core group, slave core No. 0 is designated as the second-level summary slave core, which is responsible for summarizing the results of each third-level summary slave core in the core group; the specific operation is: each third-level summary slave core sends the result to the second-level summary slave core through the RMA operation; Finally, slave core No. 0 of core group number 0 is set as the first-level summary slave core, which is responsible for summarizing the results of the second-level summary slave cores of core group number 1 to obtain the final calculation result, and then the first-level summary slave core transfers the final calculation result back to the main memory through DMA.

[0016] Preferably, according to the present invention, when a 6-core group is used for collaborative optimization, the slave cores in the core group are grouped to optimize data collection and transmission, and data collection and transmission are performed between the core groups; including: First, when executing a multi-slave core task, the cores are divided into groups. Every four cores form a core group. The logical number of each core is 0, 1, 2, 3, and LDM continuous shared space is allocated to each core in the core group. After each core is calculated, the result is placed in the LDM continuous shared space. Each core group designates the core with the logical number 0 as the third-level summary core, which is responsible for collecting the calculation results of other cores in the group. Then, in each core group, slave core No. 0 is designated as the second-level summary slave core, which is responsible for summarizing the results of each third-level summary slave core in the core group; the specific operation is: each third-level summary slave core sends the result to the second-level summary slave core through the RMA operation; Finally, slave core No. 0 of core group number 0 is set as the first-level summary slave core, which is responsible for summarizing the results of the second-level summary slave cores of core groups numbered 1, 2, 3, 4, and 5 to obtain the final calculation result, and then the first-level summary slave core transfers the final calculation result back to the main memory through DMA.

[0017] Preferably according to the present invention, data collection and transmission within and between core groups are realized through the automation interface MCGO.

[0018] A computer device includes a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the steps of a method for optimizing computing resource constraints and communication redundancy based on a new generation of Shenwei processors are implemented.

[0019] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps of a method for optimizing computing resource constraints and communication redundancy based on a new generation of Shenwei processors.

[0020] The beneficial effects of the present invention are: 1. The present invention takes into account the situation of insufficient computing resources during single-process optimization. By applying for multi-core group collaborative optimization, the slave core optimization can utilize more computing resources, greatly improving the execution efficiency of the program.

[0021] 2. The present invention takes into account that after the slave core completes the calculation task, it takes a lot of time to reduce and return the result. By optimizing the aggregation and transmission operations within and between core groups, the program execution efficiency is improved.

[0022] 3. The present invention takes into account the selection of the number of multi-core groups, the cumbersome steps involved in multi-core group data aggregation, and the transmission optimization, and designs the MCGO automation interface to simplify user operations and improve user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 This is a schematic diagram of the hardware architecture of the SW26010pro processor; Figure 2 This is a schematic diagram corresponding to a single-process multi-core group; Figure 3 Schematic diagram of the process of predicting the number of nuclear groups by training a linear regression model; Figure 4 It is a schematic diagram of the collaborative computing process of a multi-core group; Figure 5 It is a schematic diagram of the three-level summary from the core; Figure 6 It is a schematic diagram of the secondary summary from the core; Figure 7 It is a schematic diagram of the first-level summary from the core. DETAILED DESCRIPTION

[0024] The present invention will be further defined below in conjunction with the accompanying drawings and embodiments, but is not limited thereto.

[0025] Example 1 Based on the new generation of Shenwei processors, the computing resource constraints and communication redundancy optimization methods include: First, analyze the hotspot calculation part to determine the number of loops and the number of core groups required for the calculation amount of the hotspot calculation part; if the code running time of the calculation part accounts for more than half of the total code running time, then the calculation part is determined to be the hotspot calculation part; the hotspot calculation part usually refers to the code or data processing area that requires high-frequency calculation and performance optimization. These "hot spots" are often the bottleneck of program performance and require special attention to optimization. Generally, they are manifested as a large number of loops and high computational complexity. There are generally two methods for analyzing the hotspot calculation part of the program: one is to use the Swprof performance analysis tool that comes with the Shenwei supercomputer, add the corresponding compilation options during compilation, run the program and output the specific function running time ratio of the program. The second is to use the manual stub method to insert a timer in the key code segment, output the running time of the code segment, and determine the code segment that takes a long time. If the running time of this part of the code accounts for more than half of the total code running time, then this part is determined to be the hotspot calculation part.

[0026] Then, the multi-core group applies for and numbers the hotspot computing part, allowing the multi-core group to perform collaborative computing; Thirdly, after the collaborative computing is completed, the slave cores in the core group are grouped to optimize the data collection and transmission, and the data collection and transmission are then performed between the core groups; Finally, the final result is transferred back to the main memory once, thus reducing the number of transmissions between the master and slave cores and greatly improving the program execution efficiency.

[0027] Example 2 The difference between the method for optimizing computing resource constraints and communication redundancy based on the new generation of Shenwei processors described in Example 1 is that: Multi-core group application and numbering; including: For a single-process program, when the number of cycles of the hotspot calculation part reaches more than one million times, a single core group will be very laborious to handle. Therefore, it is necessary to use multi-core group collaborative computing to allow more slave cores to participate in the calculation. Apply for a multi-core group and start running, number the slave cores of the multi-core group in sequence, and then assign the hotspot calculation part to the numbered slave cores for calculation; effectively improve the calculation efficiency. The corresponding schematic diagram of a single-process multi-core group is as follows Figure 2 shown.

[0028] For a program running in multiple processes, first, divide the tasks and assign them to multiple processes to run. Each process carries its own part of the task execution program. Then, each process assigns its own hot spot computing part to its own slave core for execution. Finally, perform the reduction calculation.

[0029] When applying for multi-core group variable space, define the variables required for calculation in each core group, and choose to define private LDM variables or cross-segment variables. Private LDM variables are a high-speed local local data storage space with a capacity of 256KB for each slave core of the Shenwei multi-core processor. LDM private space is a local private space that is quickly accessed by this slave core. The cross segment is a shared space between core groups on the same chip. Both the master and slave cores can access this distributed shared area of ​​the entire chip, and support cacheable and non-cacheable access to the main memory space. Due to the shared properties of the cross segment, if the required variables are defined as cross-segment variables, they need to be locked before and after the calculation to ensure the correctness of the calculation results. However, the locking operation will affect the running efficiency of the program, so choose to define private LDM variables here. Therefore, when applying for multi-core group variable space, define the variables required for calculation in each core group, and choose to define private LDM variables. Among them, local local data memory (LDM), each slave core of the Shenwei multi-core processor has a high-speed local local data storage space LDM, and the total capacity of LDM is 256KB. LDM space is mainly divided into private space and continuous shared space. LDM private space is a local private space that is quickly accessed by the slave core. LDM continuous shared space is a space used by the slave core for shared access to LDM in the array. The slave core can access the LDM space of other slave cores. LDM continuous shared space is a space used by the slave core for shared access to LDM in the array. The slave core can access the LDM space of other slave cores.

[0030] Determine the number of core groups required for the number of cycles and the amount of calculation for the hotspot calculation part; refers to: predicting the number of core groups by training a linear regression model; including: When selecting a specific number of core groups, it is usually necessary to analyze the number of cycles and computational complexity of the hotspot calculation part to estimate the number of core groups required. To determine the number of core groups more accurately, it is also necessary to run the program to test the execution effects of different numbers of core groups and compare them. To further simplify the process of determining the number of core groups, a machine learning model can be introduced to train a model to predict the optimal number of core groups and select the optimal configuration based on features such as the number of cycles and computational complexity.

[0031] 1) Extract data from three types of programs as training data; the three types of programs are , , Program, n is the input scale of the problem, which refers to the size or quantity of input data, indicating the number of elements or data that the algorithm needs to process, and here it represents the number of loop executions; O() is the notation of asymptotic upper bound, which is used to describe the changing trend of the running time or space requirement of the algorithm as the input scale grows; It means that the running time of the algorithm is proportional to the input size; It means that the running time of the algorithm is proportional to the square of the input size; The running time of the algorithm is proportional to the cube of the input size; these programs are selected based on their characteristics of having many loops and requiring a lot of computing resources, which is in line with the application scenario of the present invention. The training data includes multiple samples, and the data of the three types of programs include features and target values. The features include the number of loops and the computational complexity; the computational complexity is obtained by judging the structure of the code; for example, the complexity of a single-layer loop depends on the number of loops, and the loop is executed n times, so it is The complexity of a nested loop is the product of the number of inner and outer loops. The computational complexity is customized to a specific value, which is convenient for use as a parameter when calling a function. The target value is the number of core groups required for each combination obtained from the actual test; by modifying the number of loops of these three types of programs, the number of loops and the computational complexity are arranged and combined, and the optimal number of core groups for each combination is tested to obtain multiple samples of training data; the training data is used to train the linear regression model, and the new feature data is used to predict the number of core groups.

[0032] 2) Use the above training data to train a linear regression model. The formula of the linear regression model is as follows: ; in, is the predicted value, is the bias term (intercept), is the weight of each feature, is the value of each feature; The training process includes: initializing the linear regression model according to the number of features, including weights and bias terms, with the initial values ​​of weights and bias terms set to 0; using the training data to train the linear regression model, adjusting the weights and bias terms through multiple iterations so that the linear regression model can better fit the training data; using the mean square error as the loss function, continuously optimizing the parameters of the linear regression model through the gradient descent method, reducing the prediction error, and obtaining a trained linear regression model; 3) Input new data into the trained linear regression model for prediction and output the predicted number of core groups. After obtaining the appropriate number of core groups, apply for a specific number of core groups in the program to optimize the use of computing resources. The specific process is as follows: Figure 3 As shown: The multi-core group performs collaborative computing on the hotspot computing part, including: When selecting 2-core group collaborative computing, the thread number range is 0-127, the core group number range is 0-1, and each thread has its own corresponding core group number; according to the characteristics of the hot spot computing part, the computing tasks are reasonably allocated to each slave core, specifically: the total task amount is divided by the total number of threads to obtain the specific task amount that each slave core needs to operate, and each slave core performs the calculation of its own part of the task; each slave core obtains the required data from the main core through DMA or direct access to the main memory, and each slave core independently completes its own part of the task. The slave core calculations between each core group do not interfere with each other, and the calculation part is reduced and RMA transmission operations are performed within the core group, and the results are transmitted back to the main core; RMA is a remote data transmission operation performed between slave cores LDM within the core group.

[0033] When selecting 3-core group collaborative computing, the thread number range is 0-191, and the core group number range is 0-2. Each thread has its own corresponding core group number. According to the characteristics of the hot spot computing part, the computing tasks are reasonably allocated to each slave core. Specifically, the total task amount is divided by the total number of threads to obtain the specific task amount that each slave core needs to operate, and each slave core performs the calculation of its own part of the task. Each slave core obtains the required data from the master core through DMA or direct access to the main memory. Each slave core independently completes its own part of the task. The slave core calculations between the core groups do not interfere with each other. The calculation part is reduced and RMA transmission operations are performed within the core group, and the results are transmitted back to the master core. When selecting 6-core group collaborative computing, the thread number range is 0-383, the core group number range is 0-5, and each thread has its own corresponding core group number; according to the characteristics of the hot spot computing part, the computing tasks are reasonably allocated to each slave core, specifically: the total task amount is divided by the total number of threads to obtain the specific task amount that each slave core needs to operate, and each slave core performs the calculation of its own part of the task; each slave core obtains the required data from the main core through DMA or direct access to the main memory, and each slave core independently completes its own part of the task. The slave core calculations between the core groups do not interfere with each other, and the calculation part is reduced and RMA transmission operations are performed within the core group, and the results are transmitted back to the main core.

[0034] The specific process of multi-core group calculation is as follows Figure 4 As shown: When optimizing single-core and multi-core groups, the tasks processed are often critical resources, which may cause competition problems when multiple slave cores access the same storage area. To avoid competition errors, it is necessary to allocate an independent private LDM space to each slave core so that it can only process and update private variables. After the task is completed, each slave core needs to summarize or transmit the calculation results back to the master core, and then the master core performs the final calculation. However, this operation will increase the aggregation and transmission overhead in single-core and multi-core group environments, so the aggregation and transmission process needs to be further optimized.

[0035] When three core groups are used for collaborative optimization, the slave cores in the core group are grouped to optimize data collection and transmission, and data collection and transmission are then performed between core groups; including: First, when executing a multi-slave core task, the cores are divided into groups. Every four cores form a core group. The logic number of each core is 0, 1, 2, 3, and each core in the core group is allocated with LDM continuous shared space. After each core calculation is completed, the result is placed in the LDM continuous shared space. Each core group designates the core with the logic number 0 as the third-level summary core, which is responsible for collecting the calculation results of other cores in the group. Because the LDM continuous shared space is used in the group, the third-level summary core directly collects the data in the space. Figure 5 As shown: Then, in each core group, slave core No. 0 is designated as the second-level summary slave core, which is responsible for summarizing the results of each third-level summary slave core in this core group; the specific operation is: each third-level summary slave core sends the result to the second-level summary slave core through the RMA operation; Figure 6 As shown: Finally, the slave core No. 0 of the core group numbered 0 is set as the first-level summary slave core, which is responsible for summarizing the results of the second-level summary slave cores of the core group numbers 1 and 2 to obtain the final calculation result, and then the first-level summary slave core transfers the final calculation result back to the main memory through DMA. The specific operation is as follows Figure 7 shown.

[0036] When the two-core group is used for collaborative optimization, the slave cores in the core group are grouped to optimize data collection and transmission, and data collection and transmission are performed between the core groups; including: First, when executing a multi-slave core task, the slave cores are divided into groups. Every four slave cores form a slave core group. The logical number of each slave core is 0, 1, 2, 3, and LDM continuous shared space is allocated to the slave cores of each slave core group. After each slave core completes the calculation, the result is placed in the LDM continuous shared space. Each slave core group designates the slave core with the logical number 0 as the third-level summary slave core, which is responsible for collecting the calculation results of other slave cores in the group. Because the LDM continuous shared space is used within the group, the third-level summary slave core directly collects the data in this space.

[0037] Then, in each core group, slave core No. 0 is designated as the second-level summary slave core, which is responsible for summarizing the results of each third-level summary slave core in the core group; the specific operation is: each third-level summary slave core sends the result to the second-level summary slave core through the RMA operation; Finally, slave core No. 0 of core group number 0 is set as the first-level summary slave core, which is responsible for summarizing the results of the second-level summary slave cores of core group number 1 to obtain the final calculation result, and then the first-level summary slave core transfers the final calculation result back to the main memory through DMA.

[0038] When the 6-core group is used for collaborative optimization, the slave cores in the core group are grouped to optimize data collection and transmission, and data collection and transmission are then performed between core groups; including: First, when executing a multi-slave core task, the slave cores are divided into groups. Every four slave cores form a slave core group. The logical number of each slave core is 0, 1, 2, 3, and LDM continuous shared space is allocated to the slave cores of each slave core group. After each slave core completes the calculation, the result is placed in the LDM continuous shared space. Each slave core group designates the slave core with the logical number 0 as the third-level summary slave core, which is responsible for collecting the calculation results of other slave cores in the group. Because the LDM continuous shared space is used within the group, the third-level summary slave core directly collects the data in this space.

[0039] Then, in each core group, slave core No. 0 is designated as the second-level summary slave core, which is responsible for summarizing the results of each third-level summary slave core in this core group; the specific operation is: each third-level summary slave core sends the result to the second-level summary slave core through the RMA operation; Figure 6 As shown: Finally, slave core No. 0 of core group number 0 is set as the first-level summary slave core, which is responsible for summarizing the results of the second-level summary slave cores of core groups numbered 1, 2, 3, 4, and 5 to obtain the final calculation result, and then the first-level summary slave core transfers the final calculation result back to the main memory through DMA.

[0040] Data collection and transmission within and between core groups are achieved through the automated interface MCGO (Multi-Core Group Optimization).

[0041] In order to facilitate programmers to call the method of the present invention, an automation interface MCGO is designed. The automation interface MCGO has four functions, namely: initialization interface function, prediction core group number function, multi-level data aggregation and transmission function, and release interface function. The design of the automation interface MCGO is aimed at better applying multi-core group data aggregation and transmission optimization. The implementation of this interface reduces the difficulty of programming based on the Shenwei supercomputer, and programmers can directly call this interface to assist programming. The description of the automation interface MCGO is shown in Table 1: Table 1 MCGO interface description;

[0042] The example of calling the MCGO interface is as follows: / / Call MCGO interface host in the main core. c #include "MCGO.h" int main() { / / Define the required variables / / Initialize the interface MCGO_Init() / / Call interface MCGO_Master(loop_count, complexity, morecore_count); / / End interface MCGO_Finalize(); return 0; } Example 2 of calling the MCGO interface is as follows: / / Call MCGO interface slave.c in the main core #include "MCGO.h" int main() { / / Define the required variables / / Initialize the interface MCGO_Init() / / Call interface MCGO_Slave (tid, morecore_count, value, result); / / End interface MCGO_Finalize(); return 0; } By implementing the above method, the method of the present invention was tested.

[0043] First, the program is optimized for common slave cores, that is, the hot spot calculation part is divided into each slave core, and after each slave core completes the calculation, the result is transferred back to the main memory to obtain the common optimization running time; then, multi-core group collaborative optimization and transmission optimization are performed to obtain the invention optimization running time. Due to the hardware of the Shenwei processor, a single process can only run a maximum of 6 core groups, so the following is a test of a maximum of 6 core groups.

[0044] Table 2 is a comparison of the results of conventional optimization and optimization by the method of the present invention; Table 2 Prediction results of multi-core group;

[0045] Example 3 A computer device includes a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the steps of the method for optimizing computing resource constraints and communication redundancy based on a new generation of Shenwei processors described in Example 1 or 2 are implemented.

[0046] Example 4 A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the method for optimizing computing resource constraints and communication redundancy based on a new generation of Shenwei processors as described in Example 1 or 2.

Claims

1. Based on the new generation of Shenwei processors, the computing resource limitation and communication redundancy optimization method is characterized by: include: First, the hotspot calculation part is analyzed to determine the number of hotspot calculation part loops and the number of core groups required for the calculation amount; if the code running time of the calculation part accounts for more than half of the total code running time, then the calculation part is determined to be a hotspot calculation part; Then, the multi-core group applies for and numbers the hotspot computing part, allowing the multi-core group to perform collaborative computing; Thirdly, after the collaborative computing is completed, the slave cores in the core group are grouped to optimize the data collection and transmission, and the data collection and transmission are then performed between the core groups; Finally, the final result is transferred back to main memory in a single pass.

2. The method for optimizing computing resource constraints and communication redundancy based on a new generation of Shenwei processors according to claim 1 is characterized in that: Multi-core group application and numbering; including: For a single-process program, when the number of cycles of the hotspot calculation part reaches more than one million times, apply for a multi-core group and start running, number the slave cores of the multi-core group in sequence, and then assign the hotspot calculation part to the numbered slave cores for calculation; For a program running in multiple processes, first, divide the tasks and assign them to multiple processes to run. Each process carries its own part of the task execution program. Then, each process assigns its own hot spot computing part to its own slave core for execution. Finally, perform the reduction calculation.

3. The method for optimizing computing resource constraints and communication redundancy based on a new generation of Shenwei processors according to claim 2 is characterized in that: When applying for multi-core group variable space, define the variables required for calculation in each core group and choose to define private LDM variables.

4. The method for optimizing computing resource constraints and communication redundancy based on a new generation of Shenwei processors according to claim 1 is characterized in that: Determine the number of core groups required for the number of cycles and the amount of calculation for the hotspot calculation part; refers to: predicting the number of core groups by training a linear regression model; including: 1) Extract data from three types of programs as training data; the three types of programs are , , Program, n represents the number of loop executions; O() is an asymptotic upper bound representation used to describe the changing trend of the running time or space requirements of the algorithm as the input size grows; It means that the running time of the algorithm is proportional to the input size; It means that the running time of the algorithm is proportional to the square of the input size; It indicates that the running time of the algorithm is proportional to the cube of the input size; the training data includes multiple samples, and the data of the three types of programs include features and target values, and the features include the number of loops and the computational complexity; the target value is the number of core groups required for each combination obtained from the actual test; by modifying the number of loops of the three types of programs, the number of loops and the computational complexity are arranged and combined, and the optimal number of core groups for each combination is tested, and multiple samples of training data are obtained; 2) Use the above training data to train a linear regression model. The formula of the linear regression model is as follows: ； in, is the predicted value, is the deviation term, is the weight of each feature, is the value of each feature; The training process includes: initializing the linear regression model according to the number of features, including weights and bias terms, with the initial values ​​of weights and bias terms set to 0; using the training data to train the linear regression model, adjusting the weights and bias terms through multiple iterations so that the linear regression model fits the training data; using the mean square error as the loss function, continuously optimizing the parameters of the linear regression model through the gradient descent method to obtain a trained linear regression model; 3) Input the new data into the trained linear regression model for prediction and output the predicted number of core groups.

5. The method for optimizing computing resource constraints and communication redundancy based on a new generation of Shenwei processors according to claim 1 is characterized in that: The multi-core group performs collaborative computing on the hotspot computing part, including: When selecting 2-core group collaborative computing, the thread number range is 0-127, and the core group number range is 0-1. Each thread has its own corresponding core group number. According to the characteristics of the hot spot computing part, the computing tasks are allocated to each slave core. Specifically, the total task amount is divided by the total number of threads to obtain the specific task amount that each slave core needs to operate, and each slave core performs the calculation of its own part of the task. Each slave core obtains the required data from the master core through DMA or direct access to the main memory. Each slave core independently completes its own part of the task. The slave core calculations between the core groups do not interfere with each other. The calculation part is reduced and RMA transmission operations are performed within the core group, and the results are transmitted back to the master core. When selecting 3-core group collaborative computing, the thread number range is 0-191, and the core group number range is 0-2. Each thread has its own corresponding core group number. According to the characteristics of the hot spot computing part, the computing tasks are allocated to each slave core. Specifically, the total task amount is divided by the total number of threads to obtain the specific task amount that each slave core needs to operate, and each slave core performs the calculation of its own part of the task. Each slave core obtains the required data from the master core through DMA or direct access to the main memory. Each slave core independently completes its own part of the task. The slave core calculations between the core groups do not interfere with each other. The calculation part is reduced and RMA transmission operations are performed within the core group, and the results are transmitted back to the master core. When selecting 6-core group collaborative computing, the thread number range is 0-383, the core group number range is 0-5, and each thread has its own corresponding core group number; according to the characteristics of the hot spot computing part, the computing tasks are assigned to each slave core, specifically: the total task amount is divided by the total number of threads to obtain the specific task amount that each slave core needs to operate, and each slave core performs the calculation of its own part of the task; each slave core obtains the required data from the main core through DMA or direct access to the main memory. Each slave core independently completes its own part of the task, and the slave core calculations between the core groups do not interfere with each other. The calculation part is reduced and RMA transmission operations are performed within the core group, and the results are transmitted back to the main core.

6. The method for optimizing computing resource constraints and communication redundancy based on a new generation of Shenwei processors according to claim 5 is characterized in that: When three core groups are used for collaborative optimization, the slave cores in the core group are grouped to optimize data collection and transmission, and data collection and transmission are then performed between core groups; including: First, when executing a multi-slave core task, the cores are divided into groups. Every four cores form a core group. The logical number of each core is 0, 1, 2, 3, and LDM continuous shared space is allocated to each core in the core group. After each core is calculated, the result is placed in the LDM continuous shared space. Each core group designates the core with the logical number 0 as the third-level summary core, which is responsible for collecting the calculation results of other cores in the group. Then, in each core group, slave core No. 0 is designated as the second-level summary slave core, which is responsible for summarizing the results of each third-level summary slave core in the core group; the specific operation is: each third-level summary slave core sends the result to the second-level summary slave core through the RMA operation; Finally, slave core No. 0 of core group number 0 is set as the first-level summary slave core, which is responsible for summarizing the results of the second-level summary slave cores of core group numbers 1 and 2 to obtain the final calculation result, and then the first-level summary slave core transfers the final calculation result back to the main memory through DMA.

7. The method for optimizing computing resource constraints and communication redundancy based on a new generation of Shenwei processors according to claim 5 is characterized in that: When the two-core group is used for collaborative optimization, the slave cores in the core group are grouped to optimize data collection and transmission, and data collection and transmission are performed between the core groups; including: First, when executing a multi-slave core task, the cores are divided into groups. Every four cores form a core group. The logical number of each core is 0, 1, 2, 3, and LDM continuous shared space is allocated to each core in the core group. After each core is calculated, the result is placed in the LDM continuous shared space. Each core group designates the core with the logical number 0 as the third-level summary core, which is responsible for collecting the calculation results of other cores in the group. Then, in each core group, slave core No. 0 is designated as the second-level summary slave core, which is responsible for summarizing the results of each third-level summary slave core in the core group; the specific operation is: each third-level summary slave core sends the result to the second-level summary slave core through the RMA operation; Finally, slave core No. 0 of core group number 0 is set as the first-level summary slave core, which is responsible for summarizing the results of the second-level summary slave cores of core group number 1 to obtain the final calculation result, and then the first-level summary slave core transfers the final calculation result back to the main memory through DMA.

8. The method for optimizing computing resource constraints and communication redundancy based on a new generation of Shenwei processors according to claim 5 is characterized in that: When the 6-core group is used for collaborative optimization, the slave cores in the core group are grouped to optimize data collection and transmission, and data collection and transmission are performed between core groups; including: First, when executing a multi-slave core task, the cores are divided into groups. Every four cores form a core group. The logical number of each core is 0, 1, 2, 3, and LDM continuous shared space is allocated to each core in the core group. After each core is calculated, the result is placed in the LDM continuous shared space. Each core group designates the core with the logical number 0 as the third-level summary core, which is responsible for collecting the calculation results of other cores in the group. Then, in each core group, slave core No. 0 is designated as the second-level summary slave core, which is responsible for summarizing the results of each third-level summary slave core in the core group; the specific operation is: each third-level summary slave core sends the result to the second-level summary slave core through the RMA operation; Finally, slave core No. 0 of core group number 0 is set as the first-level summary slave core, which is responsible for summarizing the results of the second-level summary slave cores of core groups numbered 1, 2, 3, 4, and 5 to obtain the final calculation result, and then the first-level summary slave core transfers the final calculation result back to the main memory through DMA.

9. The method for optimizing computing resource constraints and communication redundancy based on a new generation of Shenwei processors according to any one of claims 1 to 8, characterized in that: Data collection and transmission within and between nuclear groups are realized through the automated interface MCGO.

Images