A method for automatically optimizing the performance and accuracy of Ascend AI processors under the CANN computing framework

Abstract

The present invention discloses a method for automatically optimizing the performance and accuracy of the Ascend AI processor under the CANN computing architecture, comprising the following steps: Step 1, obtaining all operators in the neural network and performing binary encoding according to the data type of the operator; Step 2, initializing the candidate solution population and generating a preset number of candidate solutions; Step 3, fitness evaluation, training the neural network with the generated candidate solution settings to obtain performance data. If the performance data and overflow data of the candidate solution individual meet the preset requirements, the inverse of its performance data is used as its fitness; Step 4, selecting the candidate solution for crossover and mutation to generate the next generation population; Step 5, judging whether the preset termination condition is met. If so, outputting the best solution and optimizing the performance and accuracy of the Ascend AI processor according to the best solution; If not, returning to Step 3. Through the present invention, it is possible to achieve the goal of improving accuracy as much as possible while meeting performance requirements.

Description

Technical Field

[0001] The present invention relates to the field of AI processors, and specifically to a method for automatically optimizing the performance and accuracy of a Shengteng AI processor under a CANN computing architecture. Background Art

[0002] CANN (Compute Architecture for Neural Networks) is a unified heterogeneous computing architecture specifically designed for AI scenarios. It unleashes the immense computing power of Ascend hardware and, by providing multi-level programming interfaces, enables users to quickly build AI applications and businesses based on the Ascend platform. It supports mainstream front-end frameworks (such as MindSpore, TensorFlow, and PyTorch), while shielding users from hardware differences across various chip families. Computations can be performed on multiple AI processors (NPUs) on the Ascend platform, meeting users' comprehensive AI needs with its all-scenario, low-barrier-to-entry, and high-performance advantages. The Ascend 910 AI processor is a dedicated neural network processor for artificial intelligence (AI), boasting a computing power of up to 256 teraflops, with the latest model boasting up to 310 teraflops, double the computing power of mainstream processors in the industry. It is compatible with the MindSpore training framework. Currently, most deep neural network training scripts in the industry are developed based on TensorFlow's Python API and run on CPUs, GPUs, and TPUs by default. To enable it to utilize the NPU's powerful computing power to perform training and improve training performance, CANN provides the function of using TensorFlow to perform model adaptation training on the NPU.

[0003] During neural network training, operators are the basic units that support network calculations. Common operators include convolution, pooling, and matrix multiplication. Most deep learning model operators use 32-bit single-precision floating-point (FP32) data types. However, due to hardware limitations and software implementation differences, CANN prefers 16-bit floating-point (FP16) operators and requires certain operators (such as Conv2D) to be FP16. Using FP16 for operations can speed up the processor, but data overflow is inevitable, which can reduce model accuracy and even disrupt the model training process. Using FP32 for operations, while ensuring accuracy, increases the computational overhead of the NPU, impacting performance.

[0004] Performance and accuracy are two important indicators for measuring the adaptation and training effect of deep neural networks on NPUs. Performance refers to the training speed of the network model on the NPU, while accuracy refers to the performance of the model on the corresponding task after training (taking image classification as an example, the accuracy at this time is the classification accuracy of the model). Ideally, the performance of training on the NPU should be better than that of training on the GPU, and the accuracy of training on the NPU should be on par with that obtained by training on the GPU. However, when CANN uses mixed precision to train neural networks, the precision specified for each operator in the network may be unreasonable, which leads to the inability to meet the requirements of both performance and accuracy. To solve the above situation, CANN provides automatic mixed precision, training mode and loss scaling functions.

[0005] While these two features help improve the performance and accuracy of NPU neural network training, the problem isn't completely resolved. Some float32 operators may still experience substandard performance due to adaptation issues that take up a significant amount of time during training, and some float16 operators may experience substandard accuracy due to data overflow. In these cases, users are required to manually modify the blacklist and whitelist of mixed-precision operators. The whitelist allows operators of the current float32 type to be reduced in precision to float16, while the blacklist does not. Users can use the blacklist and whitelist to adjust the built-in optimization strategy, specifying which operators are allowed to reduce precision and which are not, thereby achieving the required performance and accuracy. In existing technical solutions, the user adjusts the blacklist and whitelist of operators based on performance data and overflow data collected during training, determining which operators should be added to the whitelist and which to the blacklist, thereby optimizing the performance and accuracy of neural network training on the Ascend AI processor. This optimization method requires users to have extensive knowledge and experience in Ascend AI processor operator problems, and requires continuous manual debugging to optimize performance and accuracy, which is very energy-consuming for users. If knowledge and experience are insufficient, it is possible to go around in circles between accuracy and performance, resulting in situations where accuracy increases but performance decreases, or performance increases but accuracy decreases. Summary of the Invention

[0006] The purpose of the present invention is to overcome the deficiencies of the prior art and provide a method for automatically optimizing the performance and accuracy of Shengteng AI processors under the CANN computing architecture, comprising the following steps:

[0007] Step 1: Get all operators in the neural network and perform binary encoding according to the data type of the operator;

[0008] Step 2: Initialize the candidate solution population and generate a preset number of candidate solutions;

[0009] Step 3: Fitness evaluation: Use the generated candidate solution settings to train the neural network and obtain performance data. If the performance data and overflow data of the candidate solution meet the preset requirements, the inverse of its performance data is used as its fitness.

[0010] Step 4: Select candidate solutions for crossover and mutation to generate the next generation population;

[0011] Step 5: Determine whether the preset termination conditions are met. If so, output the optimal solution and optimize the performance and accuracy of the Ascend AI processor based on the optimal solution. If not, return to step 3.

[0012] Furthermore, the initialization of the candidate solution population to generate a preset number of candidate solutions includes the following process:

[0013] S1, turn on CANN's automatic mixed precision and enable loss scaling, binary encode the current operator data type settings, and generate an initial individual;

[0014] S2, transforming each position of the initial candidate solution with a preset probability to generate a new candidate solution;

[0015] S3, repeat S2 until the number of generated candidate solutions reaches the preset requirement.

[0016] Furthermore, the fitness evaluation is to train the neural network with the generated candidate solution settings to obtain performance data. If the performance data and overflow data of the candidate solution individual meet the preset requirements, the inverse of the performance data is used as its fitness, which includes the following process:

[0017] S1. First, the neural network is trained with the candidate solution to obtain performance data.

[0018] S2. If the performance does not meet the standard, the fitness of the candidate solution individual is directly set to negative. The candidate solution with negative fitness will be screened out during the selection process;

[0019] If the performance meets the standard, check its overflow data. If continuous overflow data occurs, it means that the operator data type setting corresponding to the current candidate solution individual is inappropriate. Set the operator type of the first overflow data to float32, and then return to S1 to re-check the performance data;

[0020] S3. If the performance data and overflow data of the candidate solution individual meet the requirements, the inverse of its performance data is taken as its fitness.

[0021] Furthermore, the selection of candidate solutions for crossover and mutation to generate the next generation population includes the following process:

[0022] After obtaining the fitness of the current candidate solution population, the candidate solution individual selection process is first carried out. According to the fitness of the individuals in the population, a set number of candidate solution individuals with high fitness are selected from the current population and placed in the next generation population. These candidate solutions are used as parent individuals for crossover and mutation operations to generate a set number of new offspring individuals and place them in the next generation population.

[0023] Furthermore, the crossover operation using these candidate solutions as parent individuals includes the following process:

[0024] 1) First, randomly pair the parents.

[0025] 2) Randomly set the location of the intersection;

[0026] 3) Interchange the coded segments between the binary coded intersections of the paired parental individuals.

[0027] Furthermore, the mutation operation includes:

[0028] 1) Randomly generate mutation points;

[0029] 2) According to the mutation probability, the original gene fragment of the mutation point is reversed.

[0030] Furthermore, the termination condition is: the number of iterations reaches the set number of times 2, the settings corresponding to several candidate solutions with the highest fitness in the population are selected to perform complete neural network training, and complete performance and accuracy data are obtained. If the performance and accuracy meet the standards at the same time, the algorithm is terminated and the candidate solution is output to complete the performance and accuracy tuning.

[0031] The present invention provides a method for automatically modifying operator data formats within the CANN computing architecture to optimize the performance and accuracy of Ascend AI processors. This method allows users who lack domain knowledge and experience to optimize their own models without excessive effort. A genetic algorithm is used to find the optimal compromise between performance and accuracy, maximizing accuracy while meeting performance requirements. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 A flowchart of a method for automatically optimizing the performance and accuracy of Shengteng AI processors under the CANN computing architecture;

[0033] Figure 2 Schematic diagram of gene coding;

[0034] Figure 3 Initialize the flow chart for the candidate population;

[0035] Figure 4 Flowchart for evaluating the fitness of candidate solutions;

[0036] Figure 5 Schematic diagram of cross operation;

[0037] Figure 6 Schematic diagram of mutation operation. DETAILED DESCRIPTION

[0038] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings, but the protection scope of the present invention is not limited to the following.

[0039] In order to make the objectives, technical solutions, and advantages of the present invention more clearly understood, the present invention is further described in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only intended to explain the present invention and are not intended to limit the present invention. That is, the embodiments described herein are only some embodiments of the present invention, not all embodiments. Generally, the components of the embodiments of the present invention described and illustrated in the drawings herein may be arranged and designed in various different configurations.

[0040] Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but merely represents selected embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without making any creative work shall fall within the scope of protection of the present invention. It should be noted that relational terms such as "first" and "second" are merely used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations.

[0041] Moreover, the terms "comprises," "comprising," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, article, or apparatus that includes a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not preclude the presence of additional identical elements in the process, method, article, or apparatus that includes the element.

[0042] The features and performance of the present invention are further described in detail below with reference to the embodiments.

[0043] like Figure 1 As shown, a method for automatically optimizing the performance and accuracy of Shengteng AI processors under the CANN computing architecture includes the following steps:

[0044] Step 1: Get all operators in the neural network and perform binary encoding according to the data type of the operator;

[0045] Step 2: Initialize the candidate solution population and generate a preset number of candidate solutions;

[0046] Step 3: Fitness evaluation: The neural network is trained using the generated candidate solutions to obtain performance data. If the performance data and overflow data of the candidate solution meet the preset requirements, the inverse of the performance data is used as its fitness. The performance data is the time it takes for the neural network to train one step (seconds per step).

[0047] Step 4: Select candidate solutions for crossover and mutation to generate the next generation population;

[0048] Step 5: Determine whether the preset termination conditions are met. If so, output the optimal solution and optimize the performance and accuracy of the Ascend AI processor based on the optimal solution. If not, return to step 3.

[0049] The initialization of the candidate solution population to generate a preset number of candidate solutions includes the following process:

[0050] S1, turn on CANN's automatic mixed precision and enable loss scaling, binary encode the current operator data type settings, and generate an initial individual;

[0051] S2, transforming each position of the initial candidate solution with a preset probability to generate a new candidate solution;

[0052] S3, repeat S2 until the number of generated candidate solutions reaches the preset requirement.

[0053] The fitness evaluation is to train the neural network with the generated candidate solution settings to obtain performance data. If the performance data and overflow data of the candidate solution individual meet the preset requirements, the inverse of the performance data is used as its fitness, which includes the following process:

[0054] S1. First, the neural network is trained with the candidate solution to obtain performance data.

[0055] S2: If the performance does not meet the standard, the fitness of the candidate solution is directly set to negative. Candidate solutions with negative fitness will be screened out during the selection process. The performance does not meet the standard when the time spent training the neural network step on the NPU (Ascend AI Processor) is longer than the time spent training on the GPU.

[0056] If the performance meets the standard, check its overflow data. If continuous overflow data occurs, it means that the operator data type setting corresponding to the current candidate solution individual is inappropriate. Set the operator type of the first overflow data to float32, and then return to S1 to re-check the performance data;

[0057] S3. If the performance data and overflow data of the candidate solution individual meet the requirements, the inverse of its performance data is taken as its fitness.

[0058] The selection of candidate solutions for crossover and mutation to generate the next generation population includes the following process:

[0059] After obtaining the fitness of the current candidate solution population, the candidate solution individual selection process is first performed. Based on the fitness of the individuals in the population, a set number of candidate solution individuals with high fitness are selected from the current population and placed into the next generation population. These candidate solutions are used as parents to perform crossover and mutation operations to generate a set number of new offspring individuals to be placed into the next generation population. Here, the high fitness refers to a fitness value greater than a set threshold.

[0060] The crossover operation using these candidate solutions as parent individuals includes the following process:

[0061] The crossover operation using these candidate solutions as parent individuals includes the following process:

[0062] 1) First, randomly pair the parents.

[0063] 2) Randomly set the location of the intersection;

[0064] 3) Swap the coded segments between the binary coded intersections of the paired parent individuals

[0065] The mutation operation includes:

[0066] 1) Randomly generate mutation points

[0067] 2) According to the mutation probability, the original gene fragment of the mutation point is reversed.

[0068] Furthermore, the termination condition is: the number of iterations reaches the set number of times 2, the settings corresponding to several candidate solutions with the highest fitness in the population are selected to perform complete neural network training, and complete performance and accuracy data are obtained. If the performance and accuracy meet the standards at the same time, the algorithm is terminated and the candidate solution is output to complete the performance and accuracy tuning.

[0069] Specifically, the genetic encoding scheme uses a genetic algorithm to automatically adjust the data types of all operators in the network, thereby achieving the goal of automatically optimizing performance and accuracy. The algorithm's final output should be a set of settings that include all operator data types in the network. Therefore, it is first necessary to binary encode the settings of all operator data types as genes for candidate solutions of the genetic algorithm.

[0070] The specific implementation is as follows:

[0071] Assume that the current network contains the following operators: MatMulV2, L2Loss, Conv2D, MaxPool, AvgPool, TransData (for the sake of convenience, only 6 operators are listed; in reality, there may be more operators). Their corresponding data types are: float32, float16, float16, float32, float32, float16. After binary encoding, the chromosome "100110" is formed, where 1 represents the float32 data type and 0 represents the float16 data type. Figure 2 Gene coding scheme shown.

[0072] 2. Initialize the candidate solution population

[0073] In order to use the genetic algorithm to search for a solution that meets the performance and accuracy requirements, it is necessary to initialize a candidate solution population for subsequent search work. The initialization process of the candidate solution population is as follows: 1. Turn on the automatic mixed precision of CANN and enable the loss scaling function, set the current operator data type to encode as mentioned in the first section, and generate an initial individual. 2. A single candidate solution is not enough to constitute the initial population, so we need to transform each position of the initial candidate solution with a preset probability to generate a new candidate solution. 3. Repeat the second step until the number of generated candidate solutions reaches the preset requirement. Figure 3 The candidate population initialization flowchart is shown.

[0074] 3. Fitness evaluation

[0075] The purpose of fitness evaluation is to provide a quantitative measure to determine which candidate solutions qualify as parents to generate new candidate solutions. The specific evaluation process is as follows: First, the neural network is trained with the candidate solution settings to obtain performance data (this process only requires a few training steps and is very time-consuming).

[0076] If the performance does not meet the standard, the fitness of the candidate solution individual is directly set to negative, and the candidate solution with negative fitness will be screened out during the selection process.

[0077] If the performance meets the standard, its overflow data is checked. If continuous overflow data occurs, it means that the operator data type corresponding to the current candidate solution individual is not properly set. The first (when continuous data overflow occurs, the overflow of the subsequent operator is likely due to the overflow of the output of the previous operator. It is impossible to determine whether the overflow of the current operator is due to its data format or the forward operator. Therefore, a conservative strategy is adopted to solve the overflow problem of the forward operator first.) The operator type with overflow data is set to float32 (that is, the binary number at the corresponding position in the candidate solution gene is set to 1), and then return to the first step to re-check the performance data.

[0078] If the performance data and overflow data of the candidate solution meet the requirements, the inverse of its performance data is used as its fitness (the reason for this is that the performance data refers to the time required for each iteration of network training, and the smaller the time, the better the performance). Figure 4 Candidate solution fitness evaluation flow chart

[0079] 4. Select candidate solutions for crossover and mutation

[0080] After obtaining the fitness of the current candidate solution population, the candidate solution individual selection process is first carried out. According to the fitness of the individuals in the population, a certain number of candidate solution individuals with high fitness are selected from the current population through roulette and other methods and placed in the next generation population. These candidate solutions are used as parent individuals for crossover and mutation operations to generate a certain number of new offspring individuals and place them in the next generation population.

[0081] The specific steps of the crossover operation are as follows:

[0082] 1) First, randomly pair the parents

[0083] 2) Randomly set the intersection position

[0084] 3) Exchange of gene segments between paired chromosomes

[0085] like Figure 5 Schematic diagram of crossover operation shown

[0086] The specific steps of the mutation operation are as follows: 1) randomly generate mutation points; 2) invert the original gene fragment according to the mutation probability (0 to 1, 1 to 0), such as Figure 6 Mutation operation diagram

[0087] 5. Suspension condition judgment

[0088] After completing a round of iterative evolution according to the above steps, it is necessary to judge the termination conditions. There are two criteria for judgment: 1. The number of iterations reaches the specified number. 2. Select the settings corresponding to the several candidate solutions with the highest fitness in the population to conduct complete neural network training to obtain complete performance and accuracy data. If both performance and accuracy meet the standards, the algorithm is terminated and the candidate solution is output to complete the performance and accuracy tuning. Otherwise, the next round of iterations is carried out. If the algorithm has not obtained a solution that meets both performance and accuracy requirements after reaching the number of iterations, it means that the performance or accuracy problem of the current network is likely not caused by the operator data type, and the user needs to conduct further troubleshooting.

[0089] The foregoing description is merely a preferred embodiment of the present invention. It should be understood that the present invention is not limited to the form disclosed herein and should not be construed as excluding other embodiments. Rather, the present invention can be used in various other combinations, modifications, and environments and can be modified within the scope of the concept described herein through the above teachings or techniques or knowledge in the relevant field. Modifications and variations made by those skilled in the art that do not depart from the spirit and scope of the present invention are intended to be protected by the appended claims.

Claims

1. A method for automatically optimizing the performance and accuracy of Shengteng AI processors under the CANN computing architecture, characterized in that: The process includes the following steps: Step 1: Get all operators in the neural network and perform binary encoding according to the data type of the operator; Step 2: Initialize the candidate solution population and generate a preset number of candidate solutions; Step 3: Fitness evaluation: The neural network is trained with the generated candidate solutions to obtain performance data. If the performance data and overflow data of the candidate solution meet the preset threshold, the inverse of its performance data is used as its fitness. Step 4: Select candidate solutions for crossover and mutation to generate the next generation population; Step 5: Determine whether the preset termination conditions are met. If so, output the optimal solution and optimize the performance and accuracy of the Ascend AI processor based on the optimal solution. If not, return to step 3. The initialization of the candidate solution population to generate a preset number of candidate solutions includes the following process: S1, turn on the automatic mixed precision of CANN and enable the loss scaling function, binary encode the operator data type settings of all operators in the neural network, and generate an initial individual; S2, transforming each position of the initialized candidate solution with a preset probability to generate a new candidate solution; S3, repeat S2 until the number of generated candidate solutions reaches the preset requirement; The selection of candidate solutions for crossover and mutation to generate the next generation population includes the following process: After obtaining the fitness of the current candidate solution population, the candidate solution individual selection process is first carried out. According to the fitness of the individuals in the population, a set number of candidate solution individuals with high fitness are selected from the current population and placed in the next generation population. These candidate solutions are used as parent individuals for crossover and mutation operations to generate a set number of new offspring individuals and place them in the next generation population. The crossover operation using these candidate solutions as parent individuals includes the following process: 1) First, randomly pair the parents. 2) Randomly set the location of the intersection; 3) Interchange the coded segments between the binary coded intersections of the paired parental individuals.

2. The method for automatically optimizing the performance and accuracy of a Shengteng AI processor under a CANN computing architecture according to claim 1 is characterized in that: The fitness evaluation is to train the neural network with the generated candidate solutions to obtain performance data. If the performance data and overflow data of the candidate solution meet the preset threshold, the inverse of the performance data is used as its fitness. The process includes the following: S1. First, the neural network is trained with the candidate solution to obtain performance data. S2. If the performance does not meet the standard, the fitness of the candidate solution individual is directly set to negative. The candidate solution with negative fitness will be screened out during the selection process; If the performance meets the standard, check its overflow data. If continuous overflow data occurs, it means that the operator data type setting corresponding to the current candidate solution individual is inappropriate. Set the operator type of the first overflow data to float32, and then return to S1 to re-check the performance data; S3. If the performance data and overflow data of the candidate solution individual meet the preset threshold, the inverse of the performance data of the candidate solution individual is used as the fitness of the candidate solution individual.

3. The method for automatically optimizing the performance and accuracy of a Shengteng AI processor under a CANN computing architecture according to claim 1 is characterized in that: The mutation operation includes: 1) Randomly generate mutation points; 2) Invert the original coding segment of the mutation point according to the mutation probability.

4. The method for automatically optimizing the performance and accuracy of a Shengteng AI processor under a CANN computing architecture according to claim 3 is characterized in that: The termination conditions are as follows: the number of iterations reaches the set number 2, the settings corresponding to the several candidate solutions with the highest fitness in the population are selected for complete neural network training to obtain complete performance and accuracy data. If both performance and accuracy meet the standards, the algorithm is terminated and the candidate solution is output to complete the performance and accuracy tuning.

Images