Operation unit, system on chip and model quantization search method

Abstract

Embodiments of the present application provide an operation unit, a system on a chip and a model quantization search method. The operation unit comprises: an initialization subunit, configured to set initial quantization types of each network layer in a to-be-quantized model, and encode the initial quantization types to obtain an initial quantization sequence; an evolutionary iteration subunit, configured to perform a mutation operation on the initial quantization sequence to obtain a mutated quantization sequence; obtain quantization inference results corresponding to each candidate quantization model; determine a candidate quantization sequence from the initial quantization sequence and the mutated quantization sequence according to similarities between the quantization inference results and an original inference result; take the candidate quantization sequence as a new initial quantization sequence, and return to perform the mutation operation on the initial quantization sequence to obtain the mutated quantization sequence until an iteration stop condition is met; and a result determination subunit, configured to determine a target quantization sequence from the candidate quantization sequence to obtain a search result. The present scheme can improve the efficiency of model quantization search.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a computing unit, a system-on-a-chip, and a model quantization search method. Background Technology

[0002] With the rapid development of artificial intelligence technology, deep learning models have been widely used in many fields. However, as the number of model layers increases, the number of parameters in deep learning models also increases, which limits their running speed and consumes a large amount of memory.

[0003] Model quantization is an important method for accelerating model development. It involves low-bit encoding of data from each network layer during the model deployment phase (e.g., quantizing model data of type float32 into data of type int8 or int16, where float32 is a single-precision floating-point number represented by 32 bits: 1 bit for the sign bit, 8 bits for the exponent, and 23 bits for the mantissa; int is a signed integer data type, with int16 represented by 16 bits and int8 by 8 bits). This reduces model file size and speeds up the inference process. Currently, because quantization introduces quantization loss, when the quantized model cannot meet accuracy requirements, hybrid quantization is typically used. This involves using different quantization types for different network layers within the model.

[0004] Since the quantization type is not unique (such as int8 or int16 mentioned above), and the model usually has multiple network layers, there are many kinds of quantization schemes for a single model. Therefore, it is necessary to perform a model quantization search to determine the quantized model with smaller accuracy difference from the original model and better performance from among many quantization schemes.

[0005] The relevant quantization search scheme involves performing complete model inference for each possible quantization scheme, and then determining the final quantization scheme based on the quantization accuracy loss. This search method requires traversing all possible quantization schemes, resulting in a large number of searches and high search costs, thus leading to low search efficiency. For example, for a deep learning model with 5 network layers, each layer can be quantized using either uint8 (unint is an unsigned integer data type, and uint8 is represented by 8 bits) or int16 quantization types. This results in 32 possible quantization schemes. Following the above search scheme, 32 complete model inferences would be required, leading to high search costs and low search efficiency. Summary of the Invention

[0006] In view of this, embodiments of this application provide a computing unit, a system-on-a-chip, and a model quantization search method to at least partially solve the above-mentioned problems.

[0007] According to a first aspect of the embodiments of this application, a computing unit is provided, including:

[0008] An initialization subunit is used to set the initial quantization type of each network layer in the model to be quantized, and to encode the initial quantization type to obtain an initial quantization sequence;

[0009] An evolutionary iteration subunit is used to perform a mutation operation on the initial quantization sequence to obtain a mutated quantization sequence; obtain the quantization inference results corresponding to each candidate quantization model; determine candidate quantization sequences from the initial quantization sequence and the mutated quantization sequence based on the similarity between each quantization inference result and the original inference result; use the candidate quantization sequence as a new initial quantization sequence, and return to the step of performing a mutation operation on the initial quantization sequence to obtain a mutated quantization sequence, until the iteration stopping condition is met; wherein, the candidate quantization model includes: the quantization model corresponding to the initial quantization sequence and the quantization model corresponding to the mutated quantization sequence;

[0010] The result determination subunit is used to determine the target quantization sequence from the candidate quantization sequences and obtain the search results.

[0011] According to a second aspect of the embodiments of this application, a processor is provided, comprising:

[0012] The arithmetic unit as described in the first aspect above;

[0013] A high-speed cache memory is used to cache data.

[0014] According to a third aspect of the embodiments of this application, a computing device is provided, comprising:

[0015] The processor as described in the second aspect above;

[0016] A memory, coupled to the processor, is used to store data output or read by the processor.

[0017] According to a fourth aspect of the embodiments of this application, an on-chip system is provided, comprising:

[0018] The processor as described in the second aspect above;

[0019] A memory, coupled to the processor, is used to store data output or read by the processor.

[0020] According to a fifth aspect of the embodiments of this application, a data center is provided, comprising:

[0021] The computing device as described in the third aspect above and / or the system-on-a-chip as described in the fourth aspect above.

[0022] According to a sixth aspect of the embodiments of this application, a model quantization search method is provided, comprising:

[0023] Set the initial quantization type for each network layer in the model to be quantized, and encode the initial quantization type to obtain the initial quantization sequence;

[0024] Perform a mutation operation on the initial quantized sequence to obtain a mutated quantized sequence;

[0025] Obtain the quantization inference results corresponding to each candidate quantization model; determine candidate quantization sequences from the initial quantization sequence and the mutated quantization sequence based on the similarity between each quantization inference result and the original inference result; the candidate quantization model includes: the quantization model corresponding to the initial quantization sequence and the quantization model corresponding to the mutated quantization sequence;

[0026] The candidate quantization sequence is used as a new initial quantization sequence. The step of performing mutation operation on the initial quantization sequence to obtain the mutated quantization sequence is returned until the iteration stop condition is met.

[0027] The target quantization sequence is determined from the candidate quantization sequences, and the search results are obtained.

[0028] According to a seventh aspect of the present application, a computer storage medium is provided that stores a computer program thereon, which, when executed by a processor, implements the method described in the sixth aspect above.

[0029] According to the computing unit, on-chip system, and model quantization search method provided in this application embodiment, after initially setting the initial quantization type of each network layer, an evolutionary algorithm is used to simulate the natural biological evolutionary selection process: the encoded initial quantization sequence is mutated, and then, based on the similarity between the quantization inference result and the true inference result (original inference result) in the initial quantization sequence and the mutated quantization sequence, the quantization sequence is optimized and selected, and the above mutation-optimization selection process is repeated iteratively until the target quantization sequence is obtained. Then, the model to be quantized can be quantized based on the target quantization sequence. The search method provided in this application embodiment uses an evolutionary algorithm. The search process considers the individual differences of the sequence, performs sequence mutation, and then infers the quantization model (quantization scheme) corresponding to the mutated sequence. It can optimize and select the quantization scheme that meets the accuracy requirements based on the difference between the inference result and the true result, without having to infer for each possible quantization scheme (quantization model). Therefore, the number of inferences is reduced, that is, the number of searches is shortened, the search cost is reduced, and the search efficiency is improved. Attached Figure Description

[0030] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings.

[0031] Figure 1 This is a hardware structure diagram of a system-on-a-chip according to an embodiment of this application;

[0032] Figure 2 This is a structural block diagram of a computing unit according to Embodiment 1 of this application;

[0033] Figure 3 This is a flowchart illustrating the steps of a model quantization search method according to Embodiment 2 of this application;

[0034] Figure 4 This is a schematic diagram of the scenario corresponding to Embodiment 2 of this application. Detailed Implementation

[0035] To enable those skilled in the art to better understand the technical solutions in the embodiments of this application, the technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art should fall within the protection scope of the embodiments of this application. It should be noted that, unless otherwise specified, the embodiments and features in the embodiments of this application can be combined with each other.

[0036] Background Overview:

[0037] Model quantization is an important method for accelerating models. It involves low-bit conversion of data from each network layer to reduce model file size and speed up the inference process. To ensure that the quantized model meets accuracy requirements, a hybrid quantization approach is typically used, where different quantization types are applied to different network layers. Due to the variety of quantization types and the large number of network layers in a model, there are usually many quantization schemes for a specific deep learning model. Therefore, a model quantization search is necessary to identify the quantized model that has the smallest accuracy difference from the original model and offers superior performance.

[0038] Relevant quantization search methods typically involve exhaustively listing all possible quantization schemes, performing complete model inference for each scheme using quantization calibration data, and then determining the quantization scheme based on the accuracy loss between the obtained quantization inference results and the original inference results. Therefore, the above-mentioned quantization search methods are costly and inefficient.

[0039] Application environment of this application:

[0040] This application provides a model quantization search scheme, which is relatively general and independent of the deployed hardware environment. For illustrative purposes, the following description will primarily focus on a system-on-a-chip (SoC) application scenario. Those skilled in the art should understand that the embodiments of this disclosure can also be applied to other application scenarios.

[0041] See Figure 1 , Figure 1 The hardware structure diagram of a system-on-a-chip according to an embodiment of this application is shown. The system-on-a-chip 100 includes a memory 110, a processing unit cluster 120, and a computing unit cluster 130 connected by a bus. The processing unit cluster 120 includes a plurality of processing units 122; the computing unit cluster 130 includes a plurality of computing units 132. The processing units 122 are units that perform conventional processing (data processing not based on model quantization search results). The processing units 122 can take various forms such as a central processing unit (CPU), an application-specific integrated circuit (ASIC), or a field-programmable gate array (FPGA). The computing units 132 in this application are dedicated computing units designed to improve the efficiency of model quantization search.

[0042] The arithmetic unit 132 receives the scheduling from the processing unit 122. Specifically, the processing unit 122 can send a model quantization search instruction to the arithmetic unit 132. This instruction contains the address information of the model to be quantized in the memory 110 or the internal storage unit of the arithmetic unit 132. After receiving the instruction, the arithmetic unit 132 will obtain the model to be quantized based on the address information. Then, it will perform a model quantization search based on the obtained data to obtain the search results, so that the arithmetic unit 132 or the processing unit 122 located outside the arithmetic unit 132 can perform subsequent processing based on the results obtained by the arithmetic unit 132.

[0043] General overview of this application:

[0044] To address the issues of high search costs and low search efficiency in the aforementioned related technologies, this application provides an alternative model hybrid quantization search method. Specifically, after initially setting the initial quantization type for each network layer, an evolutionary algorithm is used to simulate the natural biological evolutionary selection process: the encoded initial quantization sequence undergoes a mutation operation. Then, based on the similarity between the quantization inference result and the true inference result (original inference result) in the initial quantization sequence and the mutated quantization sequence, the quantization sequence is optimized and selected. This mutation-optimization selection process is repeated iteratively until a target quantization sequence is obtained. The model to be quantized can then be quantized based on this target quantization sequence. The search scheme provided in this application uses an evolutionary algorithm. The search process considers individual differences in the sequence, performs sequence mutation, and then infers the quantization model (quantization scheme) corresponding to the mutated sequence. This allows for the optimization and selection of a quantization scheme that meets the accuracy requirements based on the difference between the inference result and the true result, without needing to infer for every possible quantization scheme (quantization model). Therefore, the number of inferences is reduced, which shortens the search time, lowers the search cost, and improves search efficiency.

[0045] Detailed implementation process of the embodiments of this application:

[0046] The specific implementation of the embodiments of this application will be further described below with reference to the accompanying drawings.

[0047] Example 1

[0048] Reference Figure 2 , Figure 2 This is a structural block diagram of a computing unit according to Embodiment 1 of this application. The computing unit corresponds to... Figure 1 The arithmetic unit 132 in the middle includes:

[0049] Initialization subunit 1322 is used to set the initial quantization type of each network layer in the model to be quantized, and to encode the initial quantization type to obtain the initial quantization sequence.

[0050] The evolutionary iteration subunit 1324 is used to perform mutation operations on the initial quantization sequence to obtain a mutated quantization sequence; obtain the quantization inference results corresponding to each candidate quantization model; determine candidate quantization sequences from the initial quantization sequence and the mutated quantization sequence based on the similarity between each quantization inference result and the original inference result; use the candidate quantization sequence as a new initial quantization sequence, and return to the step of performing mutation operations on the initial quantization sequence to obtain a mutated quantization sequence, until the iteration stopping condition is met; wherein, the candidate quantization model includes: the quantization model corresponding to the initial quantization sequence and the quantization model corresponding to the mutated quantization sequence.

[0051] The result determines subunit 1326, which is used to determine the target quantization sequence from the candidate quantization sequences and obtain the search results.

[0052] Specifically:

[0053] In this embodiment, the initial quantization type is selected from a set of preset quantization types. For example, if there are two preset quantization types: uint8 and int16, then the initial quantization type of each network layer is either uint8 or int16.

[0054] In this embodiment, the specific method for setting the initial quantization type of the initialization subunit 1322 is not limited, and can be determined according to the actual situation. For example, the initial quantization type of each network layer in the model to be quantized can be set randomly; or the initial quantization type of each network layer in the model to be quantized can be set according to a preset setting rule. In this embodiment, the specific content of the above setting rule is also not limited.

[0055] Optionally, in some embodiments, when the initialization subunit 1322 performs the step of setting the initial quantization type of each network layer in the model to be quantized, it may specifically be used for:

[0056] Obtain the raw output data of each network layer in the model to be quantized. The raw output results are the data output by each network layer when the quantization calibration data is input into the model to be quantized.

[0057] Each network layer in the model to be quantized is quantized using the benchmark quantization type to obtain the benchmark quantization model; the quantization output data of each network layer in the benchmark quantization model is obtained, wherein the quantization output data is the data output by each network layer when the quantization calibration data is input into the benchmark quantization model;

[0058] For each network layer, if the similarity between the original output data and the quantized output data is greater than a preset second similarity threshold, then the baseline quantization type is determined as the initial quantization type of that network layer; otherwise, the non-baseline quantization type is determined as the initial quantization type of that network layer.

[0059] Specifically, quantization calibration data can be data used to calibrate the accuracy of the searched quantization model. For example, if the model to be quantized is a neural network model for face recognition, the quantization calibration data can be an image containing faces; if the model to be quantized is a neural network model for lane lines, the quantization calibration data can be an image containing lane lines, and so on. The baseline quantization type is a quantization type pre-defined from multiple preset quantization types. For example, if there are two preset quantization types: uint8 and int16, then uint8 or int16 can be set as the baseline quantization type. When uint8 is set as the baseline quantization type, then int16 is a non-baseline quantization type; conversely, uint8 is a non-baseline quantization type. Typically, the baseline quantization type can be set based on the resource availability of the hardware where the model to be quantized will be deployed.

[0060] In this embodiment, the specific method for encoding the initial quantization type of the initialization subunit 1322 is not limited, and can be set according to the actual situation. Specifically, different symbols (such as numbers, letters, characters, etc.) can be used to represent different quantization types. For example, if the quantization type is uint8 and int16 (i.e., the model is quantized using a mixture of uint8 and int16), binary encoding can be used, such as using 0 to represent uint8 and 1 to represent int16, thus obtaining an initial quantization sequence composed of 0 and 1; or A can be used to represent uint8 and B to represent int16, thus obtaining an initial quantization sequence composed of A and B, and so on.

[0061] Optionally, in some embodiments, after performing a mutation operation on the initial quantization sequence to obtain a mutated quantization sequence, the evolution iteration subunit 1324 can also be used to perform a crossover operation on the initial quantization sequence and the mutated quantization sequence to obtain a crossover quantization sequence.

[0062] Correspondingly, the candidate quantization model may further include: the quantization model corresponding to the cross-quantization sequence. When performing the step of determining the candidate quantization sequence from the initial quantization sequence and the mutated quantization sequence, the evolutionary iteration subunit 1324 may specifically be used to: determine the candidate quantization sequence from the initial quantization sequence, the mutated quantization sequence, and the cross-quantization sequence.

[0063] In the evolutionary iteration process of this application, not only is the initial quantization model mutated, but the mutated quantization sequences obtained from the initial quantization model are also crossovered. This effectively expands the selection range of candidate quantization sequences, making the search process closer to the natural biological evolutionary selection process. Therefore, while improving search efficiency, it further enhances search performance and narrows the accuracy and performance differences between the searched quantization model and the original model.

[0064] In this embodiment, the mutation operation can refer to the operation of mutating (changing) each element in the initial quantization sequence according to a preset mutation rule, that is, changing an element in the initial quantization sequence to another encoded element; the crossover operation can refer to the operation of swapping an element in the initial quantization sequence to the corresponding position in the mutated quantization sequence according to a preset crossover rule. In this embodiment, the specific mutation rule corresponding to the mutation operation and the specific crossover rule for the crossover operation are not limited, and can be set according to the actual situation.

[0065] Optionally, in some embodiments, when performing the step of performing a mutation operation on the initial quantized sequence to obtain a mutated quantized sequence, the evolutionary iteration subunit 1324 may specifically be used for:

[0066] For each element in the initial quantization sequence, a random number generation operation is performed. Based on the relationship between each generated first random value and the preset mutation probability value, it is determined whether each element in the initial quantization sequence meets the mutation condition. In the initial quantization sequence, the elements that meet the mutation condition are mutated, while the elements that do not meet the mutation condition remain unchanged, thus obtaining the mutated quantization sequence.

[0067] Optionally, in some embodiments, when the evolutionary iteration subunit 1324 performs the step of determining whether each element in the initial quantization sequence satisfies the mutation condition based on the magnitude relationship between each generated first random value and a preset mutation probability value, it may specifically be used to:

[0068] If the generated first random value is greater than the preset mutation probability value, then the element corresponding to the first random value in the initial quantization sequence is determined to satisfy the mutation condition; otherwise, the element corresponding to the first random value is determined not to satisfy the mutation condition.

[0069] Optionally, in some embodiments, when performing the step of performing a crossover operation on the initial quantization sequence and the mutated quantization sequence to obtain a crossover quantization sequence, the evolutionary iteration subunit 1324 may specifically be used for:

[0070] For each element in the mutated quantization sequence, a random number generation operation is performed. Based on the relationship between each generated second random value and the preset crossover probability value, it is determined whether each element in the mutated quantization sequence satisfies the crossover condition. In the mutated quantization sequence, the elements that satisfy the crossover condition are swapped with the corresponding elements in the initial quantization sequence, while the elements that do not satisfy the crossover condition remain unchanged, thus obtaining the crossover quantization sequence.

[0071] Optionally, in some embodiments, when the evolutionary iteration subunit 1324 performs the step of determining whether each element in the mutation quantization sequence satisfies the crossover condition based on the magnitude relationship between each generated second random value and a preset crossover probability value, it may specifically be used to:

[0072] If the generated second random value is greater than the preset crossover probability value, then the element in the mutation quantization sequence corresponding to the second random value is determined to satisfy the crossover condition; otherwise, the element corresponding to the second random value is determined not to satisfy the crossover condition.

[0073] Furthermore, the number of mutation operations performed by the evolutionary iteration subunit 1324 during a single iteration can be multiple, and correspondingly, the number of crossover operations can also be multiple. To avoid the problem of too many mutation quantization sequences due to a large number of mutation operations, resulting in too many complete inference attempts, in this embodiment, the number of obtained mutation quantization sequences can be preset to not exceed a first preset number. Similarly, to avoid the problem of too many crossover quantization sequences due to a large number of crossover operations, resulting in too many complete inference attempts, in this embodiment, the number of obtained crossover quantization sequences can be preset to not exceed a second preset number. In addition, to avoid the problem of too many initial quantization sequences in the next iteration, resulting in too many complete inference attempts, in this embodiment, the number of determined candidate quantization sequences can be preset to not exceed a third preset number. In this embodiment, the specific values ​​of the first, second, and third preset numbers are not limited.

[0074] When the evolutionary iteration subunit 1324 obtains the quantization inference results corresponding to each candidate quantization model, it can specifically: input the quantization calibration data into each candidate quantization model respectively, perform complete model inference, and thus obtain the quantization inference results corresponding to each candidate quantization model.

[0075] The original inference result in the embodiments of this application can refer to the inference result obtained by inputting quantized calibration data into the original unquantized model to be quantized and performing complete model inference.

[0076] In this embodiment of the application, the determination criteria for the result determination subunit 1326 when determining the target quantization sequence are not limited, and can be set according to the actual situation. For example, the target quantization sequence can be determined from each candidate quantization sequence according to the similarity between the quantization inference result corresponding to each candidate quantization sequence and the original inference result; or a target quantization sequence can be randomly selected from multiple candidate quantization sequences obtained in the last iteration process, and so on.

[0077] Optionally, in some embodiments, the result determination subunit 1326 can determine the candidate quantization sequence among the candidate quantization sequences whose quantization inference result is close to the original inference result as the target quantization sequence.

[0078] After determining the target quantization sequence, the quantization scheme corresponding to the target quantization sequence can be determined in reverse according to the previous encoding rules. Then, the model to be quantized is quantized based on the quantization scheme to obtain the target quantization model.

[0079] In this embodiment, the specific content of the iteration stopping condition is not limited and can be set according to the actual situation. Optionally, in some embodiments, the iteration stopping condition may include: the number of iterations reaches a preset threshold, or the similarity between the quantized inference result and the original inference result reaches a preset first similarity threshold, etc.

[0080] The processing unit provided in this application embodiment, after initially setting the initial quantization type of each network layer, uses an evolutionary algorithm to simulate the natural biological evolutionary selection process: the encoded initial quantization sequence undergoes a mutation operation; then, based on the similarity between the quantization inference result and the true inference result (original inference result) in the initial quantization sequence and the mutated quantization sequence, the quantization sequence is optimized and selected; this mutation-optimization selection process is repeated iteratively until a target quantization sequence is obtained, and the model to be quantized can then be quantized based on the target quantization sequence. The search method provided in this application embodiment uses an evolutionary algorithm. The search process, considering individual differences in the sequence, performs sequence mutation, and then infers the quantization model (quantization scheme) corresponding to the mutated sequence. This allows for the optimized selection of a quantization scheme that meets the accuracy requirements based on the difference between the inference result and the true result, without needing to infer for every possible quantization scheme (quantization model). Therefore, the number of inferences is reduced, which shortens the search time, reduces the search cost, and improves search efficiency.

[0081] Example 2

[0082] Reference Figure 3 , Figure 3 This is a flowchart illustrating the steps of a model quantization search method according to Embodiment 2 of this application. Specifically, the method includes the following steps:

[0083] Step 302: Set the initial quantization type of each network layer in the model to be quantized, and encode the initial quantization type to obtain the initial quantization sequence.

[0084] Optionally, in some embodiments, this step may include:

[0085] Obtain the raw output data of each network layer in the model to be quantized; the raw output results are the data output by each network layer when the quantization calibration data is input into the model to be quantized.

[0086] Each network layer in the model to be quantized is quantized using the baseline quantization type to obtain the baseline quantization model;

[0087] Obtain the quantization output data of each network layer in the benchmark quantization model; the quantization output data is the data output by each network layer when the quantization calibration data is input into the benchmark quantization model.

[0088] For each network layer, if the similarity between the original output data and the quantized output data is greater than a preset second similarity threshold, then the baseline quantization type is determined as the initial quantization type of that network layer; otherwise, the non-baseline quantization type is determined as the initial quantization type of that network layer.

[0089] Step 304: Perform a mutation operation on the initial quantization sequence to obtain the mutated quantization sequence.

[0090] Optionally, in some embodiments, the mutation quantification sequence can be obtained in the following manner:

[0091] For each element in the initial quantized sequence, perform a random number generation operation;

[0092] Based on the relationship between each generated first random value and the preset mutation probability value, determine whether each element in the initial quantization sequence satisfies the mutation condition;

[0093] In the initial quantization sequence, elements that meet the mutation conditions are mutated, while elements that do not meet the mutation conditions remain unchanged, resulting in a mutated quantization sequence.

[0094] Furthermore, when determining whether each element in the initial quantized sequence satisfies the mutation condition based on the relationship between the generated first random values ​​and the preset mutation probability values, it is possible to:

[0095] If the generated first random value is greater than the preset mutation probability value, then the element corresponding to the first random value in the initial quantization sequence is determined to satisfy the mutation condition; otherwise, the element corresponding to the first random value is determined not to satisfy the mutation condition.

[0096] Step 306: Obtain the quantization inference results corresponding to each candidate quantization model. The candidate quantization models include: the quantization model corresponding to the initial quantization sequence and the quantization model corresponding to the mutated quantization sequence.

[0097] Step 308: Based on the similarity between each quantization inference result and the original inference result, determine the candidate quantization sequence from the initial quantization sequence and the mutated quantization sequence.

[0098] Optionally, in some embodiments, after obtaining the mutated quantization sequence in step 304, a crossover operation can be performed on the initial quantization sequence and the mutated quantization sequence to obtain a cross-quantization sequence. Correspondingly, the candidate quantization model in step 306 can further include the quantization model corresponding to the cross-quantization sequence. Thus, step 308 can include determining a candidate quantization sequence from the initial quantization sequence, the mutated quantization sequence, and the cross-quantization sequence.

[0099] Furthermore, when performing a crossover operation on the initial quantized sequence and the mutated quantized sequence to obtain a cross-quantized sequence, the following can be done:

[0100] For each element in the mutated quantized sequence, perform a random number generation operation;

[0101] Based on the relationship between each generated second random value and the preset crossover probability value, determine whether each element in the mutation quantization sequence satisfies the crossover condition;

[0102] In the mutated quantization sequence, elements that satisfy the crossover condition are swapped with the corresponding elements in the initial quantization sequence, while elements that do not satisfy the crossover condition remain unchanged, resulting in a cross-quantization sequence.

[0103] Furthermore, when determining whether each element in the mutation quantization sequence satisfies the crossover condition based on the relationship between the generated second random values ​​and the preset crossover probability value, it is possible to:

[0104] If the generated second random value is greater than the preset crossover probability value, then the element in the mutation quantization sequence corresponding to the second random value is determined to satisfy the crossover condition; otherwise, the element corresponding to the second random value is determined not to satisfy the crossover condition.

[0105] Step 310: Use the candidate quantization sequence as the new initial quantization sequence and return to step 304 until the iteration stopping condition is met.

[0106] In other words, in step 310, it can be determined whether the preset iteration stop condition is met. If not, the candidate quantization sequence is used as the new initial quantization sequence, and the process returns to step 304 to continue the iteration process. If the condition is met, the candidate quantization sequence determined in step 308 is no longer used as the new initial quantization sequence to continue the iteration process. Instead, the iteration stops, and the subsequent step 312 is executed.

[0107] Optionally, in some embodiments, the iteration stopping condition may include: the number of iterations reaching a preset number threshold, or the similarity between the quantized inference result and the original inference result reaching a preset first similarity threshold.

[0108] Step 312: Determine the target quantization sequence from the candidate quantization sequences to obtain the search results.

[0109] See Figure 4 , Figure 4 This is a schematic diagram of the scenario corresponding to Embodiment 2 of this application. The following will refer to... Figure 4 The schematic diagram shown illustrates an embodiment of this application using a specific scenario example:

[0110] Assuming the model to be quantized is a neural network model with four layers, see [link / reference]. Figure 4 The four network layers are: convolutional layer 1, activation layer, convolutional layer 2, and normalization layer; there are two preset quantization types: uint8 and int16.

[0111] Step 302 sets the initial quantization type of convolutional layer 1 to int16, activation layer to uint8, convolutional layer 2 to uint8, and normalization layer to uint8. The encoding rule used is: 0 represents uint8 and 1 represents int16, resulting in the initial quantization sequence [1,0,0,0]. Step 304 performs two or more mutation operations on the initial quantization sequence to obtain two mutated quantization sequences: mutated quantization sequence 1: [1,1,0,1] and mutated quantization sequence 2: [0,1,1,1]. Additionally, a crossover operation is performed on the initial quantization sequence and the mutated quantization sequence to obtain two crossover quantization sequences: crossover quantization sequence 1: [1,0,0,1] and crossover quantization sequence 2: [0,1,0,0]. Step 306 obtains the values ​​corresponding to the initial quantization sequence, mutated quantization sequence 1, mutated quantization sequence 2, crossover quantization sequence 1, and crossover quantization sequence 2. The quantization inference results of the model are obtained. In step 308, based on the similarity between the above quantization inference results and the original inference results, two candidate quantization sequences are determined from the initial quantization sequence, mutated quantization sequence 1, mutated quantization sequence 2, cross-quantization sequence 1, and cross-quantization sequence 2: [1,1,0,1] and [1,0,0,1]. Then, it is determined whether the iteration stopping condition is met. If not, the two candidate quantization sequences determined above: [1,1,0,1] and [1,0,0,1] are used as new initial quantization sequences, and the process returns to step 304 to continue the iteration process. If the iteration stopping condition is met, step 312 is executed, and finally the target quantization sequence is determined from the determined candidate quantization sequences. The determined target quantization sequence is: [1,0,1,1]. The search results are as follows: Convolutional layer 1 uses int16 quantization, activation layer uses uint8 quantization, convolutional layer 2 uses int16 quantization, and normalization layer uses int16 quantization.

[0112] The model quantization search method provided in this application, after initially setting the initial quantization type for each network layer, employs an evolutionary algorithm to simulate the natural biological evolutionary selection process: The encoded initial quantization sequence undergoes a mutation operation. Then, based on the similarity between the quantization inference result and the true inference result (original inference result) between the initial quantization sequence and the mutated quantization sequence, the quantization sequence is optimized and selected. This mutation-optimization selection process is iterated repeatedly until a target quantization sequence is obtained. The model to be quantized can then be quantized based on this target quantization sequence. The search method provided in this application uses an evolutionary algorithm. The search process, considering individual differences in the sequence, performs sequence mutation and then infers the quantization model (quantization scheme) corresponding to the mutated sequence. This allows for the optimized selection of a quantization scheme that meets the accuracy requirements based on the difference between the inference result and the true result, without needing to infer for every possible quantization scheme (quantization model). Therefore, it reduces the number of inferences, shortens the search time, reduces search costs, and improves search efficiency.

[0113] The model quantization search method of this embodiment has the same beneficial effects as the computing unit provided in Embodiment 1 above, and will not be repeated here. Furthermore, the implementation of each step in the model quantization search method of this embodiment can refer to the description of the corresponding part in Embodiment 1 above, and will not be repeated here either.

[0114] It should be noted that the user-related information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to sample data used for training the model, data used for analysis, stored data, displayed data, etc.) involved in the embodiments of this application are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation entry points are provided for users to choose to authorize or refuse.

[0115] Furthermore, it should be noted that, depending on the implementation needs, the various components / steps described in the embodiments of this application can be broken down into more components / steps, or two or more components / steps or parts of the operation of components / steps can be combined into new components / steps to achieve the purpose of the embodiments of this application.

[0116] Those skilled in the art will recognize that the units and method steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the embodiments of this application.

[0117] The above embodiments are only used to illustrate the embodiments of this application, and are not intended to limit the embodiments of this application. Those skilled in the art can make various changes and modifications without departing from the spirit and scope of the embodiments of this application. Therefore, all equivalent technical solutions also fall within the scope of the embodiments of this application, and the patent protection scope of the embodiments of this application should be defined by the claims.

Claims

1. A computing unit, comprising: Initialize the sub-unit to obtain the raw output data of each network layer in the model to be quantized; The original output data is the data output by each network layer when the quantization calibration data is input into the model to be quantized; Each network layer in the model to be quantized is quantized using a benchmark quantization type to obtain a benchmark quantization model; the quantization output data of each network layer in the benchmark quantization model is then obtained. The quantization output data refers to the data output by each network layer when the quantization calibration data is input into the benchmark quantization model. For each network layer, if the similarity between the original output data and the quantized output data is greater than a preset second similarity threshold, then the baseline quantization type is determined as the initial quantization type of that network layer. Otherwise, the non-benchmark quantization type is determined as the initial quantization type for this network layer; The initial quantization type is encoded to obtain the initial quantization sequence; An evolutionary iteration subunit is used to perform a mutation operation on the initial quantization sequence to obtain a mutated quantization sequence; and to obtain the quantization inference results corresponding to each candidate quantization model. Based on the similarity between each quantization inference result and the original inference result, candidate quantization sequences are determined from the initial quantization sequence and the mutated quantization sequence; the candidate quantization sequence is used as a new initial quantization sequence, and the step of performing mutation operation on the initial quantization sequence to obtain the mutated quantization sequence is returned to be executed until the iteration stopping condition is met; wherein, the candidate quantization model includes: the quantization model corresponding to the initial quantization sequence and the quantization model corresponding to the mutated quantization sequence; The result determination subunit is used to determine the target quantization sequence from the candidate quantization sequences and obtain the search results; Wherein, the model to be quantized is a neural network model for face recognition, and the quantization calibration data is an image containing a face; or, the model to be quantized is a neural network model for lane lines, and the quantization calibration data is an image containing lane lines.

2. The arithmetic unit according to claim 1, wherein, After performing the mutation operation on the initial quantized sequence to obtain the mutated quantized sequence, the evolutionary iteration subunit is further used for: Perform a crossover operation on the initial quantization sequence and the mutated quantization sequence to obtain a cross-quantization sequence; The candidate quantization model further includes: the quantization model corresponding to the cross-quantization sequence; When the evolutionary iteration subunit performs the step of determining candidate quantization sequences from the initial quantization sequence and the mutated quantization sequence, it is specifically used for: Candidate quantization sequences are determined from the initial quantization sequence, the mutated quantization sequence, and the crossover quantization sequence.

3. The arithmetic unit according to claim 1 or 2, wherein, When the evolutionary iteration subunit performs the step of performing a mutation operation on the initial quantized sequence to obtain a mutated quantized sequence, it is specifically used for: For each element in the initial quantized sequence, a random number generation operation is performed. Based on the relationship between each generated first random value and the preset mutation probability value, it is determined whether each element in the initial quantization sequence satisfies the mutation condition. In the initial quantization sequence, elements that meet the mutation conditions are mutated, while elements that do not meet the mutation conditions remain unchanged, resulting in a mutated quantization sequence.

4. The arithmetic unit according to claim 3, wherein, When the evolutionary iteration subunit performs the step of determining whether each element in the initial quantized sequence satisfies the mutation condition based on the relationship between each generated first random value and the preset mutation probability value, it is specifically used for: If the generated first random value is greater than the preset mutation probability value, then the element in the initial quantization sequence corresponding to the first random value is determined to satisfy the mutation condition; otherwise, the element corresponding to the first random value is determined not to satisfy the mutation condition.

5. The arithmetic unit according to claim 2, wherein, When the evolutionary iteration subunit performs the step of performing a crossover operation on the initial quantized sequence and the mutated quantized sequence to obtain a crossover quantized sequence, it is specifically used for: For each element in the mutation quantization sequence, a random number generation operation is performed respectively; Based on the relationship between each generated second random value and the preset crossover probability value, it is determined whether each element in the mutation quantization sequence satisfies the crossover condition; In the mutated quantization sequence, elements that satisfy the crossover condition are swapped with elements at corresponding positions in the initial quantization sequence, while elements that do not satisfy the crossover condition remain unchanged, resulting in a cross-quantization sequence.

6. The arithmetic unit according to claim 5, wherein, When the evolutionary iteration subunit performs the step of determining whether each element in the mutation quantization sequence satisfies the crossover condition based on the relationship between each generated second random value and the preset crossover probability value, it is specifically used for: If the generated second random value is greater than the preset crossover probability value, then the element in the mutation quantization sequence corresponding to the second random value is determined to satisfy the crossover condition; otherwise, the element corresponding to the second random value is determined not to satisfy the crossover condition.

7. The arithmetic unit according to claim 1, wherein, The iteration stopping conditions include: the number of iterations reaches a preset threshold, or the similarity between the quantized reasoning result and the original reasoning result reaches a preset first similarity threshold.

8. A processor, comprising: The arithmetic unit as described in any one of claims 1-7; A high-speed cache memory is used to cache data.

9. A computing device, comprising: The processor as described in claim 8; A memory, coupled to the processor, is used to store data output or read by the processor.

10. A system-on-a-chip, comprising: The processor as described in claim 8; A memory, coupled to the processor, is used to store data output or read by the processor.

11. A data center, comprising: The computing device as claimed in claim 9 and / or the system-on-a-chip as claimed in claim 10.

12. A model quantization search method, comprising: Obtain the raw output data of each network layer in the model to be quantized; The original output data is the data output by each network layer when the quantization calibration data is input into the model to be quantized; Each network layer in the model to be quantized is quantized using a benchmark quantization type to obtain a benchmark quantization model; the quantization output data of each network layer in the benchmark quantization model is then obtained. The quantization output data refers to the data output by each network layer when the quantization calibration data is input into the benchmark quantization model. For each network layer, if the similarity between the original output data and the quantized output data is greater than a preset second similarity threshold, then the baseline quantization type is determined as the initial quantization type of that network layer. Otherwise, the non-benchmark quantization type is determined as the initial quantization type for this network layer; The initial quantization type is encoded to obtain the initial quantization sequence; Perform a mutation operation on the initial quantized sequence to obtain a mutated quantized sequence; Obtain the quantitative inference results corresponding to each candidate quantitative model; Based on the similarity between each quantization inference result and the original inference result, candidate quantization sequences are determined from the initial quantization sequence and the mutated quantization sequence; The candidate quantization model includes: the quantization model corresponding to the initial quantization sequence and the quantization model corresponding to the mutated quantization sequence; The candidate quantization sequence is used as a new initial quantization sequence. The step of performing mutation operation on the initial quantization sequence to obtain the mutated quantization sequence is returned until the iteration stop condition is met. The target quantization sequence is determined from the candidate quantization sequences, and the search results are obtained; Wherein, the model to be quantized is a neural network model for face recognition, and the quantization calibration data is an image containing a face; or, the model to be quantized is a neural network model for lane lines, and the quantization calibration data is an image containing lane lines.

13. A computer storage medium having a computer program stored thereon, which, when executed by a processor, implements the method of claim 12.

Images