Dictionary index fixed-point operation-based converter quantization acceleration method in text field

Abstract

The invention discloses a dictionary index fixed-point operation-based converter quantization acceleration method and device in the text field, a storage medium, equipment and a computer program product, and the method comprises the steps: determining a reference dictionary used for representing the clustering relation of numerical values according to aggregation hierarchical clustering of a plurality of numerical values in random Gaussian distribution; determining a first localized dictionary of the first tensors according to the linear transformation of each first tensor in the converter on the reference dictionary and the determined first dictionary value of each first tensor; a first dictionary index for the first localized dictionary is determined by exponential function fitting of the first dictionary values. According to the method, extra computing resource and storage space requirements are reduced, training cost and deployment complexity are reduced, the weight and the activation value are quantized at the same time, and the problem that quantization compression application range and real-time performance of the text converter cannot be achieved at the same time is solved.

Description

Technical Field

[0001] The present application belongs to the field of quantization acceleration, and specifically relates to a method, device, storage medium, equipment and computer program product for quantization acceleration of a transformer in the text field based on dictionary index fixed-point operations. Background Art

[0002] With the rapid development of artificial intelligence technology, Transformer models have been widely used in the field of natural language processing. They can efficiently process large-scale text data and are applied to many fields such as machine translation, text generation, and semantic understanding. However, Transformer models are usually large in scale and have high computational complexity, making it difficult to run efficiently on resource-constrained devices. Therefore, quantizing and accelerating Transformer models has become an important research direction.

[0003] At present, the quantization acceleration technologies for Transformer models mainly include integer or fixed-point value-based quantization methods and dictionary-based quantization methods. Integer or fixed-point value-based quantization methods improve computational efficiency by quantizing floating-point values ​​to lower-precision integers or fixed-point values. Dictionary-based quantization methods map weights or activation values ​​to a predefined dictionary whose dictionary items are fixed-precision values.

[0004] However, methods that rely on fine-tuning require additional computing resources and storage space, increasing training costs and deployment complexity, while the quantization of activation values ​​cannot meet real-time requirements due to the difficulty in handling the dynamic range. Summary of the invention

[0005] The present application aims to provide a method, apparatus, storage medium, device and computer program product for accelerating transformer quantization in the text field based on dictionary index fixed-point operations, which at least solves the problem that the scope of application and real-time performance of text transformer quantization compression cannot be achieved at the same time.

[0006] In a first aspect, the embodiment of the present application discloses a transformer quantization acceleration method in the text field based on dictionary index fixed-point operation, including: Determine a reference dictionary for characterizing clustering relationships of a plurality of values ​​of random Gaussian distribution according to aggregate hierarchical clustering; Determine a first localized dictionary of the first tensor according to the first dictionary value determined by linear transformation of each first tensor in the transformer to the reference dictionary; the first tensor is a tensor in the transformer that conforms to a standard Gaussian distribution; the first dictionary value is floating point data; The first dictionary index of the first localized dictionary is determined by fitting an exponential function of the first dictionary value; the first index value in the first dictionary index is integer data; the first index value is used to enable the converter to obtain the execution result of the first target operation command based on the call of the target first tensor corresponding to the target first index value in the first index value when obtaining the first target operation command; the first target operation command is an operation command in which the target first tensor participates in the operation.

[0007] In a second aspect, the embodiment of the present application further discloses a transformer quantization acceleration device in the text field based on dictionary index fixed-point operation, comprising: A benchmark module, for determining a benchmark dictionary for characterizing the clustering relationship of the values ​​according to the aggregated hierarchical clustering of a plurality of values ​​of random Gaussian distribution; A localization module, configured to determine a first localized dictionary of each first tensor according to a first dictionary value determined by linear transformation of each first tensor in the transformer to the reference dictionary; the first tensor is a tensor in the transformer that conforms to a standard Gaussian distribution; and the first dictionary value is floating point data; An index module is used to determine the first dictionary index of the first localized dictionary by fitting an exponential function of the first dictionary value; the first index value in the first dictionary index is an integer data; the first index value is used to enable the converter to obtain the execution result of the first target operation command based on the call of the target first tensor corresponding to the target first index value in the first index value when obtaining the first target operation command; the first target operation command is an operation command in which the target first tensor participates in the operation.

[0008] In a third aspect, an embodiment of the present application further discloses a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the steps described in the first aspect are implemented.

[0009] In a fourth aspect, an embodiment of the present application further discloses an electronic device, comprising a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the computer program implements the steps described in the first aspect when executed by the processor.

[0010] In a fifth aspect, an embodiment of the present application further discloses a computer program product, on which a computer program is stored, and when the computer program is executed by a processor, the steps described in the first aspect are implemented.

[0011] In summary, in the embodiment of the present application, by aggregating and hierarchically clustering multiple values ​​of a random Gaussian distribution to generate a reference dictionary, the numerical characteristics of the Gaussian distribution can be effectively characterized, so that subsequent quantization operations have higher accuracy, reducing the need to regenerate the dictionary each time, and improving computational efficiency; then, according to the statistical characteristics of each first tensor in the converter, the reference dictionary is linearly transformed to generate a first localized dictionary adapted to the tensor, and the characteristics of each tensor are combined with the reference dictionary, making the quantization process more flexible and efficient. Through linear transformation, it is ensured that the generated localized dictionary can accurately represent the data characteristics of each tensor, reducing quantization errors; finally, an exponential function is fitted to the dictionary value of the first localized dictionary to generate a first dictionary index. Using the exponential function fitting, complex floating-point operations are simplified to integer index operations, so that when the converter obtains the first target operation command, the corresponding target tensor can be quickly called according to the index value, reducing computational complexity and significantly improving computational efficiency. Therefore, based on the method of the embodiment of the present application, through the generation of a benchmark dictionary, linear transformation of tensors, exponential function fitting and generation of dictionary indexes, efficient quantization acceleration of the transformer model is achieved without fine-tuning, reducing the need for additional computing resources and storage space, reducing training costs and deployment complexity, and quantizing weights and activation values ​​at the same time, effectively solving the problem of the difficult-to-handle dynamic range of activation values ​​in the prior art, meeting real-time requirements, and solving the problem that the scope of application and real-time performance of text transformer quantization compression cannot be achieved at the same time. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Various other advantages and benefits will become apparent to those of ordinary skill in the art by reading the detailed description of the preferred embodiments below. The accompanying drawings are only for the purpose of illustrating the preferred embodiments and are not to be considered as limiting the present application. Also, the same reference symbols are used throughout the accompanying drawings to represent the same components. In the accompanying drawings: Figure 1 It is a flowchart of the steps of a transformer quantization acceleration method in the text field based on dictionary index fixed-point operation provided by an embodiment of the present application; Figure 2 It is a flowchart of another method for accelerating transformer quantization in the text field based on dictionary index fixed-point operation provided by an embodiment of the present application; Figure 3 It is a data transfer process in the embodiment of the present application; Figure 4 It is a structural schematic diagram of a transformer quantization acceleration device in the text field based on dictionary index fixed-point operation provided by an embodiment of the present application; Figure 5 is a block diagram of an electronic device provided in an embodiment of the present application; Figure 6It is a block diagram of another electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0013] The exemplary embodiments of the present application will be described in more detail below with reference to the accompanying drawings. Although the exemplary embodiments of the present application are shown in the accompanying drawings, it should be understood that the present application can be implemented in various forms and should not be limited by the embodiments set forth herein. On the contrary, these embodiments are provided in order to enable a more thorough understanding of the present application and to fully convey the scope of the present application to those skilled in the art.

[0014] Transformer models in the text field, such as the Bidirectional Encoder Representations from Transformers (BERT) model, rely mainly on their internal multi-layer attention mechanisms during their reasoning operations. Each layer of attention mechanism performs weighted calculations on the input data to capture long-range dependencies and generate richer contextual representations. These attention mechanisms generate attention weights by calculating the similarity between the query, key, and value matrices, and perform a weighted summation of the input data based on these weights. In this process, the outputs of each layer of the model and the intermediate activation values ​​are continuously updated to reflect the complex relationships of the input data. Therefore, the transformer model can effectively process and understand the complex semantic structures in natural language, and achieve efficient reasoning for tasks such as text classification, question-answering systems, and machine translation.

[0015] This application uses a rule found in a large number of experiments: the values ​​in the Transformer model driven by the attention mechanism generally present a Gaussian distribution, and the distribution of each layer is the same. Based on this, the statistical features of each first tensor in the transformer can be linearly transformed by a benchmark dictionary generated by aggregating hierarchical clustering of these Gaussian distribution values ​​to generate a localized dictionary to ensure that the dictionary can accurately represent the data characteristics of each tensor. Then, by fitting the localized dictionary with an exponential function and generating a dictionary index that is easy to calculate, efficient fixed-point operations can be achieved. Specifically, by fitting the exponential function, complex floating-point operations are simplified to integer index operations, which greatly reduces the resources and time required for calculation, so that the transformer can quickly call the corresponding target tensor according to the integer index value when performing reasoning, significantly reducing the complexity of the operation and improving the calculation efficiency and real-time performance of the model. Finally, using these dictionary indexes, when the operation command is obtained, the corresponding target tensor can be quickly called to replace the complex attention operation mechanism in the original Transformer model.

[0016] Based on this, Figure 1This embodiment provides a transformer quantization acceleration method in the text field based on dictionary index fixed-point operation, which specifically includes the following steps: Step 101 : Determine a reference dictionary for characterizing the clustering relationship of the values ​​based on the aggregated hierarchical clustering of multiple values ​​of random Gaussian distribution.

[0017] In some embodiments of the present application, a reference dictionary for characterizing the clustering relationship of values ​​is determined based on the aggregated hierarchical clustering of multiple values ​​of a random Gaussian distribution. The reason for performing this step is to generate a reference dictionary that can effectively characterize the numerical characteristics of the Gaussian distribution. The process of performing this step includes generating random Gaussian distribution samples, and applying the Agglomerative Hierarchical Clustering (AC) method to cluster these samples into several clusters, with the center point value of each cluster as a dictionary item. Agglomerative hierarchical clustering is a bottom-up clustering method that first treats each data point as an independent cluster, and then gradually merges the clusters with the smallest distance until a predetermined goal is reached. After performing this step, the generated reference dictionary can be reused in subsequent quantization operations, reducing the need to regenerate the dictionary each time and improving computational efficiency.

[0018] In a specific example, we first generate random Gaussian distribution data containing 50,000 samples, which meet the conditions of mean 0 and standard deviation 1. Then, we apply the aggregated hierarchical clustering method to cluster the generated Gaussian distribution samples, and finally obtain 16 clusters. The center point value of each cluster is used as the fixed point value (dictionary item) to form a benchmark dictionary. According to the above execution process, the generated benchmark dictionary can effectively characterize the numerical characteristics of the Gaussian distribution and provide a reliable basis for subsequent quantization operations.

[0019] Step 102 , determining a first localized dictionary of the first tensor according to the first dictionary value determined for each first tensor according to the linear transformation of each first tensor in the transformer to the reference dictionary.

[0020] Among them, the first tensor is a tensor in the transformer that conforms to the standard Gaussian distribution; the first dictionary value is floating-point data.

[0021] In some embodiments of the present application, the first dictionary value of each first tensor determined according to the linear transformation of the reference dictionary by each first tensor in the converter is used to determine the first localized dictionary of the first tensor. The reason for performing this step is to generate a localized dictionary adapted to the characteristics of each tensor, so as to make the quantization process more flexible and efficient. The process of performing this step includes calculating the mean and standard deviation of each first tensor, and performing a linear transformation on the fixed-point value of the reference dictionary to generate a localized dictionary adapted to the tensor. The fixed-point value refers to the floating-point data in the dictionary, and the linear transformation is used to ensure that the generated localized dictionary can accurately represent the data characteristics of each tensor. After performing this step, the generated localized dictionary can reduce the quantization error and improve the quantization accuracy.

[0022] In a specific example, the mean m and standard deviation s of a weight tensor in the transformer can be calculated first. Then the fixed-point values ​​in the reference dictionary are subjected to a linear transformation operation, that is, the formula Dict'=s*Dict+m is used, where Dict' represents the first localized dictionary and Dict represents the reference dictionary. The reference dictionary is scaled and offset to generate a localized dictionary adapted to the weight tensor. The localized dictionary generated according to the above execution process can accurately represent the data characteristics of the weight tensor, reduce quantization errors, and improve quantization accuracy.

[0023] Step 103 : Determine a first dictionary index for a first localized dictionary by fitting an exponential function to the first dictionary value.

[0024] Among them, the first index value in the first dictionary index is integer data; the first index value is used to enable the converter to obtain the execution result of the first target operation command according to the call of the target first tensor corresponding to the target first index value in the first index value when obtaining the first target operation command; the first target operation command is an operation command in which the target first tensor participates in the operation.

[0025] In some embodiments of the present application, a first dictionary index for a first localized dictionary is determined by fitting an exponential function to a first dictionary value. The reason for performing this step is to simplify complex floating-point operations into integer index operations, thereby improving computational efficiency. The process of performing this step includes fitting an exponential function to the first dictionary value to generate an integer index value. Exponential function fitting is a mathematical method that maps floating-point fixed-point values ​​to integer index values. After performing this step, the generated first dictionary index can quickly call the corresponding target tensor according to the index value when the converter obtains the first target operation command, thereby reducing computational complexity.

[0026] In a specific example, an exponential function fitting method is applied to the floating-point fixed-point values ​​in the localized dictionary of a weight tensor to map them to integer index values. These integer index values ​​include a sign bit and an integer index bit, the sign bit indicates the positive or negative value, and the integer index bit indicates the quantization level. For example, 1011 represents the third quantization level of a negative number. According to the above execution process, the generated first dictionary index can quickly call the corresponding target tensor according to the index value when the converter obtains the first target operation command, thereby reducing the computational complexity and improving the computational efficiency.

[0027] In summary, in the embodiment of the present application, by aggregating and hierarchically clustering multiple values ​​of a random Gaussian distribution to generate a reference dictionary, the numerical characteristics of the Gaussian distribution can be effectively characterized, so that subsequent quantization operations have higher precision, reducing the need to regenerate the dictionary each time, and improving computational efficiency; then, according to the statistical characteristics of each first tensor in the converter, the reference dictionary is linearly transformed to generate a first localized dictionary adapted to the tensor, and the characteristics of each tensor are combined with the reference dictionary, making the quantization process more flexible and efficient. Through linear transformation, it is ensured that the generated localized dictionary can accurately represent the data characteristics of each tensor, reducing quantization errors; finally, an exponential function is fitted to the dictionary value of the first localized dictionary to generate a first dictionary index. Using the exponential function fitting, complex floating-point operations are simplified to integer index operations, so that when the converter obtains the first target operation command, the corresponding target tensor can be quickly called according to the index value, reducing computational complexity and significantly improving computational efficiency. Therefore, based on the method of the embodiment of the present application, through the generation of a benchmark dictionary, linear transformation of tensors, exponential function fitting and generation of dictionary indexes, efficient quantization acceleration of the transformer model is achieved without fine-tuning, reducing the need for additional computing resources and storage space, reducing training costs and deployment complexity, and quantizing weights and activation values ​​at the same time, effectively solving the problem of the difficult-to-handle dynamic range of activation values ​​in the prior art, meeting real-time requirements, and solving the problem that the scope of application and real-time performance of text transformer quantization compression cannot be achieved at the same time.

[0028] Figure 2 This is another transformer quantization acceleration method in the text field based on dictionary index fixed-point operation provided by this embodiment, which specifically includes the following steps: Step 201 : Determine a reference dictionary for characterizing the clustering relationship of the values ​​based on the aggregated hierarchical clustering of multiple values ​​of random Gaussian distribution.

[0029] The method shown in this step has been described in step 101 and will not be repeated here. Optionally, step 201 includes the following sub-steps: Sub-step 2011, generating multiple groups of values ​​according to a preset sample size, so that each group of values ​​obeys a random Gaussian distribution.

[0030] In some embodiments of the present application, multiple groups of values ​​are generated according to a preset sample capacity so that each group of values ​​obeys a random Gaussian distribution. The reason for performing this step is to ensure that the generated samples can accurately reflect the characteristics of the Gaussian distribution and provide a basis for subsequent clustering operations. The process of performing this step includes generating multiple groups of values ​​according to a preset sample capacity, and each group of values ​​is distributed according to a standard Gaussian distribution. The sample capacity refers to the number of generated values ​​to ensure that each group of values ​​is representative. After performing this step, the generated multiple groups of values ​​can effectively characterize the Gaussian distribution and provide a reliable data basis for subsequent quantization processing and dictionary generation.

[0031] In a specific example, five sets of values ​​are first generated according to the preset sample size, each containing 50,000 data points. Each set of values ​​is generated according to a standard Gaussian distribution with a mean of 0 and a standard deviation of 1. These generated sets of values ​​can accurately reflect the characteristics of the Gaussian distribution and ensure that they are representative in subsequent aggregate hierarchical clustering operations. According to the above execution process, multiple sets of values ​​that conform to the random Gaussian distribution are generated, laying the foundation for subsequent dictionary generation.

[0032] Sub-step 2012, based on the aggregate hierarchical clustering operation on the multiple groups of values, generate multiple reference dictionaries corresponding to each group of values, so that each reference dictionary contains multiple value clusters.

[0033] Different reference dictionaries contain the same number of value clusters; each value cluster has a corresponding central value.

[0034] In some embodiments of the present application, based on the aggregated hierarchical clustering operation on multiple groups of values, multiple reference dictionaries corresponding to each group of values ​​are generated, so that each reference dictionary contains multiple clusters of values. The reason for performing this step is to utilize the numerical characteristics of Gaussian distribution to generate multiple reference dictionaries through clustering operations, and these dictionaries can effectively represent the characteristics of the clusters of values. The process of performing this step includes performing an aggregated hierarchical clustering operation on multiple groups of values, dividing each group of values ​​into a number of clusters of values, and each cluster of values ​​has a corresponding central value. The aggregated hierarchical clustering method is a bottom-up clustering method that generates a hierarchical clustering result by gradually merging clusters with the smallest distance. After performing this step, the multiple reference dictionaries generated can accurately reflect the clustering relationship of each group of values, providing a basis for the subsequent generation of a benchmark dictionary.

[0035] In a specific example, the aggregated hierarchical clustering method is first applied to the five groups of Gaussian distribution values ​​generated. Each group of values ​​is gradually merged with the clusters with the smallest distance according to the aggregated hierarchical clustering method, and finally 16 value clusters are generated for each group. Each value cluster has a corresponding central value, which is used as a dictionary item in the reference dictionary. Five reference dictionaries are generated according to the above execution process, each of which contains 16 value clusters, laying the foundation for the generation of the benchmark dictionary.

[0036] Sub-step 2013, obtaining a reference dictionary by averaging the central values ​​of multiple reference dictionaries.

[0037] In some embodiments of the present application, a reference dictionary is obtained based on an averaging operation on the central values ​​of multiple reference dictionaries. The reason for performing this step is to generate a reference dictionary that can represent the characteristics of a Gaussian distribution by performing an averaging operation on multiple reference dictionaries. The process of performing this step includes performing an averaging operation on the central values ​​of multiple reference dictionaries generated previously, and taking the average value of these central values ​​as the dictionary item of the reference dictionary. The central value refers to the center point of each cluster of values ​​in the reference dictionary, representing the statistical characteristics of the values ​​within the cluster. After performing this step, the generated reference dictionary can accurately reflect the characteristics of the Gaussian distribution and provide a reliable data basis for subsequent quantization operations.

[0038] In a specific example, the center values ​​in the five previously generated reference dictionaries are first extracted, each of which contains 16 center values. Then the center values ​​at each position are averaged, that is, the average value of the center values ​​of each corresponding position in the five reference dictionaries is calculated, and finally a reference dictionary containing 16 center values ​​is generated. According to the above execution process, by averaging the center values ​​of multiple reference dictionaries, a reference dictionary that can accurately characterize the characteristics of the Gaussian distribution is successfully generated, providing reliable data support for subsequent quantization processing.

[0039] Step 202, determining the dictionary value of each tensor according to the linear transformation of each tensor in the transformer to the reference dictionary.

[0040] In some embodiments of the present application, the dictionary value of each tensor is determined based on the linear transformation of each tensor in the converter to the reference dictionary. The reason for performing this step is to generate a dictionary value that adapts to the characteristics of each tensor, making the quantization process more accurate and efficient. The process of performing this step includes calculating the statistical characteristics of each tensor, such as the mean and standard deviation, and performing a linear transformation on the fixed-point value of the reference dictionary to generate a dictionary value that adapts to the tensor. The fixed-point value refers to the floating-point data in the dictionary, and the linear transformation ensures that the generated dictionary value can accurately represent the data characteristics of each tensor. After performing this step, the generated dictionary value can reduce the quantization error and improve the quantization accuracy.

[0041] In a specific example, the mean m and standard deviation s of a weight tensor in the transformer are first calculated. Then, the fixed-point values ​​in the reference dictionary are scaled and offset by a linear transformation operation, that is, the formula Dict'=s*Dict+m is used to generate dictionary values ​​adapted to the weight tensor. The dictionary values ​​generated according to the above execution process can accurately represent the data characteristics of the weight tensor, reduce quantization errors, and improve quantization accuracy.

[0042] Step 203 , according to a preset statistical matching condition, compare the statistical features of each tensor with the dictionary value to determine the first tensor and the second tensor in the tensor.

[0043] The second tensor is a tensor in the transformer other than the tensor that conforms to the standard Gaussian distribution.

[0044] In some embodiments of the present application, the statistical features of each tensor are compared with the dictionary values ​​according to the preset statistical matching conditions to determine the first tensor and the second tensor in the tensor. The reason for performing this step is to distinguish between tensors that conform to the standard Gaussian distribution and tensors that do not conform to the standard Gaussian distribution, so as to properly process different types of tensors. The process of performing this step includes calculating the statistical features (such as mean and standard deviation) of each tensor and comparing them with the dictionary values ​​to determine the type of tensor. Statistical features refer to the data characteristics of a tensor, including parameters such as mean and standard deviation. After executing this step, the first tensor and the second tensor can be accurately distinguished, providing a basis for quantization and processing in subsequent steps.

[0045] In a specific example, the mean or standard deviation of each tensor in the converter can be calculated first. Then, the statistical features of each tensor are compared with the dictionary values ​​in the reference dictionary. According to the preset matching conditions, the tensor that conforms to the standard Gaussian distribution is determined as the first tensor, and the tensor that does not conform to the standard Gaussian distribution is determined as the second tensor. According to the above execution process, the first tensor and the second tensor can be accurately distinguished, which provides a basis for subsequent quantization and processing.

[0046] Optionally, step 203 includes the following sub-steps: Sub-step 2031, calculate the mean of each tensor.

[0047] In some embodiments of the present application, the mean of each tensor is calculated. The reason for performing this step is to obtain the statistical characteristics of each tensor as a basis for distinguishing the first tensor from the second tensor in the subsequent steps. The process of performing this step includes performing statistical calculations on all values ​​in each tensor to obtain their mean. The mean refers to the arithmetic mean of the values ​​of all elements in the tensor, representing the central trend of the tensor. After performing this step, the mean of each tensor can be accurately obtained, providing data support for subsequent quantization and classification operations.

[0048] In a specific example, the mean of a weight tensor needs to be calculated. First, all elements in the tensor are statistically summed, and then the sum is divided by the total number of elements to obtain the mean of the tensor. According to the above execution process, the mean of the weight tensor can be accurately calculated, providing reliable statistical data for determining the tensor type in subsequent steps.

[0049] Sub-step 2032, when the ratio between the mean of the tensor and the dictionary value of the tensor is less than a preset scale threshold, the tensor is determined as a first tensor, and when the ratio between the mean of the tensor and the dictionary value of the tensor is greater than or equal to the preset scale threshold, the tensor is determined as a second tensor.

[0050] In some embodiments of the present application, when the ratio between the mean of the tensor and the dictionary value of the tensor is less than a preset scale threshold, the tensor is determined as a first tensor, and when the ratio between the mean of the tensor and the dictionary value of the tensor is greater than or equal to the preset scale threshold, the tensor is determined as a second tensor. The reason for performing this step is to distinguish the first tensor from the second tensor according to the ratio of the statistical characteristics of the tensor to the dictionary value, so as to properly process different types of tensors. The process of performing this step includes calculating the mean of each tensor, and then calculating the ratio of the mean to the dictionary value of the tensor, and comparing it with the preset scale threshold. If the ratio is less than the threshold, the tensor is determined as the first tensor; if the ratio is greater than or equal to the threshold, the tensor is determined as the second tensor. The ratio refers to the proportional relationship between the mean of the tensor and the dictionary value, which is used to determine the type of the tensor. After performing this step, the type of tensor can be accurately distinguished, providing a basis for subsequent quantization and processing.

[0051] In a specific example, first calculate the mean of a tensor as m, and extract the dictionary value of the tensor as d. Next, calculate the ratio of m to d and compare it with the preset scale threshold. For example, the preset scale threshold is 1.5. If the ratio m / d is less than 1.5, the tensor is determined as the first tensor; if the ratio m / d is greater than or equal to 1.5, the tensor is determined as the second tensor. According to the above execution process, the first tensor and the second tensor can be accurately distinguished, which provides a reliable basis for the quantization and processing of subsequent steps.

[0052] Step 204: Generate a localized dictionary for the first tensor according to the first dictionary value corresponding to the first tensor.

[0053] In some embodiments of the present application, a localized dictionary of the first tensor is generated according to the first dictionary value corresponding to the first tensor. The reason for performing this step is to utilize the specific dictionary value of each first tensor to generate a localized dictionary that matches its data characteristics, thereby further improving the quantization accuracy. The process of performing this step includes applying the first dictionary value of each first tensor to the generated reference dictionary, and generating a localized dictionary adapted to the tensor through a linear transformation step. The dictionary value refers to the floating-point data in the reference dictionary, which can better represent the corresponding first tensor after linear transformation. After performing this step, the generated localized dictionary can accurately describe the characteristics of each first tensor, reduce quantization errors, and improve model performance.

[0054] In a specific example, first, based on the statistical characteristics of a weight tensor, the first dictionary value is extracted from the reference dictionary. Then, these dictionary values ​​are linearly transformed and exponentially fitted to convert them into data characteristics that are suitable for the weight tensor. Finally, a localized dictionary for the weight tensor is generated to ensure that the dictionary can accurately represent the numerical characteristics of the tensor in subsequent quantization operations. The localized dictionary generated according to the above execution process can accurately describe the corresponding first tensor, reduce quantization errors, and improve the accuracy and efficiency of model quantization.

[0055] Step 205 : Determine a first dictionary index for a first localized dictionary by fitting an exponential function to the first dictionary value.

[0056] Among them, the first index value in the first dictionary index is integer data; the first index value is used to enable the converter to obtain the execution result of the first target operation command according to the call of the target first tensor corresponding to the target first index value in the first index value when obtaining the first target operation command; the first target operation command is an operation command in which the target first tensor participates in the operation.

[0057] The method shown in this step has been described in step 103 and will not be repeated here. Optionally, step 205 includes the following sub-steps: Sub-step 2051, according to An exponential function of the form of is fit to the first dictionary values ​​to determine a two-element array corresponding to each first dictionary value, and a two-element parameter.

[0058] Among them, the first variable of the binary array is used to represent the size relationship between each first dictionary value and the average value of the first dictionary value through one bit of integer data; the second variable of the binary array is used to represent the exponential value of each first dictionary value to the first parameter a in the binary parameter after decomposition according to the exponential function through multi-bit integer data; b is the second parameter in the binary parameter.

[0059] In some embodiments of the present application, according to The exponential function of the first dictionary value is fitted in the form of to determine the binary array corresponding to each first dictionary value, and the binary parameter. The reason for performing this step is to map each first dictionary value to a binary array and a binary parameter that are easy to calculate through exponential function fitting. The process of performing this step includes performing exponential function fitting on each first dictionary value to determine the fitting result, which contains a binary array and a binary parameter. The first variable of the binary array is used to characterize the relationship between each first dictionary value and the average value of the first dictionary value through one bit of integer data, and the second variable is used to characterize the exponential value of each first dictionary value after decomposition according to the exponential function to the first parameter a in the binary parameter through multi-bit integer data, and b is the second parameter in the binary parameter. After performing this step, the generated binary array and binary parameter can be used for subsequent quantization and calculation, thereby improving the calculation efficiency of the model.

[0060] In a specific example, apply an exponential function fitting method to the first dictionary value of a tensor, according to After fitting, the binary array and binary parameters corresponding to each first dictionary value are determined. The first variable of the binary array represents the magnitude relationship between each first dictionary value and its average value, the second variable represents the exponential value of the exponential function, and a and b in the binary parameters are the parameters in the fitting. According to the above execution process, the binary array and binary parameters for subsequent quantification and calculation are generated, which improves the calculation efficiency of the model.

[0061] Sub-step 2052: merge the binary parameters corresponding to each first dictionary value to obtain a first index value corresponding to each first dictionary value.

[0062] In some embodiments of the present application, the binary parameters corresponding to each first dictionary value are merged and recorded to obtain the first index values ​​corresponding to each first dictionary value. The reason for performing this step is to integrate the binary parameters obtained by exponential function fitting to generate a first index value for fast search and calculation. The process of performing this step includes merging the binary parameters in the binary array corresponding to each first dictionary value, and recording the merged result as the first index value. Binary parameters refer to two parameters obtained by exponential function fitting, which are used to represent numerical relationships and exponential values, respectively. After performing this step, the generated first index value can be used for fast search and calculation of the converter, thereby improving the computational efficiency of the model.

[0063] In a specific example, the first dictionary value of a tensor is integrated with its corresponding binary parameter. The binary parameter includes a first variable for representing a numerical relationship and a second variable for representing an exponential value. These binary parameters are merged into a complete record and used as the first index value of the first dictionary value. According to the above execution process, a complete first index value is generated, which can be used for fast search and calculation of the transformer, thereby improving the calculation efficiency of the model.

[0064] The following is an explanation of the principle of computational simplification brought about by exponential function fitting: For a data set that conforms to the Gaussian distribution, assume that the obtained 4-bit quantization index is used in model inference to simplify the multiplication and addition operations and achieve model quantization acceleration (the calculation of the outlier subset is not considered at this time). Assuming that there is now a weight tensor, it can be expressed as follows according to the above steps: , similarly, the activation value tensor is expressed as . Where n is the dimension of A and W, A i , and W i are the i-th components of A and W respectively, a is the first parameter used for fitting, Indicates the sign bit, represents its corresponding standard deviation, is the bias value related to the mean, standard deviation, and the second parameter b used for fitting. During the inference process of the model, each output activation is the sum of the activation value and the weight product of the previous step, which is also the step with the highest computational complexity in Transformer. Given the form of the input value, the output activation is expressed as:

[0065] It can be seen that the output activation is composed of the sum of four items. These four items are analyzed separately: The focus of the first item is the sum of the integers on the exponents. When the obtained 4-bit quantized value is used, the range of this sum only includes 15 unique values. Therefore, for the addition operation of this item, we can directly count the number of times the sum of the quantized values ​​on each exponent appears, and then accumulate the number of occurrences and the corresponding values ​​through multiplication and addition to get the main value of this part. And the multiplication and addition operation here is the addition of a series of integer multiples of known numbers. We can get the values ​​of these numbers before model inference without additional calculations; The second term can be decomposed into a constant and a part similar to the first term, the only difference is that here it is not the sum of the quantized values ​​that needs to be counted, but the individual quantized values, which contain 8 unique values. Similar to part b, by multiplying and accumulating the number of occurrences and the corresponding values, and adding the constant bias, the main value of this part can be obtained; The third item is the processing of the activation value vector. The processing steps of the second item are completely similar and will not be repeated here; The fourth item can be decomposed into some fixed constants about the fitting parameters a and b, and the sum of the multiplication results of the symbolic values. The former are some constants that can be calculated in advance, and the latter are completely operations between integers, so its computational complexity is very low.

[0066] It can be seen that by adding up the above steps, the quantization of the output activation value is completed. The successful use of the index value counting greatly reduces a large number of floating-point multiplication and addition operations, thereby achieving quantization acceleration of the overall model.

[0067] Step 206 : Determine a second localized dictionary for characterizing the clustering relationship of the second tensor according to the aggregated hierarchical clustering of the second tensor, and generate a second dictionary index for the second localized dictionary.

[0068] Among them, the second index value in the second dictionary index is used to enable the converter to obtain the execution result of the second target operation command by calling the target second tensor corresponding to the target second index value in the second index value when obtaining the second target operation command; the second target operation command is an operation command in which the target second tensor participates in the operation.

[0069] In some embodiments of the present application, based on the aggregated hierarchical clustering of the second tensor, a second localized dictionary is determined to characterize the clustering relationship of the second tensor, and a second dictionary index for the second localized dictionary is generated. The reason for performing this step is to cluster the second tensors that do not conform to the standard Gaussian distribution so as to generate a localized dictionary and a dictionary index that can effectively characterize the characteristics of these tensors. The process of performing this step includes performing aggregated hierarchical clustering on the second tensor, generating a second localized dictionary representing its clustering relationship, and generating a second dictionary index by exponential function fitting or other methods. The second dictionary index is used to quickly call the corresponding target second tensor according to the target second index value and execute the operation command when the converter obtains the second target operation command. After executing this step, the generated second localized dictionary and second dictionary index can effectively characterize the second tensor and improve the computational efficiency and performance of the model.

[0070] In a specific example, some tensors (second tensors) in the converter that do not conform to the standard Gaussian distribution are first clustered hierarchically to generate a second localized dictionary representing their clustering relationship. Then, a second dictionary index is generated for the second localized dictionary to characterize the target second tensor. According to the above execution process, when the converter obtains the second target operation command, it can quickly call the corresponding target second tensor according to the target second index value and execute the operation command, further improving the computing efficiency and performance.

[0071] Figure 3 This is the data flow process in the embodiment of the present application: S1. Aggregation hierarchical classification: Aggregation hierarchical classification is performed on the basis of Gaussian distribution to generate a benchmark dictionary. This step divides the Gaussian distribution samples into multiple clusters through clustering operations. The center point value of each cluster forms the dictionary item of the benchmark dictionary, laying the foundation for subsequent quantization operations; S2. Calculate the variance and mean to classify the tensors: Classify the tensors according to the variance and mean to generate Gaussian value subsets and outlier subsets. This step determines whether the tensor conforms to the standard Gaussian distribution by calculating the variance and mean of the tensor, and regards the tensors that conform to the standard Gaussian distribution as the Gaussian value subset, and the rest as the outlier subset; S3.1, Linear transformation: Perform linear transformation on the Gaussian value subset to generate a quantization dictionary. In this step, according to the characteristics of the Gaussian value subset, the reference dictionary is linearly transformed to adapt it to the data characteristics of the Gaussian value subset, generate a quantization dictionary, and improve the quantization accuracy; S3.2, separate clustering: cluster the outlier subset separately and generate an outlier dictionary. This step clusters the outlier subset and generates a dictionary that adapts to the data characteristics of these outliers to ensure the accuracy of the quantization process.

[0072] Finally, in the actual model calculation process (S0), the weight / embedding tensor and activation value tensor of the Transformer model are processed in the quantization process to generate a 4-bit quantization index for index value calculation, which converts the operation form during model inference into a form suitable for quantization, thereby improving the computational efficiency and performance of the model.

[0073] In summary, in the embodiment of the present application, by aggregating and hierarchically clustering multiple values ​​of a random Gaussian distribution to generate a reference dictionary, the numerical characteristics of the Gaussian distribution can be effectively characterized, so that subsequent quantization operations have higher precision, reducing the need to regenerate the dictionary each time, and improving computational efficiency; then, according to the statistical characteristics of each first tensor in the converter, the reference dictionary is linearly transformed to generate a first localized dictionary adapted to the tensor, and the characteristics of each tensor are combined with the reference dictionary, making the quantization process more flexible and efficient. Through linear transformation, it is ensured that the generated localized dictionary can accurately represent the data characteristics of each tensor, reducing quantization errors; finally, an exponential function is fitted to the dictionary value of the first localized dictionary to generate a first dictionary index. Using the exponential function fitting, complex floating-point operations are simplified to integer index operations, so that when the converter obtains the first target operation command, the corresponding target tensor can be quickly called according to the index value, reducing computational complexity and significantly improving computational efficiency. Therefore, based on the method of the embodiment of the present application, through the generation of a benchmark dictionary, linear transformation of tensors, exponential function fitting and generation of dictionary indexes, efficient quantization acceleration of the transformer model is achieved without fine-tuning, reducing the need for additional computing resources and storage space, reducing training costs and deployment complexity, and quantizing weights and activation values ​​at the same time, effectively solving the problem of the difficult-to-handle dynamic range of activation values ​​in the prior art, meeting real-time requirements, and solving the problem that the scope of application and real-time performance of text transformer quantization compression cannot be achieved at the same time.

[0074] like Figure 4 As shown, the embodiment of the present application also discloses a text field transformer quantization acceleration device 30 based on dictionary index fixed-point operation, including: A reference module 301 is used to determine a reference dictionary for characterizing the clustering relationship of the values ​​according to the aggregated hierarchical clustering of multiple values ​​of random Gaussian distribution; The localization module 302 is used to determine a first localized dictionary of the first tensor according to the first dictionary value of each first tensor determined by linear transformation of each first tensor in the transformer to the reference dictionary; the first tensor is a tensor in the transformer that conforms to the standard Gaussian distribution; the first dictionary value is floating point data; The index module 303 is used to determine the first dictionary index of the first localized dictionary by fitting an exponential function of the first dictionary value; the first index value in the first dictionary index is an integer data; the first index value is used to enable the converter to obtain the execution result of the first target operation command according to the call of the target first tensor corresponding to the target first index value in the first index value when obtaining the first target operation command; the first target operation command is an operation command in which the target first tensor participates in the operation.

[0075] Optionally, the benchmark module 301 includes: The value generation submodule is used to generate multiple groups of values ​​according to a preset sample capacity, so that each group of values ​​obeys a random Gaussian distribution; A clustering submodule, for generating a plurality of reference dictionaries corresponding to each group of values ​​according to an aggregated hierarchical clustering operation on a plurality of groups of values, so that each reference dictionary contains a plurality of value clusters; different reference dictionaries contain the same number of value clusters; and each value cluster has a corresponding central value; The averaging submodule is used to obtain a reference dictionary by averaging the central values ​​of multiple reference dictionaries.

[0076] Optionally, the localization module 302 includes: A linear transformation submodule, for determining a dictionary value for each tensor according to a linear transformation of the reference dictionary by each tensor in the transformer; A first screening submodule, for comparing the statistical features of each tensor with the dictionary value according to a preset statistical matching condition to determine a first tensor in the tensors; The first dictionary submodule is used to generate a localized dictionary of the first tensor according to a first dictionary value corresponding to the first tensor.

[0077] Optionally, the first screening submodule includes: The mean unit is used to calculate the mean of each tensor; The screening unit is used to determine the tensor as the first tensor when the ratio between the mean value of the tensor and the dictionary value of the tensor is less than a preset scale threshold.

[0078] Optionally, the text field transformer quantization acceleration device 30 based on dictionary index fixed-point operation also includes: A second screening module is used to compare the statistical features of each tensor with the dictionary value according to a preset statistical matching condition to determine a second tensor in the tensor; the second tensor is a tensor in the transformer other than the tensor that conforms to the standard Gaussian distribution; The second dictionary module is used to determine a second localized dictionary for characterizing the clustering relationship of the second tensor based on the aggregated hierarchical clustering of the second tensor, and generate a second dictionary index for the second localized dictionary; the second index value in the second dictionary index is used to enable the converter to obtain the execution result of the second target operation command based on the call of the target second tensor corresponding to the target second index value in the second index value when obtaining the second target operation command; the second target operation command is an operation command in which the target second tensor participates in the operation.

[0079] Optionally, the index module 303 includes: The fitting submodule is used to The exponential function of the first dictionary value is fitted in the form of to determine the binary array corresponding to each first dictionary value and the binary parameter; the first variable of the binary array is used to represent the magnitude relationship between each first dictionary value and the average value of the first dictionary value through one bit of integer data; the second variable of the binary array is used to represent the exponential value of each first dictionary value after decomposition according to the exponential function to the first parameter a in the binary parameter through multi-bit integer data; b is the second parameter in the binary parameter; The index submodule is used to merge and record the binary parameters corresponding to each first dictionary value to obtain a first index value corresponding to each first dictionary value.

[0080] In summary, in the embodiment of the present application, by aggregating and hierarchically clustering multiple values ​​of a random Gaussian distribution to generate a reference dictionary, the numerical characteristics of the Gaussian distribution can be effectively characterized, so that subsequent quantization operations have higher precision, reducing the need to regenerate the dictionary each time, and improving computational efficiency; then, according to the statistical characteristics of each first tensor in the converter, the reference dictionary is linearly transformed to generate a first localized dictionary adapted to the tensor, and the characteristics of each tensor are combined with the reference dictionary, making the quantization process more flexible and efficient. Through linear transformation, it is ensured that the generated localized dictionary can accurately represent the data characteristics of each tensor, reducing quantization errors; finally, an exponential function is fitted to the dictionary value of the first localized dictionary to generate a first dictionary index. Using the exponential function fitting, complex floating-point operations are simplified to integer index operations, so that when the converter obtains the first target operation command, the corresponding target tensor can be quickly called according to the index value, reducing computational complexity and significantly improving computational efficiency. Therefore, based on the method of the embodiment of the present application, through the generation of a benchmark dictionary, linear transformation of tensors, exponential function fitting and generation of dictionary indexes, efficient quantization acceleration of the transformer model is achieved without fine-tuning, reducing the need for additional computing resources and storage space, reducing training costs and deployment complexity, and quantizing weights and activation values ​​at the same time, effectively solving the problem of the difficult-to-handle dynamic range of activation values ​​in the prior art, meeting real-time requirements, and solving the problem that the scope of application and real-time performance of text transformer quantization compression cannot be achieved at the same time.

[0081] The embodiment of the present application also provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, each process of the above-mentioned transformer quantization acceleration method embodiment in the text field based on dictionary index fixed-point operation is implemented, and the same technical effect can be achieved. To avoid repetition, it is not repeated here. The computer-readable storage medium is, for example, a read-only memory (ROM), a random access memory (RAM), a disk or an optical disk, etc.

[0082] Figure 5 700 is a block diagram of an electronic device 700 provided in an embodiment of the present application. For example, the electronic device 700 may be a mobile phone, a computer, a digital broadcast terminal, a messaging device, a game console, a tablet device, a medical device, a fitness device, a personal digital assistant, etc.

[0083] Reference Figure 5, the electronic device 700 may include one or more of the following components: a processing component 702 , a memory 704 , a power component 706 , a multimedia component 708 , an audio component 710 , an input / output (I / O) interface 712 , a sensor component 714 , and a communication component 716 .

[0084] The processing component 702 generally controls the overall operation of the electronic device 700, such as operations associated with display, phone calls, data communications, camera operations, and recording operations. The processing component 702 may include one or more processors 720 to execute instructions to complete all or part of the steps of the above-mentioned transformer quantization acceleration method for the text field based on dictionary index fixed-point operations. In addition, the processing component 702 may include one or more modules to facilitate the interaction between the processing component 702 and other components. For example, the processing component 702 may include a multimedia module to facilitate the interaction between the multimedia component 708 and the processing component 702.

[0085] The memory 704 is used to store various types of data to support the operation of the electronic device 700. Examples of such data include instructions for any application or method operating on the electronic device 700, contact data, phone book data, messages, pictures, multimedia, etc. The memory 704 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0086] The power supply component 706 provides power to the various components of the electronic device 700. The power supply component 706 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to the electronic device 700.

[0087] The multimedia component 708 includes a screen that provides an output interface between the electronic device 700 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touch, slide, and gestures on the touch panel. The touch sensor may not only sense the demarcation of the touch or slide action, but also detect the duration and pressure associated with the touch or slide operation. In some embodiments, the multimedia component 708 includes a front camera and / or a rear camera. When the electronic device 700 is in an operating mode, such as a shooting mode or a multimedia mode, the front camera and / or the rear camera may receive external multimedia data. Each front camera and rear camera may be a fixed optical lens system or have a focal length and optical zoom capability.

[0088] The audio component 710 is used to output and / or input audio signals. For example, the audio component 710 includes a microphone (MIC), and when the electronic device 700 is in an operation mode, such as a call mode, a recording mode, and a voice recognition mode, the microphone is used to receive an external audio signal. The received audio signal can be further stored in the memory 704 or sent via the communication component 716. In some embodiments, the audio component 710 also includes a speaker for outputting audio signals.

[0089] I / O interface 712 provides an interface between processing component 702 and peripheral interface modules, such as keyboards, click wheels, buttons, etc. These buttons may include but are not limited to: home button, volume button, start button, and lock button.

[0090] The sensor assembly 714 includes one or more sensors for providing various aspects of status assessment for the electronic device 700. For example, the sensor assembly 714 can detect the open / closed state of the electronic device 700, the relative positioning of components, such as the display and keypad of the electronic device 700, and the sensor assembly 714 can also detect the position change of the electronic device 700 or a component of the electronic device 700, the presence or absence of user contact with the electronic device 700, the orientation or acceleration / deceleration of the electronic device 700, and the temperature change of the electronic device 700. The sensor assembly 714 may include a proximity sensor configured to detect the presence of nearby objects without any physical contact. The sensor assembly 714 may also include an optical sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, the sensor assembly 714 may also include an acceleration sensor, a gyroscope sensor, a magnetic sensor, a pressure sensor, or a temperature sensor.

[0091] The communication component 716 is used to facilitate wired or wireless communication between the electronic device 700 and other devices. The electronic device 700 can access a wireless network based on a communication standard, such as WiFi, a carrier network (such as 2G, 3G, 4G or 7G), or a combination thereof. In an exemplary embodiment, the communication component 716 receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 716 also includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology and other technologies.

[0092] In an exemplary embodiment, the electronic device 700 can be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors or other electronic components to implement the transformer quantization acceleration method in the text field based on dictionary index fixed-point operations provided in an embodiment of the present application.

[0093] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 704 including instructions, and the instructions can be executed by a processor 720 of an electronic device 700 to complete the above-mentioned transformer quantization acceleration method in the text field based on dictionary index fixed-point operation. For example, the non-transitory storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.

[0094] Figure 6 8 is a block diagram of an electronic device 800 according to an exemplary embodiment. For example, the electronic device 800 may be provided as a server. Figure 6 The electronic device 800 includes a processing component 822, which further includes one or more processors, and a memory resource represented by a memory 832 for storing instructions that can be executed by the processing component 822, such as an application. The application stored in the memory 832 may include one or more modules, each corresponding to a set of instructions. In addition, the processing component 822 is configured to execute instructions to perform the transformer quantization acceleration method in the text field based on dictionary index fixed-point operation provided in an embodiment of the present application.

[0095] The electronic device 800 may also include a power supply component 826 configured to perform power management of the electronic device 800, a wired or wireless network interface 850 configured to connect the electronic device 800 to a network, and an input / output (I / O) interface 858. The electronic device 800 may operate based on an operating system stored in the memory 832, such as Windows Server™, Mac OS X™, Unix™, Linux™, FreeBSD™ or the like.

[0096] An embodiment of the present application also provides a computer program product, including a computer program, which implements a transformer quantization acceleration method in the text field based on dictionary index fixed-point operations when the computer program is executed by a processor.

[0097] Those skilled in the art will readily appreciate other embodiments of the present application after considering the specification and practicing the application disclosed herein. The present application is intended to cover any modification, use or adaptation of the present application, which follows the general principles of the present application and includes common knowledge or customary techniques in the art that are not disclosed in the present disclosure. The specification and examples are intended to be exemplary only, and the true scope and spirit of the present application are indicated by the following claims.

[0098] It should be understood that the present application is not limited to the precise structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present application is limited only by the appended claims.

[0099] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0100] It is easy for a person skilled in the art to think that any combination of the above embodiments is feasible, so any combination of the above embodiments is an implementation scheme of the present application. However, due to space limitations, this specification will not describe them in detail here.

[0101] The transformer quantization acceleration method for the text field based on dictionary index fixed-point operations provided herein is not inherently related to any specific computer, virtual system or other device. Various general systems can also be used together with the teachings based on this. According to the above description, it is obvious to construct the structure required for the system with the solution of the present application. In addition, the present application is not directed to any specific programming language. It should be understood that the content of the present application described herein can be implemented using various programming languages, and the above description of specific languages ​​is to disclose the best implementation mode of the present application.

[0102] In the description provided herein, a large number of specific details are described. However, it is understood that the embodiments of the present application can be practiced without these specific details. In some instances, well-known methods, structures and techniques are not shown in detail so as not to obscure the understanding of this description.

[0103] Similarly, it should be understood that in order to streamline the present application and aid in understanding one or more of the various application aspects, in the above description of the exemplary embodiments of the present application, the various features of the present application are sometimes grouped together into a single embodiment, figure, or description thereof. However, the disclosed method should not be interpreted as reflecting the following intention: the claimed application requires more features than the features explicitly recited in each claim. More specifically, as reflected in the claims, the application aspects lie in less than all the features of the single embodiment disclosed above. Therefore, the claims that follow the specific embodiment are hereby expressly incorporated into the specific embodiment, with each claim itself serving as a separate embodiment of the present application.

[0104] Those skilled in the art will appreciate that the modules in the devices in the embodiments may be adaptively changed and arranged in one or more devices different from the embodiments. The modules or units or components in the embodiments may be combined into one module or unit or component, and further may be divided into a plurality of submodules or subunits or subcomponents. All features disclosed in this specification (including the accompanying claims, abstracts and drawings) and all processes or units of any method or device so disclosed may be combined in any combination, except that at least some of such features and / or processes or units are mutually exclusive. Unless otherwise expressly stated, each feature disclosed in this specification (including the accompanying claims, abstracts and drawings) may be replaced by an alternative feature that provides the same, equivalent or similar purpose.

[0105] In addition, those skilled in the art will appreciate that, although some embodiments described herein include certain features included in other embodiments but not other features, the combination of features of different embodiments is meant to be within the scope of the present application and form different embodiments. For example, in the claims, any one of the claimed embodiments can be used in any combination.

[0106] The various component embodiments of the present application can be implemented in hardware, or in software modules running on one or more processors, or in a combination thereof. It should be understood by those skilled in the art that a microprocessor or a digital signal processor (DSP) can be used in practice to implement some or all functions of some or all components in the transformer quantization acceleration method for the text field based on dictionary index fixed-point operations according to the embodiment of the present application. The present application can also be implemented as a device or apparatus program (e.g., a computer program and a computer program product) for executing part or all of the methods described herein. Such a program implementing the present application can be stored on a computer-readable medium, or can be in the form of one or more signals. Such a signal can be downloaded from an Internet website, or provided on a carrier signal, or provided in any other form.

[0107] In another embodiment provided by the present invention, a computer program product containing instructions is also provided. When the computer is run on a computer, the computer executes the text field transformer quantization acceleration method based on dictionary index fixed-point operation of the embodiment of the present application.

[0108] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented by software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present invention is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from a website site, computer, server or data center to another website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrated. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state hard disk Solid State Disk (SSD)), etc.

[0109] It should be noted that the above embodiments illustrate the present application rather than limit the present application, and that those skilled in the art may design alternative embodiments without departing from the scope of the appended claims. In the claims, any reference symbol between brackets should not be constructed as a limitation to the claims. The word "comprising" does not exclude the presence of elements or steps not listed in the claims. The word "one" or "an" preceding an element does not exclude the presence of multiple such elements. The present application may be implemented by means of hardware including several different elements and by means of a suitably programmed computer. In a unit claim that lists several devices, several of these devices may be embodied by the same hardware item. The use of the words first, second, and third, etc. does not indicate any order. These words may be interpreted as names.

[0110] It should be noted that, for the sake of simplicity, the method embodiments of the present application are described as a series of action combinations, but those skilled in the art should be aware that the embodiments of the present application are not limited by the described action sequence, because according to the embodiments of the present application, certain steps can be performed in other sequences or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions involved are not necessarily required by the embodiments of the present application.

[0111] Each embodiment in this specification is described in a related manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the embodiments of the system or device, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiments.

[0112] The above description is only a preferred embodiment of the present invention and is not intended to limit the protection scope of the present invention. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention are included in the protection scope of the present invention.

Claims

1. A transformer quantization acceleration method in the text field based on dictionary index fixed-point operation, characterized in that: include: Determine a reference dictionary for characterizing clustering relationships of a plurality of values ​​of random Gaussian distribution according to aggregate hierarchical clustering; Determine a first localized dictionary of each first tensor according to the first dictionary value determined by performing a linear transformation of each first tensor in the transformer on the reference dictionary; The first tensor is a tensor in the converter that conforms to a standard Gaussian distribution; the first dictionary value is floating point data; Determine a first dictionary index for the first localized dictionary by fitting an exponential function to the first dictionary value; a first index value in the first dictionary index is an integer data; The first index value is used to enable the converter to obtain the execution result of the first target operation command by calling the target first tensor corresponding to the target first index value in the first index value when obtaining the first target operation command; the first target operation command is an operation command in which the target first tensor participates in the operation.

2. The transformer quantization acceleration method in the text field based on dictionary index fixed-point operation as claimed in claim 1, characterized in that: The step of determining a reference dictionary for characterizing the clustering relationship of the values ​​according to the aggregated hierarchical clustering of the multiple values ​​of the random Gaussian distribution includes: Generate multiple groups of values ​​according to a preset sample capacity, so that each group of values ​​obeys a random Gaussian distribution; According to the aggregated hierarchical clustering operation on the multiple groups of the values, a plurality of reference dictionaries corresponding to each group of the values ​​are generated, so that each of the reference dictionaries contains a plurality of value clusters; different reference dictionaries contain the same number of value clusters; each of the value clusters has a corresponding central value; The reference dictionary is obtained by averaging central values ​​of a plurality of the reference dictionaries.

3. The transformer quantization acceleration method in the text field based on dictionary index fixed-point operation as claimed in claim 1, characterized in that: The method of determining a first localized dictionary of each first tensor according to the linear transformation of the reference dictionary by each first tensor in the transformer and the first dictionary value of each first tensor determined comprises: Determining a dictionary value of each tensor according to a linear transformation of the reference dictionary by each tensor in the transformer; According to a preset statistical matching condition, compare the statistical features of each of the tensors with the dictionary value to determine a first tensor in the tensors; Generate a localized dictionary for the first tensor according to a first dictionary value corresponding to the first tensor.

4. The transformer quantization acceleration method in the text field based on dictionary index fixed-point operation as claimed in claim 3, characterized in that: The step of comparing the statistical features of each of the tensors with the dictionary value according to a preset statistical matching condition to determine the first tensor in the tensors includes: computing the mean of each of said tensors; When a ratio between the mean value of the tensor and the dictionary value of the tensor is less than a preset scale threshold, the tensor is determined as a first tensor.

5. The transformer quantization acceleration method in the text field based on dictionary index fixed-point operation as claimed in claim 3, characterized in that: The method further comprises: According to a preset statistical matching condition, the statistical feature of each tensor is compared with the dictionary value to determine a second tensor in the tensor; the second tensor is a tensor in the converter other than the tensor that conforms to the standard Gaussian distribution; According to the aggregated hierarchical clustering of the second tensor, a second localized dictionary is determined to characterize the clustering relationship of the second tensor, and a second dictionary index for the second localized dictionary is generated; the second index value in the second dictionary index is used to enable the converter to obtain the execution result of the second target operation command by calling the target second tensor corresponding to the target second index value in the second index value when obtaining the second target operation command; the second target operation command is an operation command in which the target second tensor participates in the operation.

6. The transformer quantization acceleration method in the text field based on dictionary index fixed-point operation as claimed in claim 1, characterized in that: The determining the first dictionary index for the first localized dictionary by fitting an exponential function to the first dictionary value includes: according to The exponential function of the first dictionary value is fitted in the form of to determine a binary array corresponding to each of the first dictionary values ​​and a binary parameter; the first variable of the binary array is used to represent the magnitude relationship between each of the first dictionary values ​​and the average value of the first dictionary values ​​through one bit of integer data; the second variable of the binary array is used to represent the exponential value of each of the first dictionary values ​​after decomposition according to the exponential function to the first parameter a in the binary parameter through multi-bit integer data; b is the second parameter in the binary parameter; The binary parameters corresponding to each of the first dictionary values ​​are merged and recorded to obtain first index values ​​corresponding to each of the first dictionary values.

7. A transformer quantization acceleration device in the text field based on dictionary index fixed-point operation, characterized in that: include: A benchmark module, for determining a benchmark dictionary for characterizing the clustering relationship of the values ​​according to the aggregated hierarchical clustering of a plurality of values ​​of random Gaussian distribution; A localization module, configured to determine a first localized dictionary of each first tensor according to a first dictionary value determined by performing a linear transformation of each first tensor in a transformer on the reference dictionary; The first tensor is a tensor in the converter that conforms to a standard Gaussian distribution; the first dictionary value is floating point data; An indexing module, configured to determine a first dictionary index for the first localized dictionary by fitting an exponential function to the first dictionary value; a first index value in the first dictionary index is an integer data; The first index value is used to enable the converter to obtain the execution result of the first target operation command by calling the target first tensor corresponding to the target first index value in the first index value when obtaining the first target operation command; the first target operation command is an operation command in which the target first tensor participates in the operation.

8. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the transformer quantization acceleration method in the text field based on dictionary index fixed-point operation as described in any one of claims 1 to 6 are implemented.

9. An electronic device, characterized in that: The invention comprises a processor, a memory and a computer program stored in the memory and executable on the processor, wherein the computer program, when executed by the processor, implements the steps of a transformer quantization acceleration method in the text field based on dictionary index fixed-point operations as described in any one of claims 1 to 6.

10. A computer program product, characterized in that The computer program product stores a computer program, and when the computer program is executed by a processor, the steps of the transformer quantization acceleration method in the text field based on dictionary index fixed-point operation as described in any one of claims 1 to 6 are implemented.

Images