Model compression method and apparatus, training method and apparatus, and text data processing method and apparatus

Abstract

The present disclosure relates to the technical field of artificial intelligence, and in particular relates to the technical fields of deep learning, natural language processing, etc. Provided are a model compression method and apparatus, a training method and apparatus, and a text data processing method and apparatus. A specific implementation of the model compression method involves: according to the number of concurrently deployed calculation units, dividing initial model parameters of a model to be compressed, so as to obtain initial local model parameters corresponding to the plurality of calculation units, wherein the calculation units are used for processing the same task associated with the model to be compressed; according to an initial input activation value, rematching the correspondence between initial weight parameters and the calculation units, so as to obtain target local model parameters corresponding to the plurality of calculation units, wherein a target input activation value among the target local model parameters gradually increases in a data processing direction; and performing quantization on the target local model parameters, so as to obtain a compressed model.

Description

Model compression method, training method, text data processing method and device

[0001] This application claims priority to Chinese patent application No. 202311212565.1 filed on September 19, 2023, the contents of which are incorporated herein by reference. Technical Field

[0002] The present disclosure relates to the field of artificial intelligence technology, particularly to deep learning, natural language processing, and other technical fields, and more particularly to a model compression method, training method, and text data processing method and device. Background Art

[0003] With the increasing application of artificial intelligence in natural language processing, the number of parameters in large language generation models has gradually increased. This is especially true for the Transformer weight parameters in these models, which have a high degree of dispersion in parameter distribution. Consequently, deploying large language generation models requires significant memory and computing resources.

[0004] Summary of the Invention

[0005] The present disclosure provides a model compression method, a compression model training method, and a text data processing method and device.

[0006] According to one aspect of the present disclosure, a model compression method is provided, comprising: dividing the initial model parameters of a model to be compressed according to the number of computing units deployed in parallel to obtain initial local model parameters corresponding to the multiple computing units, wherein the computing units are used to process the same task associated with the model to be compressed, and the initial local model parameters include initial weight parameters on the local processing channels corresponding to the computing units and initial input activation values ​​determined based on the initial weight parameters; rematching the correspondence between the initial weight parameters and the computing units according to the initial input activation values ​​to obtain target local model parameters corresponding to the multiple computing units, wherein the target input activation values ​​in the target local model parameters gradually increase along the direction of data processing; and quantizing the target local model parameters to obtain a compressed model.

[0007] According to another aspect of the present disclosure, a method for training a compression model is provided, comprising: inputting sample text data into the compression model to obtain a first matching probability for a plurality of predetermined categories; the sample text data is annotated with sample category information; determining a loss value of the compression model based on the sample category information and the first matching probability; and adjusting model parameters of the compression model based on the loss value to obtain a trained compression model; wherein the compression model is a compressed deep learning model obtained using the above-mentioned model compression method.

[0008] According to another aspect of the present disclosure, a text data processing method is provided, including: inputting the text data to be processed into a compression model to obtain a matching probability representing the category to which the text data belongs; the matching probability includes a probability value of the text data belonging to each predetermined category in a plurality of predetermined categories; and determining the target category to which the text data belongs based on the matching probability; wherein the compression model is a compressed deep learning model obtained using the above-mentioned model compression method.

[0009] According to another aspect of the present disclosure, a model compression device is provided, comprising: a parameter division module, a parameter rearrangement module, and a quantization module. The parameter division module is used to divide the initial model parameters of the model to be compressed according to the number of computing units deployed in parallel, and obtain initial local model parameters corresponding to multiple computing units, wherein the computing units are used to process the same task associated with the model to be compressed, and the initial local model parameters include initial weight parameters on the local processing channels corresponding to the computing units and initial input activation values ​​determined based on the initial weight parameters; the parameter rearrangement module is used to re-match the correspondence between the initial weight parameters and the computing units according to the initial input activation values, and obtain target local model parameters corresponding to the multiple computing units, wherein the target input activation values ​​in the target local model parameters gradually increase along the direction of data processing; and the quantization module is used to quantize the target local model parameters to obtain a compressed model.

[0010] According to another aspect of the present disclosure, a compression model training device is provided, comprising: a first acquisition module, a first determination module, and an adjustment module. The first acquisition module is used to input sample text data into the compression model to obtain first matching probabilities for multiple predetermined categories; the sample text data is annotated with sample category information. The first determination module is used to determine the loss value of the compression model based on the sample category information and the first matching probability. The adjustment module is used to adjust the model parameters of the compression model based on the loss value to obtain a trained compression model. The compression model is a compressed deep learning model obtained using the above-mentioned model compression method.

[0011] According to another aspect of the present disclosure, a text data processing device is provided, comprising: a second acquisition module and a second determination module. The second acquisition module is configured to input the text data to be processed into a compression model to obtain a matching probability representing the category to which the text data belongs; the matching probability includes a probability value for the text data belonging to each of a plurality of predetermined categories. The second determination module is configured to determine the target category to which the text data belongs based on the matching probability; the compression model is a compressed deep learning model obtained using the aforementioned model compression method.

[0012] According to another aspect of the present disclosure, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the method described above.

[0013] According to another aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable the computer to execute the method described above.

[0014] According to another aspect of the present disclosure, a computer program product is provided, comprising a computer program, which implements the method described above when executed by a processor.

[0015] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present disclosure, nor are they intended to limit the scope of the present disclosure. Other features of the present disclosure will become readily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] The accompanying drawings are provided to facilitate a better understanding of the present invention and do not constitute a limitation of the present disclosure.

[0017] FIG1 schematically illustrates an exemplary system architecture to which a model compression method, a training method, or a text data processing method and apparatus can be applied according to an embodiment of the present disclosure;

[0018] FIG2 schematically shows a flow chart of a model compression method according to an embodiment of the present disclosure;

[0019] FIG3A schematically illustrates a schematic diagram of performing matrix multiplication calculations with computing units deployed in column parallel according to an embodiment of the present disclosure;

[0020] FIG3B schematically illustrates a schematic diagram of performing matrix multiplication calculations with computing units deployed in parallel in rows according to an embodiment of the present disclosure;

[0021] FIG4 schematically shows an architecture diagram of a model to be compressed according to an embodiment of the present disclosure;

[0022] FIG5 schematically shows a diagram of weight rearrangement of a feedforward neural network of a model to be compressed according to an embodiment of the present disclosure;

[0023] FIG6 schematically shows a flow chart of a method for training a compression model according to an embodiment of the present disclosure;

[0024] FIG7 schematically shows a flow chart of a text data processing method according to an embodiment of the present disclosure;

[0025] FIG8 schematically shows a block diagram of a model compression device according to an embodiment of the present disclosure;

[0026] FIG9 schematically shows a block diagram of a training apparatus for a compression model according to an embodiment of the present disclosure;

[0027] FIG10 schematically shows a block diagram of a text data processing apparatus according to an embodiment of the present disclosure; and

[0028] FIG11 schematically shows a block diagram of an electronic device suitable for implementing a model compression method, a compression model training method, or a text data processing method according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0029] The following description of exemplary embodiments of the present disclosure is made in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding. These details should be considered as merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0030] In the field of natural language processing, as the accuracy requirements for large language generation models increase, the number of parameters in these models is also increasing. To enable pre-trained large language generation models to be used on devices with low computing power or in applications with high latency requirements, pre-trained models can be compressed to reduce their size and inference time.

[0031] Quantizing model parameters is an effective way to achieve model compression. Model quantization primarily involves converting the weight parameters and / or activation values ​​within the model (for example, converting some or all floating-point data to low-bit integer data) to speed up model inference and reduce hardware requirements. For example, converting float32 (32-bit floating-point) data to int8 (8-bit fixed-point) or int4 (4-bit fixed-point) data.

[0032] However, the weight parameters of the Transformer matrix multiplication in current large-scale language generation models are relatively dispersed, with a large number of outliers. When quantizing large-scale language generation models, the presence of these discrete points causes a significant loss of accuracy.

[0033] In view of this, an embodiment of the present disclosure provides a model compression method, including: dividing the initial model parameters of the model to be compressed according to the number of computing units deployed in parallel, and obtaining initial local model parameters corresponding to multiple computing units, wherein the computing units are used to process the same task associated with the model to be compressed, and the initial local model parameters include initial weight parameters on the local processing channel corresponding to the computing units and initial input activation values ​​determined based on the initial weight parameters; according to the initial input activation values, re-matching the correspondence between the initial weight parameters and the computing units to obtain target local model parameters corresponding to multiple computing units, wherein the target input activation values ​​in the target local model parameters gradually increase along the direction of data processing; and quantizing the target local model parameters to obtain a compressed model.

[0034] FIG1 schematically illustrates an exemplary system architecture to which a model compression method, a compression model training method, or a text data processing method and apparatus can be applied according to an embodiment of the present disclosure.

[0035] It should be noted that FIG1 is merely an example of a system architecture to which the embodiments of the present disclosure can be applied, to help those skilled in the art understand the technical content of the present disclosure, but does not mean that the embodiments of the present disclosure cannot be used in other devices, systems, environments, or scenarios. For example, in another embodiment, the exemplary system architecture to which the model compression method and apparatus can be applied may include a terminal device, but the terminal device can implement the model compression method and apparatus provided by the embodiments of the present disclosure without interacting with a server.

[0036] As shown in FIG1 , the system architecture 100 according to this embodiment may include a first terminal device 101, a second terminal device 102, a third terminal device 103, a network 104, and a server 105. The network 104 is a medium for providing communication links between the first terminal device 101, the second terminal device 102, the third terminal device 103, and the server 105. The network 104 may include various connection types, such as wired and / or wireless communication links, etc.

[0037] The user may use the first terminal device 101, the second terminal device 102, and the third terminal device 103 to interact with the server 105 via the network 104 to receive or send messages, etc. Various communication client applications may be installed on the first terminal device 101, the second terminal device 102, and the third terminal device 103, such as knowledge reading applications, web browser applications, search applications, instant messaging tools, email clients, and / or social platform software (for example only).

[0038] The first terminal device 101 , the second terminal device 102 , and the third terminal device 103 may be various electronic devices having display screens and supporting web browsing, including but not limited to smart phones, tablet computers, laptop computers, desktop computers, and the like.

[0039] The server 105 may be a server that provides various services, such as a background management server (for example only) that supports the content browsed by the user using the first terminal device 101, the second terminal device 102, and the third terminal device 103. The background management server may analyze and process received data such as user requests, and feed back processing results (such as web pages or data obtained or generated according to user requests) to the terminal device.

[0040] It should be noted that the model compression method, compression model training method, or text data processing method provided in the embodiments of the present disclosure can generally be executed by the first terminal device 101, the second terminal device 102, or the third terminal device 103. Accordingly, the model compression device, compression model training device, or text data processing device provided in the embodiments of the present disclosure can also be set in the first terminal device 101, the second terminal device 102, or the third terminal device 103.

[0041] Alternatively, the model compression method, compression model training method or text data processing method provided in the embodiments of the present disclosure may also generally be executed by the server 105. Accordingly, the model compression device, compression model training device or text data processing device provided in the embodiments of the present disclosure may generally be set in the server 105. The model compression method, compression model training method or text data processing method provided in the embodiments of the present disclosure may also be executed by a server or server cluster that is different from the server 105 and can communicate with the first terminal device 101, the second terminal device 102 or the third terminal device 103 and / or the server 105. Accordingly, the model compression device, compression model training device or text data processing device provided in the embodiments of the present disclosure may also be set in a server or server cluster that is different from the server 105 and can communicate with the first terminal device 101, the second terminal device 102 or the third terminal device 103 and / or the server 105.

[0042] It should be understood that the number of terminal devices, networks, and servers in Figure 1 is merely illustrative and any number of terminal devices, networks, and servers may be provided as required.

[0043] In the technical solution disclosed herein, the collection, storage, use, processing, transmission, provision, disclosure and application of personal user information are in compliance with the provisions of relevant laws and regulations, necessary confidentiality measures have been taken, and do not violate public order and good morals.

[0044] In the technical solution of the present disclosure, the user's authorization or consent is obtained before obtaining or collecting the user's personal information.

[0045] FIG2 schematically shows a flow chart of a model compression method according to an embodiment of the present disclosure.

[0046] As shown in FIG. 2 , the method includes operations S210 to S230 .

[0047] In operation S210 , initial model parameters of the model to be compressed are divided according to the number of computing units deployed in parallel, to obtain initial local model parameters corresponding to the multiple computing units.

[0048] In operation S220 , the correspondence between the initial weight parameters and the computing units is re-matched according to the initial input activation values ​​to obtain target local model parameters corresponding to the plurality of computing units.

[0049] In operation S230 , the target local model parameters are quantized to obtain a compressed model.

[0050] According to an embodiment of the present disclosure, the model to be compressed can be a large language generation model. The computing units used to process the same task associated with the model to be compressed can be deployed in parallel on the channels of the same hidden layer of the model to be compressed, for example: the hidden layer of the query feature matrix (Q matrix), the hidden layer of the key feature matrix (K matrix), and the hidden layer of the value feature matrix (V matrix) used to calculate the attention weight in the Transformer network architecture. The computing unit can be a GPU (Graphics Processing Unit).

[0051] According to an embodiment of the present disclosure, in a distributed deployment scenario, the parallel deployment mode of the computing units can be row parallel or column parallel. Taking the weight parameter matrix as an example, row parallel means that the elements in the same row of the weight parameter matrix are used as units, and the channels of the hidden layer corresponding to each computing unit include the weight parameters of a certain row or several adjacent rows in the weight parameter matrix. Column parallel means that the elements in the same column of the weight parameter matrix are used as units, and the channels of the hidden layer corresponding to each computing unit include the weight parameters of a certain column or several adjacent columns in the weight parameter matrix.

[0052] For example, the hidden layer in Transformer is usually column-parallel, while the output layer in Transformer is usually row-parallel.

[0053] According to an embodiment of the present disclosure, the initial local model parameters include the initial weight parameters on the local processing channel corresponding to the computing unit and the initial input activation value determined based on the initial weight parameters. For example, by running the model to be compressed and performing multiple rounds of sampling, the initial weight parameters on the local processing channel corresponding to each computing unit in the model to be compressed and the initial input activation value determined based on the initial weight parameters can be collected. Initial input activation value

[0054] For example, for a hidden layer, eight computing units are deployed in column parallel, which can be n1 to n8. By running the model to be compressed, the initial input activation value X1 on the channel of the hidden layer corresponding to n1, the initial input activation value X2 on the channel of the hidden layer corresponding to n2, ..., and the initial input activation value X8 on the channel of the hidden layer corresponding to n8 can be collected from the initial model parameters. At this time, the correspondence between the initial weight parameters and the computing units is as follows: computing unit n1 corresponds to the weight parameters in the first column of the weight parameter matrix, computing unit n2 corresponds to the weight parameters in the second column of the weight parameter matrix, ..., and computing unit n3 corresponds to the weight parameters in the eighth column of the weight parameter matrix.

[0055] Rearrange the weight parameters corresponding to each computational unit in ascending order of initial input activation value. For example, if initial input activation value X1 > initial input activation value X2, then the rearranged weight parameters corresponding to computational unit n1 are in the second column of the weight parameter matrix, and the weight parameters corresponding to computational unit n2 are in the first column of the weight parameter matrix. Similarly, after rearranging the weight parameters corresponding to each computational unit, you can run the model again to collect the target local model parameters.

[0056] At this time, because the weight parameters within each hidden layer's channel are rearranged and matched in ascending order of the initial input activation values, the target input activation value X1' determined by the weight parameters within the channel corresponding to computational unit n1 is smaller than the target input activation values ​​X2' to X8' determined by the weight parameters within the channels corresponding to other computational units n2 to n8 in the same hidden layer. In other words, the target input activation values ​​in the target local model parameters gradually increase along the direction of data processing.

[0057] Therefore, the distribution of the target input activation values ​​determined by the weight parameters corresponding to computational unit n1 is more concentrated than the distribution of the initial input activation values. For example, the distribution range of the initial input activation value X1 may be 1 to 100, while the distribution range of the target input activation values ​​may be 1 to 10. This effectively reduces the loss of accuracy during the model quantization process when quantizing the target local model parameters.

[0058] According to the embodiments of the present disclosure, the initial model parameters of the model to be compressed are divided according to the number of computing units deployed in parallel, achieving physical division of the initial model parameters in a distributed deployment scenario and reducing the computational time during the model compression process. According to the initial input activation values, the correspondence between the initial weight parameters and the computing units is re-matched, so that the distribution of the input activation values ​​corresponding to each computing unit is more concentrated, thereby reducing the loss of model accuracy when quantizing the target local model parameters.

[0059] In a distributed deployment scenario, channels within the same hidden layer are deployed in different computational units. Similarly, each computational unit is required to handle data processing tasks for different hidden layers. For the same hidden layer, different computational units may have different numbers of weight parameters within the corresponding channels, and their processing speeds may also vary.

[0060] For example, in a hidden layer, computing units n1 and n2 are deployed in column-parallel fashion. The channel of computing unit n1 contains the weight parameters of the first and second columns of the weight parameter matrix (3x4 dimensions); the channel of computing unit n2 contains the weight parameters of the third column of the weight parameter matrix. For the same data feature matrix, the matrix multiplication result in this hidden layer is obtained by matrix multiplying the data feature matrix with the weight parameters in the two computing units, and then concatenating them. Therefore, when the number of parameters in the two computing units is unequal, the matrix multiplication calculation speeds of the two computing units will be different. It is necessary to wait for the matrix multiplication result of n1 before executing the concatenation process, which reduces the data processing speed of the model.

[0061] In view of this, according to an embodiment of the present disclosure, a first target number of initial local model parameters corresponding to each computing unit can be obtained based on the number of computing units and the number of initial model parameters; and according to the first target number, the initial model parameters of the model to be compressed are evenly divided to obtain the initial local model parameters corresponding to the computing units.

[0062] For example, if the number of computing units deployed in parallel is N and the number of initial model parameters is M, the first target number can be obtained according to M / N, so that the initial model parameters are evenly distributed to each computing unit, thereby ensuring that the processing speed of each computing unit is the same.

[0063] For example: the initial model parameters can be a 10-column weight parameter matrix. If the number of calculation units is 5, then the first target number can be determined to be 2, that is, the initial local model parameters corresponding to the calculation unit n1 are the weight parameters of the 1st-2nd columns, and the initial local model parameters corresponding to the calculation unit n2 are the weight parameters of the 3rd-4th columns, and so on.

[0064] According to an embodiment of the present disclosure, the initial model parameters are evenly divided according to the number of computing units, so that the number of initial local model parameters corresponding to each computing unit is the same, and the data processing speeds of multiple computing units deployed in parallel in the same hidden layer are synchronized.

[0065] The distribution of weight parameters varies across large language generation models used in different scenarios. To improve the distribution concentration of target input activation values ​​after weight parameter rearrangement, the initial input activation values ​​in the initial model parameters can be clustered based on the number of computational units to obtain clustering results. Based on the clustering results, the initial model parameters of the model to be compressed can be partitioned to obtain initial local model parameters corresponding to the computational units.

[0066] According to embodiments of the present disclosure, a clustering algorithm can be used to cluster the initial input activation values ​​of the model to be compressed according to the number of computational units, thereby obtaining clusters corresponding to the number of computational units. Based on the data distribution of the initial input activation values ​​in each cluster, the number of initial local model parameters assigned to the computational units corresponding to that cluster can be determined.

[0067] According to an embodiment of the present disclosure, the number of initial local model parameters allocated to the computing units corresponding to each cluster may be determined based on the data distribution range of the initial input activation values ​​in the cluster.

[0068] For example, the data range of the initial input activation value in a cluster is 1 to 100. By configuring the mapping relationship between the input activation value threshold and the model parameter percentage, the percentage of the initial local model parameters allocated to the computing unit corresponding to the cluster is determined. Then, based on the total number of initial local model parameters and the percentage, the number of initial local model parameters allocated to the computing unit is obtained.

[0069] According to an embodiment of the present disclosure, the number of initial input activation values ​​in each cluster can be determined based on the clustering results; the second target number of initial local model parameters corresponding to each computing unit can be determined based on the number of initial input activation values ​​and the number of computing units; and the initial model parameters of the model to be compressed can be divided according to the second target number to obtain the initial local model parameters corresponding to the computing units.

[0070] For example, a clustering algorithm can be used to cluster the initial input activation values ​​of the compression model to obtain the initial input activation values ​​of multiple clusters. The number of input activation values ​​of each cluster can be counted to determine the number of weight parameters assigned to each cluster.

[0071] According to an embodiment of the present disclosure, the number of input activation values ​​of each cluster may be proportional to the number of weight parameters assigned to each cluster.

[0072] For example: the number of input activation values ​​of the initial input activation value in cluster P1 is 100, the number of input activation values ​​of the initial input activation value in cluster P2 is 1000, and the number of initial weight parameters can be 220. Then it can be determined that the number of initial local model parameters in the computing unit corresponding to cluster P1 is 20, and the number of initial local model parameters in the computing unit corresponding to cluster P2 is 200.

[0073] According to an embodiment of the present disclosure, the initial weight parameters are first clustered and then divided. Although the number of weight parameters assigned to each computing unit varies, this method can assign weight parameters used to determine similar input activation values ​​to a single computing unit, thereby improving the distribution concentration of the target input activation values ​​after the weight parameters are rearranged.

[0074] In model quantization scenarios, models are typically compressed based on the same quantization precision, such as INT4 or INT8. However, in actual applications, compression scenarios with mixed quantization precisions also exist.

[0075] According to an embodiment of the present disclosure, the quantization accuracy of each computing unit can be determined based on the initial input activation value in the initial model parameters; and the initial model parameters can be divided according to the quantization accuracy of each computing unit and the number of computing units to obtain initial local model parameters corresponding to multiple computing units.

[0076] For example, for three computing units deployed in parallel, the initial input activation values ​​corresponding to each computing unit are collected separately. Among them, the initial input activation value of the third computing unit is the largest. Therefore, the quantization precision of the first and second computing units can be determined to be 4 bits, and the quantization precision of the third computing unit can be determined to be 8 bits. Since computing units with low quantization precision execute faster, more initial model parameters can be allocated to computing units with low quantization precision. Computing units with high quantization precision execute slower, so fewer initial model parameters can be allocated to computing units with high quantization precision, thereby balancing the inference speed of computing units in the same hidden layer.

[0077] According to an embodiment of the present disclosure, the initial model parameters are divided according to the quantization accuracy of each computing unit and the number of computing units to obtain initial local model parameters corresponding to multiple computing units, which may include the following operations: determining a third target number of initial local model parameters corresponding to each computing unit according to the quantization accuracy of each computing unit and the number of computing units; and dividing the initial model parameters of the model to be compressed according to the third target number to obtain initial local model parameters corresponding to the computing units.

[0078] According to an embodiment of the present disclosure, the initial model parameters can be evenly divided according to the number of computing units to obtain the number of local model parameters for the initial division. The number of local model parameters for the initial division can then be modified according to the proportion of computing units with high bit quantization precision to obtain the number of local model parameters for computing units with different quantization precisions.

[0079] For example, there may be five computing units deployed in parallel. The quantization precision of the first four computing units may be 4 bits, and the quantization precision of the fifth computing unit may be 8 bits. The initial number of model parameters (1000) may be divided according to the number of computing units (5), resulting in an initial number of local model parameters of 200 for each computing unit. The number of local model parameters in the initial division is adjusted based on the ratio of the number of computing units with 8-bit quantization precision to the total number of computing units (1 / 5).

[0080] For example, the local model parameter correction amount = the number of local model parameters initially divided × the proportion of computing units with high quantization precision / the number of computing units with low quantization precision = 10. Based on the sum of the local model parameter correction amount of 10 and the number of local model parameters initially divided of 200, the number of local model parameters for computing units with 4-bit quantization precision is 210, and the remaining number of model parameters is the number of local model parameters for computing units with 8-bit quantization precision: 1000 - 210 × 4 = 160.

[0081] According to an embodiment of the present disclosure, the initial model parameters are divided according to the quantization accuracy and the number of computing units. The difference in the number of weight parameters of computing units with different quantization accuracy can be used to balance the difference in execution speed of computing units with different quantization accuracy, so that the model quantization speed in the mixed bit compression scenario tends to be balanced.

[0082] For the Transformer model, which involves a large number of matrix multiplication calculations, the rearrangement of weight parameters cannot change the calculation results themselves, that is, the calculation results before and after the weight parameters are rearranged should be mathematically equivalent.

[0083] FIG3A schematically shows a schematic diagram of performing matrix multiplication calculations with computing units deployed in column parallel according to an embodiment of the present disclosure.

[0084] As shown in FIG3A , in embodiment 300A, the matrix multiplication 301 of the input feature matrix and the weight parameter matrix can be assigned to calculation units 302a and 302b for execution in a column-parallel manner. The matrix multiplication of the first column of the input feature matrix and the weight parameter matrix is ​​performed by calculation unit 302a. The matrix multiplication of the second column of the input feature matrix and the weight parameter matrix is ​​performed by calculation unit 302b. The calculation results of calculation units 302a and 302b are concatenated to obtain a matrix multiplication result equivalent to 301.

[0085] FIG3B schematically illustrates a schematic diagram of performing matrix multiplication calculations with computing units deployed in parallel in rows according to an embodiment of the present disclosure.

[0086] As shown in FIG3B , in embodiment 300B, the matrix multiplication 301 of the input feature matrix and the weight parameter matrix can be assigned to calculation units 302c and 302d for execution in a row-parallel manner. The matrix multiplication of the input feature matrix and the first submatrix of the weight parameter matrix is ​​performed by calculation unit 302c. The matrix multiplication of the input feature matrix and the second submatrix of the weight parameter matrix is ​​performed by calculation unit 302d. The calculation results of calculation units 302c and 302d are added to obtain a matrix multiplication result equivalent to 301.

[0087] Based on the basic principle of matrix multiplication and segmentation mentioned above, the initial weight parameters can be sorted according to the initial input activation values ​​to obtain the arrangement order of the initial weight parameters; and the initial weight parameters can be distributed in sequence to the local processing channels corresponding to the computing units according to the arrangement order to obtain the target local model parameters.

[0088] According to an embodiment of the present disclosure, the initial weight parameters are sequentially allocated to the local processing channels corresponding to the computing units according to the arrangement order to obtain the target local model parameters, which may include the following operations: generating a target correspondence between the initial weight parameters and the computing units according to the arrangement order and the arrangement positions of the computing units; and sequentially allocating the initial weight parameters to the local processing channels corresponding to the computing units according to the target correspondence to obtain the target local model parameters.

[0089] For example, the channel corresponding to computing unit n1 may be C1, the channel corresponding to computing unit n2 may be C2, the channel corresponding to computing unit n3 may be C3, and the channel corresponding to computing unit n4 may be C4. The order of the initial input activation values ​​may be such that channel C1 is ranked third, channel C2 is ranked second, channel C3 is ranked fourth, and channel C4 is ranked first.

[0090] According to an embodiment of the present disclosure, the initial weight parameters can be realigned with the computing units by reordering the channels, and the generated target correspondence relationship can be: the weight parameters in the original channel C4 correspond to the computing unit n1, the weight parameters in the original channel C2 correspond to the computing unit n2, the weight parameters in the original channel C1 correspond to the computing unit n3, and the weight parameters in the channel C3 correspond to the computing unit n4.

[0091] According to an embodiment of the present disclosure, based on the segmentation principle of matrix multiplication, after the weight parameter matrix is ​​divided according to the calculation units, it is rearranged based on the order of the input activation values, which can effectively reduce the data distribution range of the input activation values ​​of each calculation unit, thereby reducing the loss of model accuracy caused by the quantization process.

[0092] Based on the splitting principle of the matrix multiplication described above, it can be seen that the calculation results of column-parallel and row-parallel computational units are processed differently. In the Transformer model, the computational units used for attention calculation in the hidden layer are column-parallel, while the computational units in the attention output layer are row-parallel. To maintain the accuracy of the model's prediction results, the weight parameters of each hidden layer's computational units need to be restored to their original order before calculating the matrix multiplication. Therefore, an index array can be introduced to quickly restore the original weight parameter matrix based on the index array.

[0093] According to an embodiment of the present disclosure, generating a target correspondence between initial weight parameters and computing units based on the arrangement order and the arrangement position of the computing units may include the following operations: generating an index array based on the arrangement order and the arrangement position of the computing units, wherein the index array includes a correspondence between the arrangement order and the arrangement position; and determining the target correspondence between the initial weight parameters and the computing units based on the index array.

[0094] For example, the initial input activation values ​​may be arranged in the order of channel C1 being the third, channel C2 being the second, channel C3 being the fourth, and channel C4 being the first. The generated index array may include the mapping relationship between computation unit n1 and channel C4, the mapping relationship between computation unit n2 and channel C2, and so on.

[0095] According to the embodiments of the present disclosure, based on the index array, the target correspondence between the rearranged computational unit n1 and the weight parameters can be obtained. Furthermore, since the channel number identifiers correspond one-to-one with the arrangement positions of the original computational units, the original weight parameter matrix can be quickly restored based on the index data. This ensures data equivalence before and after the weight rearrangement, maintaining the accuracy of the model prediction results.

[0096] FIG4 schematically shows an architecture diagram of a model to be compressed according to an embodiment of the present disclosure.

[0097] As shown in FIG4 , the architecture 400 of the model to be compressed may include: an attention network 410 and a feedforward neural network 420 .

[0098] The attention network 410 includes a first normalization layer (LayerNorm) 411, a value feature processing layer (V proj) 412_1, a query feature processing layer (Q proj) 412_2, a key feature processing layer (K ​​proj) 412_3, a first reordering recovery operator (Reorder) 413_1, a second reordering recovery operator (Reorder) 413_2, a third reordering recovery operator (Reorder) 413_3, a first matrix multiplication processing layer (BatchedMatMul) 414_1, a second matrix multiplication processing layer (BatchedMatMul) 414_2, a Softmax layer 415, an output layer (Out proj) 416, and a first feature fusion layer (add) 417.

[0099] In the attention network 410, the value feature processing layer (V proj) 412_1, the query feature processing layer (Q proj) 412_2, and the key feature processing layer (K ​​proj) 412_3 are all column-parallel linear layers. During offline quantization, the weight parameters in each column-parallel linear layer are quantized according to the model compression method described above. Therefore, a reordering operator is deployed at the output of each of the value feature processing layer (V proj) 412_1, the query feature processing layer (Q proj) 412_2, and the key feature processing layer (K ​​proj) 412_3. This operator reorders the weight parameters reordered during offline quantization according to the index array (R_index) derived from the initial input activation values. These weight parameters are then fed into the matrix multiplication layer for matrix multiplication, ensuring the accuracy of the model calculation results.

[0100] The feedforward neural network 420 includes a second normalization layer (LayerNorm) 421 , a first fully connected layer (FFN1, Feed Forward Networks) 422 , a second fully connected layer (FFN2, Feed Forward Networks) 423 and a second feature fusion layer (add) 424 .

[0101] In the feedforward neural network 420, since the calculation of the fully connected layer does not involve a complex matrix multiplication process, the input dimensions of the weights of the first fully connected layer 421 can be rearranged, and then the output dimensions of the weights of the second fully connected layer 422 can be rearranged according to the same index array (R_index). The weight rearrangement process can be completed without deploying any operators.

[0102] FIG5 schematically shows a diagram of weight rearrangement of a feedforward neural network of a model to be compressed according to an embodiment of the present disclosure.

[0103] As shown in Figure 5, in embodiment 500, the features output by the attention module 510 are normalized by the LayerNorm layer 521, and then copied to the input FFN1 layer 522. Four computational units, Card1 through Card4, are deployed in parallel in the FFN1 layer 522. Weight parameter matrices A5221, B5222, C5223, and D5224 are distributed to computational units Card1 through Card4 in a column-parallel manner. Because weight parameter matrices A5221, B5222, C5223, and D5224 are weighted matrices that have been rearranged based on the initial input activation values, the output feature dimensions are different for the same input feature 501.

[0104] According to an embodiment of the present disclosure, the grayscale of the color block can represent the size of the input activation value. The darker the grayscale of the color block, the larger the input activation value. It can be seen that for the input feature 501 input to the FFN1 layer 522, the distribution of its input activation value is relatively dispersed (the color distribution of the color block is uneven). After processing by the calculation unit after rearranging the weight parameters, the input activation value distribution of the output feature 502 is relatively concentrated, and the color of the color block in the output feature 502 is relatively light, indicating that the input activation value of the output feature 502 is relatively small. The input activation value distribution of the output feature 505 is relatively concentrated, and the color of the color block in the output feature 505 is relatively dark, indicating that the input activation value of the output feature 505 is relatively large. Among the features output from the calculation unit Card1 to the calculation unit Card4, the area of ​​the dark blocks in the output features 502 to the output features 505 gradually increases, indicating that the input activation values ​​corresponding to the output features 502 to the output features 505 gradually increase.

[0105] According to an embodiment of the present disclosure, in the FFN2 layer 523, the weight parameter matrix a5231, weight parameter matrix b5232, weight parameter matrix c5233, and weight parameter matrix d5234 are distributed to the computing units Card1 to Card4 in a row-parallel manner. The weight parameter matrix a5231, weight parameter matrix b5232, weight parameter matrix c5233, and weight parameter matrix d5234 are obtained by rearranging the weight parameter matrix of the FFN2 layer based on the same index array (R_index) as FFN1. Based on the basic principle of matrix multiplication and slicing described above, the output features 5231 to 5234 of the computing units deployed in row parallel need to be added to obtain the output feature 506.

[0106] According to the embodiment of the present disclosure, the add layer 505 is used for feature fusion to obtain prediction results. It is a conventional processing layer in the Transformer model and will not be described in detail here.

[0107] The offline quantization process is to collect the input activation value of the layer to be quantized to determine the truncation value of the input activation, and convert the layer to be quantized into a low-bit operation based on the truncation value to achieve model compression.

[0108] In a distributed deployment scenario, the embodiment of the present disclosure adopts a parallel processing strategy for each computing unit, that is, each computing unit separately collects and counts the input activation values ​​of the target local model parameters in each hidden layer to perform quantization of the hidden layer.

[0109] According to an embodiment of the present disclosure, quantizing the target local model parameters to obtain a compression model may include the following operations: determining a quantization factor for the target local model parameters corresponding to each computing unit; and quantizing the target local model parameters corresponding to each computing unit according to the quantization factor to obtain a compression model.

[0110] According to an embodiment of the present disclosure, a cutoff threshold is determined for each hidden layer within a target input activation value range for each computational unit. For example, the target input activation value range may be 1 to 100, and the maximum target input activation value may be determined as the cutoff threshold, i.e., the cutoff threshold is 100.

[0111] According to an embodiment of the present disclosure, for the range of target input activation values ​​of each computing unit in each hidden layer, the model can be run multiple times to collect the range of target input activation values ​​of the computing unit in the hidden layer in each round, and the average of multiple maximum values ​​of the target input activation values ​​collected in multiple rounds can be determined as the truncation threshold.

[0112] For example, if the maximum value of the target input activation value collected in the first round is 100 and the maximum value of the target input activation value collected in the second round is 90, the cutoff threshold can be determined to be 95.

[0113] According to an embodiment of the present disclosure, based on a parallel processing strategy, the target input activation value of each computing unit is individually counted to determine the quantization factor of each computing unit in the corresponding quantization layer, which can effectively improve the quantization speed.

[0114] According to an embodiment of the present disclosure, for the range of target input activation values ​​of each computing unit in each hidden layer, the truncation threshold can be determined based on the distribution information by obtaining the distribution information of multiple target input activation values ​​corresponding to the target weight parameters.

[0115] For example, we can cluster multiple target input activation values ​​based on a clustering algorithm to obtain outliers. We can use the input activation value corresponding to the point closest to the outlier as the cutoff threshold, or we can use the input activation value corresponding to the outlier closest to the cluster center as the cutoff threshold.

[0116] Since the distribution of target input activation values ​​is relatively concentrated after weight rearrangement, individual outliers have little impact on the loss of model accuracy. It is also possible to filter multiple target input activation values ​​based on the distribution information to obtain concentrated target input activation values, and determine the truncation threshold from the concentrated target input activation values.

[0117] For example, outliers are filtered out from the target input activation values, and the maximum input activation value among the remaining concentrated target input activation values ​​is determined as the truncation threshold.

[0118] According to an embodiment of the present disclosure, by determining a truncation threshold based on the distribution of target input activation values, the quantization speed can be improved without increasing the accuracy loss of the model quantization process.

[0119] FIG6 schematically shows a flow chart of a method for training a compression model according to an embodiment of the present disclosure.

[0120] As shown in FIG6 , the training method 600 may include operations S610 to S630 .

[0121] In operation S610 , sample text data is input into a compression model to obtain first matching probabilities for a plurality of predetermined categories.

[0122] In operation S620, a loss value of the compression model is determined based on the sample category information and the first matching probability.

[0123] In operation S630 , model parameters of the compression model are adjusted according to the loss value to obtain a trained compression model.

[0124] According to an embodiment of the present disclosure, the compression model can be obtained by loading the compressed deep learning model obtained by the compression method of the deep learning model described above.

[0125] According to embodiments of the present disclosure, the sample text data may be part or all of the sample text data in the training set used to train the initial model (i.e., the deep learning model before compression). For example, 10%, 15%, or other proportions of sample text data may be randomly extracted from the training set according to actual needs.

[0126] According to an embodiment of the present disclosure, the first matching probability may represent a matching probability value obtained by the compression model from predicting the sample text data that the sample text data belongs to a category among a plurality of predetermined categories. The sample text data may be annotated with the real sample category information of the sample text data.

[0127] According to an embodiment of the present disclosure, a loss value of the compression model can be obtained based on a loss function, sample category information, and the first matching probability. For example, the loss function can be a cross entropy loss function, a mean square error loss function (i.e., an L2 loss function), or a hinge loss function, etc., which is not limited in this disclosure.

[0128] According to the embodiments of the present disclosure, the gradient descent algorithm can be used to adjust the network parameters such as the weight parameters in the compression model with the goal of minimizing the loss value, so as to train the compression model until the loss value is minimized and a trained compression model is obtained.

[0129] According to an embodiment of the present disclosure, by training the compression model using sample text data, the prediction accuracy of the compression model can be further improved.

[0130] FIG7 schematically shows a flow chart of a text data processing method according to an embodiment of the present disclosure.

[0131] As shown in FIG. 7 , the text data processing method 700 may include operations S710 to S720 .

[0132] In operation S710 , text data to be processed is input into a compression model to obtain a matching probability representing a category to which the text data belongs.

[0133] In operation S720 , the target category to which the text data belongs is determined based on the matching probability.

[0134] According to an embodiment of the present disclosure, the implementation principle of operation S710 is similar to the implementation principle of operation S610 described above. The matching probability may include a probability value of the text data belonging to each of the plurality of predetermined categories, which will not be described in detail here. It is understood that the compression model may be, for example, a trained compression model obtained using the compression model training method described above.

[0135] According to an embodiment of the present disclosure, the category corresponding to the maximum matching probability may be determined as the target category of the text data.

[0136] According to the embodiments of the present disclosure, by processing text data using a compression model, it is possible to complete the processing of relatively complex text data within hardware with limited computing resources, thereby reducing the limitation of computing resource usage on the application scope of deep learning models.

[0137] FIG8 schematically shows a block diagram of a model compression device according to an embodiment of the present disclosure.

[0138] As shown in FIG. 8 , the model compression device 800 may include: a parameter division module 810 , a parameter rearrangement module 820 and a quantization module 830 .

[0139] The parameter division module 810 is used to divide the initial model parameters of the model to be compressed according to the number of computing units deployed in parallel, and obtain initial local model parameters corresponding to multiple computing units, wherein the computing units are used to process the same task associated with the model to be compressed, and the initial local model parameters include the initial weight parameters on the local processing channel corresponding to the computing units and the initial input activation values ​​determined based on the initial weight parameters.

[0140] The parameter rearrangement module 820 is used to rematch the correspondence between the initial weight parameters and the computing units according to the initial input activation values ​​to obtain target local model parameters corresponding to multiple computing units, wherein the target input activation values ​​in the target local model parameters gradually increase along the direction of data processing.

[0141] The quantization module 830 is used to quantize the target local model parameters to obtain a compressed model.

[0142] According to an embodiment of the present disclosure, a parameter partitioning module may include: a first obtaining submodule and a first partitioning submodule. The first obtaining submodule is configured to obtain a first target number of initial local model parameters corresponding to each computing unit based on the number of computing units and the number of initial model parameters. The first partitioning submodule is configured to evenly partition the initial model parameters of the model to be compressed according to the first target number to obtain initial local model parameters corresponding to the computing units.

[0143] According to an embodiment of the present disclosure, the parameter partitioning module may include: a clustering submodule and a second partitioning submodule. The clustering submodule is configured to cluster initial input activation values ​​in the initial model parameters based on the number of computing units to obtain clustering results. The second partitioning submodule is configured to partition the initial model parameters of the model to be compressed based on the clustering results to obtain initial local model parameters corresponding to the computing units.

[0144] According to an embodiment of the present disclosure, the clustering submodule may include: a first determination unit, a second determination unit, and a first division unit. The first determination unit is configured to determine the number of initial input activation values ​​in each cluster based on the clustering results. The second determination unit is configured to determine a second target number of initial local model parameters corresponding to each computing unit based on the number of initial input activation values ​​and the number of computing units. The first division unit is configured to divide the initial model parameters of the model to be compressed according to the second target number to obtain the initial local model parameters corresponding to the computing units.

[0145] According to an embodiment of the present disclosure, a parameter partitioning module may include: a first determination submodule and a third partitioning submodule. The first determination submodule is configured to determine the quantization precision of each computing unit based on the initial input activation value in the initial model parameters. The third partitioning submodule is configured to partition the initial model parameters based on the quantization precision of each computing unit and the number of computing units, thereby obtaining initial local model parameters corresponding to the multiple computing units.

[0146] According to an embodiment of the present disclosure, the third division submodule may include: a third determination unit and a second division unit. The third determination unit is configured to determine a third target number of initial local model parameters corresponding to each computing unit based on the quantization accuracy of each computing unit and the number of computing units. The second division unit is configured to divide the initial model parameters of the model to be compressed according to the third target number to obtain the initial local model parameters corresponding to the computing units.

[0147] According to an embodiment of the present disclosure, the parameter rearrangement module may include a sorting submodule and an allocation submodule. The sorting submodule is configured to sort the initial weight parameters according to the initial input activation values ​​to obtain an arrangement order of the initial weight parameters. The allocation submodule is configured to sequentially allocate the initial weight parameters to the local processing channels corresponding to the computing units according to the arrangement order to obtain target local model parameters.

[0148] According to an embodiment of the present disclosure, the allocation submodule may include: a generation unit and an allocation unit. The generation unit is configured to generate a target correspondence between initial weight parameters and the computation units based on the arrangement order and arrangement positions of the computation units. The allocation unit is configured to sequentially allocate the initial weight parameters to the local processing channels corresponding to the computation units according to the target correspondence, thereby obtaining target local model parameters.

[0149] According to an embodiment of the present disclosure, the generation unit may include: a generation subunit and a first determination subunit. The generation subunit is configured to generate an index array based on the arrangement order and the arrangement position of the calculation unit, wherein the index array includes a correspondence between the arrangement order and the arrangement position. The first determination subunit is configured to determine, based on the index array, a target correspondence between the initial weight parameter and the calculation unit.

[0150] According to an embodiment of the present disclosure, the quantization module may include: a second determination submodule and a quantization submodule. The second determination submodule is configured to determine a quantization factor for a target local model parameter corresponding to each computing unit. The quantization submodule is configured to quantize the target local model parameter corresponding to each computing unit according to the quantization factor to obtain a compression model.

[0151] According to an embodiment of the present disclosure, the second determination submodule may include: a fourth determination unit and a fifth determination unit. The fourth determination unit is configured to determine, for each target local model parameter corresponding to each computing unit, a truncation threshold corresponding to the target weight parameter. The fifth determination unit is configured to determine a quantization factor based on the truncation threshold.

[0152] According to an embodiment of the present disclosure, the fourth determination unit may include an acquisition subunit and a second determination subunit. The acquisition subunit is configured to acquire, for each target local model parameter corresponding to each computation unit, distribution information of a plurality of target input activation values ​​corresponding to a target weight parameter. The second determination subunit is configured to determine a truncation threshold based on the distribution information.

[0153] According to an embodiment of the present disclosure, the second determination subunit is used to: filter multiple target input activation values ​​according to distribution information to obtain concentrated target input activation values; and determine a truncation threshold from the concentrated target input activation values.

[0154] FIG9 schematically shows a block diagram of a training apparatus for a compression model according to an embodiment of the present disclosure.

[0155] As shown in Figure 9, the training device 900 may include: a first acquisition module 910, a first determination module 920, and an adjustment module 930. The first acquisition module is used to input sample text data into the compression model to obtain a first matching probability for multiple predetermined categories; the sample text data is annotated with sample category information. The first determination module is used to determine the loss value of the compression model based on the sample category information and the first matching probability. The adjustment module is used to adjust the model parameters of the compression model based on the loss value to obtain a trained compression model. The compression model is a compressed deep learning model obtained using the model compression method described above.

[0156] FIG10 schematically shows a block diagram of a text data processing apparatus according to an embodiment of the present disclosure.

[0157] As shown in Figure 10, the text data processing device 1000 may include a second obtaining module 1010 and a second determining module 1020. The second obtaining module 1010 is configured to input the text data to be processed into a compression model to obtain a matching probability representing the category to which the text data belongs; the matching probability includes a probability value for the text data belonging to each of a plurality of predetermined categories. The second determining module 1020 is configured to determine the target category to which the text data belongs based on the matching probability; the compression model is a compressed deep learning model obtained using the model compression method described above.

[0158] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0159] According to an embodiment of the present disclosure, an electronic device includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the method described above.

[0160] According to an embodiment of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to cause a computer to execute the method described above.

[0161] According to an embodiment of the present disclosure, a computer program product includes a computer program, and when the computer program is executed by a processor, it implements the method described above.

[0162] FIG11 shows a schematic block diagram of an example electronic device 1100 that can be used to implement an embodiment of the present disclosure. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0163] As shown in FIG11 , the device 1100 includes a computing unit 1101 that can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 1102 or a computer program loaded from a storage unit 1108 into a random access memory (RAM) 1103. Various programs and data required for the operation of the device 1100 may also be stored in the RAM 1103. The computing unit 1101, the ROM 1102, and the RAM 1103 are connected to each other via a bus 1104. An input / output (I / O) interface 1105 is also connected to the bus 1104.

[0164] Multiple components in device 1100 are connected to I / O interface 1105, including: an input unit 1106, such as a keyboard, mouse, etc.; an output unit 1107, such as various types of displays, speakers, etc.; a storage unit 1108, such as a magnetic disk, optical disk, etc.; and a communication unit 1109, such as a network card, modem, wireless communication transceiver, etc. The communication unit 1109 allows device 1100 to exchange data with other devices via a computer network such as the Internet and / or various telecommunication networks.

[0165] The computing unit 1101 can be a variety of general-purpose and / or specialized processing components with processing and computing capabilities. Some examples of the computing unit 1101 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units that run machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The computing unit 1101 performs the various methods and processes described above, such as the model compression method, the compression model training method, or the text data processing method. For example, in some embodiments, the model compression method, the compression model training method, or the text data processing method can be implemented as a computer software program that is tangibly contained in a machine-readable medium, such as the storage unit 1108. In some embodiments, part or all of the computer program can be loaded and / or installed on the device 1100 via the ROM 1102 and / or the communication unit 1109. When the computer program is loaded into the RAM 1103 and executed by the computing unit 1101, one or more steps of the model compression method, the compression model training method, or the text data processing method described above can be performed. Alternatively, in other embodiments, the computing unit 1101 may be configured to execute a model compression method, a compression model training method, or a text data processing method in any other appropriate manner (e.g., by means of firmware).

[0166] Various embodiments of the systems and techniques described above can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), system-on-chip systems (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include being implemented in one or more computer programs that are executable and / or interpreted on a programmable system that includes at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0167] The program code for implementing the method of the present disclosure can be written in any combination of one or more programming languages. These program codes can be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device so that when the program code is executed by the processor or controller, the functions / operations specified in the flow chart and / or block diagram are implemented. The program code can be executed entirely on the machine, partially on the machine, as a stand-alone software package, partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0168] In the context of the present disclosure, a machine-readable medium can be a tangible medium that can contain or store a program for use by or in conjunction with an instruction execution system, device or equipment. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium can include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0169] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0170] The systems and techniques described herein can be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer having a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network (LAN), a wide area network (WAN), and the Internet.

[0171] A computer system may include a client and a server. The client and server are generally remote from each other and typically interact through a communication network. The client-server relationship arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. The server may be a cloud server, a server in a distributed system, or a server integrated with a blockchain.

[0172] It should be understood that the various forms of the processes shown above can be used to reorder, add, or delete steps. For example, the steps described in this disclosure can be performed in parallel, sequentially, or in a different order, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved. This is not limited herein.

[0173] The above specific embodiments do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art will appreciate that various modifications, combinations, sub-combinations, and substitutions may be made based on design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure shall be included within the scope of protection of this disclosure.

Claims

1. A model compression method, comprising: Dividing the initial model parameters of the model to be compressed according to the number of computing units deployed in parallel to obtain initial local model parameters corresponding to the plurality of computing units, wherein the computing units are used to process the same task associated with the model to be compressed, and the initial local model parameters include initial weight parameters on the local processing channels corresponding to the computing units and initial input activation values ​​determined based on the initial weight parameters; According to the initial input activation value, rematching the correspondence between the initial weight parameter and the computing unit to obtain target local model parameters corresponding to the plurality of computing units, wherein the target input activation value in the target local model parameter gradually increases along the direction of data processing; and The target local model parameters are quantized to obtain a compressed model.

2. The method according to claim 1, wherein: The initial model parameters of the model to be compressed are divided according to the number of computing units deployed in parallel to obtain initial local model parameters corresponding to the plurality of computing units, including: According to the number of the computing units and the number of the initial model parameters, obtaining a first target number of initial local model parameters corresponding to each computing unit; and According to the first target quantity, the initial model parameters of the model to be compressed are evenly divided to obtain initial local model parameters corresponding to the calculation unit.

3. The method according to claim 1, wherein: The initial model parameters of the model to be compressed are divided according to the number of computing units deployed in parallel to obtain initial local model parameters corresponding to the plurality of computing units, including: Based on the number of the computing units, clustering the initial input activation values ​​in the initial model parameters to obtain a clustering result; and Based on the clustering result, the initial model parameters of the model to be compressed are Divide and obtain initial local model parameters corresponding to the computing unit.

4. The method according to claim 3, wherein: The dividing the initial model parameters of the model to be compressed based on the clustering result to obtain the initial local model parameters corresponding to the calculation unit includes: Based on the clustering results, determining the number of initial input activation values ​​in each cluster; Determining a second target number of initial local model parameters corresponding to each computing unit based on the number of the initial input activation values ​​and the number of the computing units; and According to the second target quantity, the initial model parameters of the model to be compressed are divided to obtain initial local model parameters corresponding to the calculation units.

5. The method according to claim 1, wherein: The initial model parameters of the model to be compressed are divided according to the number of computing units deployed in parallel to obtain initial local model parameters corresponding to the plurality of computing units, including: Determining the quantization accuracy of each computing unit according to the initial input activation value in the initial model parameters; and The initial model parameters are divided according to the quantization accuracy of each computing unit and the number of the computing units to obtain initial local model parameters corresponding to the multiple computing units.

6. The method according to claim 5, wherein: The dividing the initial model parameters according to the quantization accuracy of each computing unit and the number of the computing units to obtain initial local model parameters corresponding to the multiple computing units includes: Determining a third target number of initial local model parameters corresponding to each of the computing units according to the quantization accuracy of each of the computing units and the number of the computing units; and According to the third target quantity, the initial model parameters of the model to be compressed are The division is performed to obtain initial local model parameters corresponding to the calculation unit.

7. The method according to claim 1, wherein: The re-matching of the correspondence between the initial weight parameters and the computing units according to the initial input activation values ​​to obtain target local model parameters corresponding to the multiple computing units includes: Sorting the initial weight parameters according to the initial input activation values ​​to obtain an arrangement order of the initial weight parameters; and According to the arrangement order, the initial weight parameters are sequentially allocated to the local processing channels corresponding to the computing units to obtain the target local model parameters.

8. The method according to claim 7, wherein: The step of allocating the initial weight parameters to the local processing channels corresponding to the computing units in sequence according to the arrangement order to obtain the target local model parameters includes: generating a target correspondence between the initial weight parameter and the computing unit according to the arrangement order and the arrangement position of the computing unit; and According to the target corresponding relationship, the initial weight parameters are sequentially allocated to the local processing channels corresponding to the computing units to obtain the target local model parameters.

9. The method according to claim 8, wherein: Generating the target correspondence between the initial weight parameter and the computing unit according to the arrangement order and the arrangement position of the computing unit includes: Generate an index array according to the arrangement order and the arrangement positions of the calculation units, wherein the index array includes a corresponding relationship between the arrangement order and the arrangement positions; and According to the index array, a target correspondence between the initial weight parameter and the calculation unit is determined.

10. The method according to claim 1, wherein: The step of quantizing the target local model parameters to obtain a compression model includes: Determining a quantization factor for a target local model parameter corresponding to each computing unit; and According to the quantization factor, the target local model parameters corresponding to each calculation unit are quantized to obtain the compression model.

11. The method according to claim 10, wherein: The step of determining a quantization factor for a target local model parameter corresponding to each computing unit includes: For the target local model parameter corresponding to each computing unit, determining a truncation threshold corresponding to the target weight parameter; and Based on the truncation threshold, the quantization factor is determined.

12. The method according to claim 11, wherein: The step of determining a truncation threshold corresponding to a weight parameter of a target processing layer for a target local model parameter corresponding to each computing unit includes: For the target local model parameters corresponding to each computing unit, obtaining distribution information of a plurality of target input activation values ​​corresponding to the target weight parameters; and The truncation threshold is determined according to the distribution information.

13. The method according to claim 12, wherein: The step of determining the cutoff threshold according to the distribution information includes: filtering the plurality of target input activation values ​​according to the distribution information to obtain a concentrated distribution of target input activation values; and The cutoff threshold is determined from the centrally distributed target input activation values.

14. A method for training a compression model, comprising: Inputting the sample text data into the compression model to obtain first matching probabilities for a plurality of predetermined categories; The sample text data is annotated with sample category information; Determining a loss value of the compression model according to the sample category information and the first matching probability; as well as According to the loss value, the model parameters of the compression model are adjusted to obtain a trained compression model; wherein the compression model is a compressed deep learning model obtained by the method described in any one of claims 1 to 13.

15. A text data processing method, comprising: Inputting the text data to be processed into the compression model to obtain a matching probability representing the category to which the text data belongs; The matching probability includes a probability value that the text data belongs to each of a plurality of predetermined categories; as well as According to the matching probability, the target category to which the text data belongs is determined; wherein the compression model is a compressed deep learning model obtained by the method described in any one of claims 1 to 13.

16. A model compression device, comprising: A parameter division module, used to divide the initial model parameters of the model to be compressed according to the number of computing units deployed in parallel, and obtain initial local model parameters corresponding to the plurality of computing units, wherein the computing units are used to process the same task associated with the model to be compressed, and the initial local model parameters include initial weight parameters on the local processing channels corresponding to the computing units and initial input activation values ​​determined based on the initial weight parameters; a parameter rearrangement module, configured to re-match the correspondence between the initial weight parameters and the computing units according to the initial input activation values, to obtain target local model parameters corresponding to the plurality of computing units, wherein the target input activation values ​​in the target local model parameters gradually increase along the direction of data processing; and The quantization module is used to quantize the target local model parameters to obtain a compressed model.

17. The device according to claim 16, wherein: The parameter division module comprises: The first obtaining submodule is used to obtain the initial local model parameters corresponding to each computing unit according to the number of the computing units and the number of the initial model parameters. a first target number of numbers; and The first division submodule is used to evenly divide the initial model parameters of the model to be compressed according to the first target quantity to obtain initial local model parameters corresponding to the calculation unit.

18. The device according to claim 16, wherein: The parameter division module comprises: a clustering submodule, configured to cluster the initial input activation values ​​in the initial model parameters based on the number of the computing units to obtain a clustering result; and The second division submodule is used to divide the initial model parameters of the model to be compressed based on the clustering result to obtain initial local model parameters corresponding to the calculation unit.

19. The device according to claim 18, wherein The clustering submodule includes: A first determining unit, configured to determine the number of initial input activation values ​​in each cluster based on the clustering result; a second determining unit, configured to determine a second target number of initial local model parameters corresponding to each computing unit based on the number of the initial input activation values ​​and the number of the computing units; and The first division unit is used to divide the initial model parameters of the model to be compressed according to the second target quantity to obtain initial local model parameters corresponding to the calculation unit.

20. The device according to claim 16, wherein: The parameter division module comprises: A first determination submodule, configured to determine the quantization accuracy of each computing unit according to the initial input activation value in the initial model parameters; and The third division submodule is used to divide the initial model parameters according to the quantization accuracy of each computing unit and the number of the computing units to obtain The initial local model parameters corresponding to the multiple calculation units.

21. The device according to claim 20, wherein: The third division submodule comprises: A third determining unit, configured to determine a third target number of initial local model parameters corresponding to each of the computing units according to the quantization accuracy of each of the computing units and the number of the computing units; and The second division unit is used to divide the initial model parameters of the model to be compressed according to the third target quantity to obtain initial local model parameters corresponding to the calculation unit.

22. The device according to claim 16, wherein: The parameter rearrangement module comprises: a sorting submodule, used to sort the initial weight parameters according to the initial input activation values ​​to obtain an arrangement order of the initial weight parameters; and The allocation submodule is used to allocate the initial weight parameters to the local processing channels corresponding to the computing units in sequence according to the arrangement order to obtain the target local model parameters.

23. The device according to claim 22, wherein: The allocation submodule comprises: a generating unit, configured to generate a target corresponding relationship between the initial weight parameter and the computing unit according to the arrangement order and the arrangement position of the computing unit; and An allocating unit is used to allocate the initial weight parameters to the local processing channels corresponding to the computing units in sequence according to the target corresponding relationship to obtain the target local model parameters.

24. The device according to claim 23, wherein: The generating unit comprises: A generating subunit is used to generate a subunit according to the arrangement order and the arrangement of the calculation unit. position, generating an index array, wherein the index array includes a correspondence between the arrangement order and the arrangement position; and The first determining subunit is used to determine the target corresponding relationship between the initial weight parameter and the calculation unit according to the index array.

25. The device according to claim 16, wherein: The quantization module comprises: A second determination submodule is used to determine a quantization factor for a target local model parameter corresponding to each computing unit; and The quantization submodule is used to quantize the target local model parameters corresponding to each computing unit according to the quantization factor to obtain the compression model.

26. The device according to claim 25, wherein The second determining submodule includes: a fourth determining unit, configured to determine, for each target local model parameter corresponding to each computing unit, a truncation threshold corresponding to the target weight parameter; and A fifth determining unit is configured to determine the quantization factor based on the truncation threshold.

27. The device according to claim 26, wherein: The fourth determining unit includes: an acquisition subunit, configured to acquire, for each target local model parameter corresponding to each computing unit, distribution information of a plurality of target input activation values ​​corresponding to the target weight parameter; and The second determining subunit is used to determine the truncation threshold according to the distribution information.

28. The device according to claim 27, wherein The second determining subunit is used for: filtering the plurality of target input activation values ​​according to the distribution information to obtain a concentrated distributed target input activation value; as well as The cutoff threshold is determined from the centrally distributed target input activation values.

29. A training device for a compression model, comprising: A first obtaining module, used for inputting sample text data into a compression model to obtain first matching probabilities for a plurality of predetermined categories; The sample text data is annotated with sample category information; A first determination module, configured to determine a loss value of the compression model according to the sample category information and the first matching probability; as well as An adjustment module is used to adjust the model parameters of the compression model according to the loss value to obtain a trained compression model; wherein the compression model is a compressed deep learning model obtained by the method described in any one of claims 1 to 13.

30. A text data processing device, comprising: A second acquisition module is used to input the text data to be processed into the compression model to obtain a matching probability representing the category to which the text data belongs; The matching probability includes a probability value that the text data belongs to each of a plurality of predetermined categories; as well as The second determination module is used to determine the target category to which the text data belongs based on the matching probability; wherein the compression model is a compressed deep learning model obtained by the method described in any one of claims 1 to 13.

31. An electronic device comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 15.

32. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the method according to any one of claims 1-15.

33. A computer program product comprising a computer program, which, when executed by a processor, implements the method according to any one of claims 1 to 15.

Images