Model training method and device and data processing method and equipment

Abstract

The invention relates to a model training method and device and a data processing method and equipment. The model training method comprises the steps that lexical element pruning is carried out on a multi-layer first encoder module of a network model, the lexical elements are lexical elements in an input sequence, and the lexical elements are determined according to input text data or image data; adjusting the number of layers of the first encoder module and the number of layers of the second encoder module according to the lexical element pruning data; and performing module pruning on the multi-layer second encoder module of the network model. According to the invention, the low-layer encoder module and the high-layer encoder module adopt different pruning granularities to prune, so that the pruning effect is improved.

Description

Technical Field

[0001] The present disclosure relates to the field of machine learning, and in particular to a model training method and device, and a data processing method and device. Background Art

[0002] At present, large models are being used more and more widely, and the application scenarios are becoming increasingly diverse. The number of calls to model inference services has increased dramatically, and in order not to affect the user experience, the delay of model inference has to be low. Large models are stacked based on the Transformer structure. However, the calculation of the multi-head attention matrix in the Transformer structure consumes more computing resources, increases the delay of model inference, and affects the use of services.

[0003] Reasoning compression for large models can be roughly divided into five directions: parameter sharing, low-rank decomposition, distillation, pruning, and quantization. In the related pruning technology, according to the pruning granularity of the Transformer structure, it can be divided into pruning based on the Encoder block, pruning based on the Attention Header (multi-head attention mechanism in Transformer), and pruning based on tokens (each word in the input sequence is a token). Summary of the invention

[0004] Through research, the inventors found that the relevant technology is based on the Token pruning scheme PoWER-Transformer. In the training and fine-tuning stage, the configuration information of the number of tokens that each layer of Encoder needs to retain is calculated by learning. During inference, the tokens of the input token of each layer are pruned according to the number of tokens configured in each Encoder, thereby reducing the complexity of the Attention matrix calculation, thereby improving the computational efficiency of the inference stage and reducing the overall computing delay.

[0005] The related technology ignores the correlation between multiple layers of encoders and does not perform pruning of relevant granularity according to the characteristics of high- and low-layer encoders, resulting in a low overall pruning granularity.

[0006] In view of at least one of the above technical problems, the present disclosure provides a model training method and device, a data processing method and equipment, and the low-level encoder module and the high-level encoder module use different pruning granularities for pruning, thereby improving the pruning effect.

[0007] According to one aspect of the present disclosure, a model training method is provided, comprising:

[0008] Performing word-unit pruning on the multi-layer first encoder module of the network model, wherein the word-unit is a word-unit in the input sequence, and the word-unit is determined according to the input text data or image data;

[0009] Adjusting the number of layers of the first encoder module and the number of layers of the second encoder module according to the word unit pruning data;

[0010] Perform module pruning on the multi-layer second encoder module of the network model.

[0011] In some embodiments of the present disclosure, adjusting the number of layers of the first encoder module and the number of layers of the second encoder module according to the word-unit pruning data includes:

[0012] Get word pruning data;

[0013] Determining the number of layers of the first encoder module according to the word-unit pruning data;

[0014] The number of layers of the second encoder module is determined according to the total number of layers of the encoder modules in the network model and the number of layers of the first encoder module.

[0015] In some embodiments of the present disclosure, the word-gram pruning data includes the number of adjacent word-grams and the adjacent word-gram dependencies.

[0016] In some embodiments of the present disclosure, obtaining word-unit pruning data includes:

[0017] Determine the number of adjacent word units of the first encoder module in each layer by learning the training sample data of the task;

[0018] During the downstream task fine-tuning process of the network model, the adjacent word dependencies of the first encoder module of each layer are calculated according to the number of adjacent word units.

[0019] In some embodiments of the present disclosure, determining the number of layers of the first encoder module according to the word-unit pruning data includes:

[0020] Determine the total loss function of the network model according to the word-unit pruning data of the first encoder module of each layer;

[0021] The number of layers of the first encoder module is determined according to the total loss function of the network model.

[0022] In some embodiments of the present disclosure, determining the total loss function of the network model according to the word-unit pruning data of the first encoder module of each layer includes:

[0023] Determine a loss function of the first encoder module of each layer according to the word-unit pruning data of the first encoder module of each layer;

[0024] According to the loss function of the first encoder module of each layer, the total loss function of the network model is determined.

[0025] In some embodiments of the present disclosure, determining the number of layers of the first encoder module according to the total loss function of the network model includes:

[0026] At least one of the number of adjacent word units of the last layer of the first encoder module and the number of layers of the first encoder module is adjusted according to the total loss function of the network model.

[0027] In some embodiments of the present disclosure, adjusting at least one of the number of adjacent word units of the last layer of the first encoder module and the number of layers of the first encoder module according to the total loss function of the network model includes:

[0028] In each iteration, if the total loss function of the network model increases, it is determined whether the number of adjacent word units of the first encoder module of the last layer is less than or equal to the first threshold;

[0029] When the number of adjacent word units of the last layer of the first encoder module is less than or equal to the first threshold, the last layer of the first encoder module is removed, and the number of layers of the first encoder module is reduced by 1;

[0030] When the number of adjacent word units of the first encoder module of the last layer is greater than the first threshold, the number of adjacent word units of the first encoder module of the last layer is reduced by 1.

[0031] In some embodiments of the present disclosure, adjusting at least one of the number of adjacent word units of the last layer of the first encoder module and the number of layers of the first encoder module according to the total loss function of the network model includes:

[0032] In each iteration process, if the total loss function of the network model decreases, it is determined whether the number of adjacent word units of the first encoder module of the last layer is greater than or equal to a second threshold, wherein the second threshold is greater than the first threshold;

[0033] When the number of adjacent word units of the last layer of the first encoder module is greater than or equal to the second threshold, the number of layers of the first encoder module is increased by 1, and one layer of the first encoder module is expanded as the new last layer of the first encoder module;

[0034] When the number of adjacent word units of the first encoder module of the last layer is less than the second threshold, the number of adjacent word units of the first encoder module of the last layer is increased by 1.

[0035] In some embodiments of the present disclosure, the value range of the number of adjacent word units is 2 to 6, the first threshold is 2, and the second threshold is 6.

[0036] In some embodiments of the present disclosure, the number of layers of the first encoder module ranges from 10 to 16.

[0037] In some embodiments of the present disclosure, the performing module pruning on the second encoder module includes:

[0038] According to the similarity of the outputs of adjacent second encoder modules, module pruning is performed on the second encoder modules.

[0039] In some embodiments of the present disclosure, performing module pruning on the second encoder module according to the similarity of the outputs of adjacent second encoder modules includes:

[0040] Starting from the second encoder module of the first layer, for each second encoder module, calculate the similarity between the word element vector matrix output by the second encoder module of this layer and the word element vector matrix output by the second encoder module of the next layer;

[0041] Determining whether the similarity is greater than a predetermined similarity;

[0042] When the similarity is greater than the predetermined similarity, the calculation of the subsequent second encoder module is exited, and the word element vector matrix output by the second encoder module of the next layer is used as the output result of the entire encoder structure.

[0043] According to another aspect of the present disclosure, there is provided a data processing method, comprising:

[0044] Determine a word sequence according to text data or image data as an input sequence;

[0045] A natural language processing model is used to perform natural language processing on the input sequence, wherein the natural language processing model is a natural language processing model trained by the model training method of any of the above embodiments.

[0046] According to another aspect of the present disclosure, a model training device is provided, comprising:

[0047] A first pruning module is configured to perform word-unit pruning on the multi-layer first encoder module of the network model, wherein the word-unit is a word-unit in an input sequence, and the word-unit is determined according to input text data or image data;

[0048] A layer number determination module, configured to adjust the number of layers of the first encoder module and the number of layers of the second encoder module according to the word unit pruning data;

[0049] The second pruning module is configured to perform module pruning on the multi-layer second encoder module of the network model.

[0050] According to another aspect of the present disclosure, a model training device is provided, comprising:

[0051] a memory configured to store instructions;

[0052] The processor is configured to execute the instructions so that the model training device implements the model training method as described in any of the above embodiments.

[0053] According to another aspect of the present disclosure, there is provided a data processing device, comprising:

[0054] A text acquisition device, configured to determine a word unit sequence according to text data or image data as an input sequence;

[0055] The model training device as described in any of the above embodiments is configured to train a natural language processing model;

[0056] The natural language processing device is configured to perform natural language processing on an input sequence using a trained natural language processing model.

[0057] According to another aspect of the present disclosure, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores computer instructions, and when the instructions are executed by a processor, the model training method as described in any of the above embodiments, or the data processing method as described in any of the above embodiments is implemented.

[0058] According to another aspect of the present disclosure, a computer program product is provided, including a computer program, wherein when the computer program is executed by a processor, the model training method as described in any of the above embodiments, or the data processing method as described in any of the above embodiments is implemented.

[0059] The low-level encoder module and the high-level encoder module disclosed in the present invention use different pruning granularities to perform pruning, thereby improving the pruning effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] In order to more clearly illustrate the embodiments of the present disclosure or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0061] Figure 1 Schematic diagram of some embodiments of the model training method disclosed in the present invention.

[0062] Figure 2 The embodiments provide schematic diagrams of overall pruning schemes in some embodiments of the present disclosure.

[0063] Figure 3 A schematic diagram of adjusting the number of layers of the first encoder module and the number of layers of the second encoder module according to word-unit pruning data in some embodiments of the present disclosure.

[0064] Figure 4 A schematic diagram of the calculation of adjacent Token dependencies in some embodiments of the present disclosure.

[0065] Figure 5 A schematic diagram of adding an underlying Encoder regularization term to the overall loss function in some embodiments of the present disclosure.

[0066] Figure 6 A schematic diagram of module pruning of the second encoder module in some embodiments of the present disclosure.

[0067] Figure 7 Schematic diagram of some embodiments of the data processing method disclosed herein.

[0068] Figure 8 Schematic diagrams of some embodiments of the model training device disclosed herein.

[0069] Fig. 9 Schematic diagram of the structure of other embodiments of the model training device disclosed in the present invention.

[0070] Fig.10 Schematic diagrams of some embodiments of the data processing device disclosed herein. DETAILED DESCRIPTION

[0071] The technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings in the embodiments of the present disclosure. Obviously, the described embodiments are only part of the embodiments of the present disclosure, rather than all of the embodiments. The following description of at least one exemplary embodiment is actually only illustrative and is by no means intended to limit the present disclosure and its application or use. Based on the embodiments in the present disclosure, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present disclosure.

[0072] Unless specifically stated otherwise, the relative arrangement of components and steps, the numerical expressions and numerical values ​​set forth in these embodiments do not limit the scope of the present disclosure.

[0073] At the same time, it should be understood that for the convenience of description, the sizes of the various parts shown in the drawings are not drawn according to the actual proportional relationship.

[0074] Technologies, methods, and apparatus known to ordinary technicians in the relevant field may not be discussed in detail, but where appropriate, such technologies, methods, and apparatus should be considered part of the authorization specification.

[0075] In all examples shown and discussed herein, any specific values ​​should be interpreted as merely exemplary and not as limiting. Therefore, other examples of the exemplary embodiments may have different values.

[0076] It should be noted that like reference numerals and letters refer to similar items in the following figures, and therefore, once an item is defined in one figure, it need not be further discussed in subsequent figures.

[0077] The inventors also found through research that related technologies are all based on a pruning method of a certain granularity, such as token granularity pruning, without combining the characteristics of the encoder layers, making full use of information and combining other pruning granularities for pruning, resulting in an unobvious pruning effect.

[0078] The related technology achieves good model effects based on Transformer-type large models. However, during the model reasoning process, the calculation efficiency of this type of LLM (Large Language Model) model is low and the calculation delay is high. The present invention is used to solve the problem of high reasoning calculation delay and high consumption of computing resources of this type of LLM model.

[0079] In view of at least one of the above technical problems, the present disclosure provides a model training method and device, a data processing method and equipment, and the present disclosure proposes a discriminative model training method and device based on scale factor weighting, which is described below through specific embodiments.

[0080] Figure 1 Schematic diagram of some embodiments of the model training method of the present disclosure. Preferably, this embodiment can be executed by the model training device of the present disclosure. The model pruning of the present disclosure is included in the model training, and the model training method includes the model pruning method. Figure 1 The model training method of the embodiment includes at least one step from step 1 to step 3.

[0081] Step 1: Token pruning is performed on the multi-layer first encoder module of the network model, wherein the token is a token in an input sequence, and the token is determined based on input text data, image data, video data, or other data.

[0082] Step 2: Adjust the number of layers of the first encoder module and the number of layers of the second encoder module according to the word-unit pruning data.

[0083] Step 3: prune the multi-layer second encoder module of the network model, wherein the network model includes a multi-layer first encoder module and a multi-layer second encoder module.

[0084] In some embodiments of the present disclosure, step 3 may include: performing module pruning on the second encoder module by adopting an Early Exit method.

[0085] In some embodiments of the present disclosure, step 3 may include: performing module pruning on the second encoder module according to the similarity output by adjacent second encoder modules.

[0086] In some embodiments of the present disclosure, the input sequence sequentially passes through the multiple layers of the first encoder module and the multiple layers of the second encoder module in order.

[0087] In some embodiments of the present disclosure, the first encoder module may be a low-layer Encoder module, and the second encoder module may be a high-layer Encoder module, where the Encoder module is the part for encoding in the Transformer structure.

[0088] In some embodiments of the present disclosure, the network model may be an LLM model. A large language model (LLM) is a language model composed of an artificial neural network with many parameters (usually billions of weights or more), and is trained on a large amount of unlabeled text using self-supervised learning or semi-supervised learning.

[0089] In some embodiments of the present disclosure, for the multi-layer (e.g., 96 layers) Encoder module adopted by the LLM model, due to the correlation between Encoder modules, and the low-layer Encoder is basically used to cross short-distance sequence dependencies of the sequence, and the high-layer Encoder is used to cross long-distance sequence dependencies of the sequence.

[0090] In some embodiments of the present disclosure, crossing short-distance sequence dependencies means that the low-layer Encoder crosses adjacent token information; crossing long-distance sequence dependencies means that the high-layer Encoder is used to cross token information that is farther away. For example: in "Hello, what did you eat tonight", the low-layer crosses the information of "day" and "up" near the token "night"; the high-layer Encoder crosses the information of "night" and the farther tokens "eat" or "what".

[0091] In order to achieve the highest network pruning rate in the above embodiments of the present disclosure, it is proposed to perform token pruning on the low-layer Encoder network, and adopt an Early Exit structure for the high-layer Encoder layer, combining structured pruning and unstructured pruning to fully prune useless layers and parameters.

[0092] Figure 2 The embodiment gives a schematic diagram of the overall pruning scheme in some embodiments of the present disclosure. Figure 2 It also gives a schematic diagram of the network structure of the network model in some embodiments of the present disclosure.

[0093] In some embodiments of the present disclosure, as Figure 2 shown, for the low-level Encoder structure used to obtain short-distance dependencies of the sequence, the solution of the present disclosure can dynamically obtain the number of layers (Layer_loc layers) of the first encoder module through pre-training the model and fine-tuning of specific downstream tasks, and is used to identify that the Encoder module with Layer <= Layer_loc performs short-distance association relationship acquisition of the sequence during inference.

[0094] In some embodiments of the present disclosure, as Figure 2 shown, if the total number of layers of the encoder module is n and the number of layers of the first encoder module is loc, where n is greater than loc, and both n and loc are natural numbers greater than 0.

[0095] In some embodiments of the present disclosure, as Figure 2 shown, the first encoder module includes the first encoder module of the first layer (Layer_1) to the first encoder module of the loc layer (Layer_loc), a total of loc layers.

[0096] In some embodiments of the present disclosure, as Figure 2 shown, the second encoder module includes the second encoder module of the (loc + 1)th layer (Layer_loc + 1) to the second encoder module of the nth layer (Layer_n), a total of n - loc layers.

[0097] In some embodiments of the present disclosure, as Figure 2 shown, the high-level Encoder module of Layer_loc + 1 < Layer <= Layer_n performs early exit by adopting the Early Exit mechanism to further reduce the number of Encoder layers that need to be calculated during the inference process to improve the inference calculation efficiency and reduce the overall inference latency, where model prediction and inference are online services that provide machine learning model scoring calculations, and latency is the total time consumed for one request calculation.

[0098] The solution of the present disclosure is divided into sparse pruning of the low-level Encoder module ( Figure 1 step 1) of the embodiment), and Early Exit early pruning of the high-level Encoder module ( Figure 1 step 2) of the embodiment). Next, these two processes will be described in detail respectively.

[0099] Figure 3 It is a schematic diagram for adjusting the number of layers of the first encoder module and the number of layers of the second encoder module according to the token pruning data in some embodiments of the present disclosure. Preferably, this embodiment can be executed by the model training device of the present disclosure. As Figure 3As shown, the model training method disclosed in the present invention (e.g. Figure 1 Step 1) of the embodiment may include at least one of steps 21 to 23.

[0100] Step 21, obtaining word-unit pruning data.

[0101] In some embodiments of the present disclosure, the word-gram pruning data includes the number of adjacent word-grams and the adjacent word-gram dependencies.

[0102] In some embodiments of the present disclosure, step 21 may include at least one of steps 211 to 212 .

[0103] Step 211, by learning the training sample data of the task, determine the number of adjacent word units of each layer of the first encoder module, wherein the number of adjacent word units of each layer of the first encoder module is the number of word units of adjacent word unit dependencies that need to be calculated by the first encoder module of this layer.

[0104] Step 212, during the downstream task fine-tuning process of the network model, the adjacent word-unit dependency of the first encoder module of each layer is calculated according to the number of adjacent word-units.

[0105] In some embodiments of the present disclosure, step 212 may include performing N-gram (N adjacent token association relationship calculation) on each layer during downstream task fine-tuning. N-gram (N-gram) refers to N words that appear consecutively in a text. The N-gram model is a probabilistic language model based on a (N-1)-order Markov chain, which infers the structure of a sentence by the probability of the occurrence of N words.

[0106] In some embodiments of the present disclosure, step 212 may include: in the low-level Encoder, for the Algo-unit (Algorithm) & self Attention layer, the overall calculation logic is: the parameters in the selfAttention of each layer are not hard pruned, and the specific parameter values ​​are still retained, for example: these pruned parameters are not set to empty or 0, and the original parameter values ​​are retained, such as 0.3526; during the training & fine-tuning process, the value of the Attention matrix (QKT) is first calculated, and the number of adjacent Tokens N that need to be calculated for this layer retained in the Algo-unit (the value is taken with the current Token as the center, and the left side is taken first and then the right side. For example, if N=3, one Token on the right and one Token on the left are taken; if N=4, one Token on the left is added, and 2 Tokens on the left and 1 Token on the right are taken) are taken out to calculate the adjacent weight data of the Attention matrix. The schematic diagram of the adjacent Tokens is as follows Figure 4 shown. Figure 4 A schematic diagram of the calculation of adjacent Token dependencies in some embodiments of the present disclosure.

[0107] like Figure 4 As shown in the figure, the calculation process of the dependency relationship between adjacent tokens in two layers, N=3 and N=2, is shown respectively. Figure 4 As shown in the lower part of , if N = 3, the dependency relationship of the current token and the tokens on the left and right is calculated for each token (if there is no token on the left of the initial token, no calculation is performed; if there is no token on the right of the last token, no calculation is performed). Figure 4 As shown in the upper part of , when N=2, the dependency relationship between the current token and the adjacent token on the right is calculated (if there is no token on the left of the initial token, no calculation is performed; if there is no token on the right of the last token, no calculation is performed). Figure 4 As shown in the left figure in the upper part, 0.25 has no adjacent data to cross-calculate, so Figure 4 As shown in the right figure in the upper part, x2_1 is only connected to z2_1; Figure 4 As shown in the left figure in the upper part, the second row 0.21 has data 0.15 on the left, and x2_2 needs to connect the two data. Figure 4 As shown, the Mask element also has an initialized random value, but it is pruned and no longer of interest, so it is replaced by a Mask tag.

[0108] In some embodiments of the present disclosure, the dependency relationship between the current token (x2_2) and its adjacent right token (x2_3) is obtained by parameters through learning (similar to the full connection method).

[0109] Step 22: Determine the number of layers of the first encoder module according to the word-unit pruning data.

[0110] In some embodiments of the present disclosure, step 22 may include at least one of steps 221 to 222 .

[0111] Step 221, determining the total loss function of the network model according to the word-unit pruning data of the first encoder module of each layer.

[0112] In some embodiments of the present disclosure, step 221 may include adding word-gram pruning data of the first encoder module of each layer to the total loss function of the network model, wherein the word-gram pruning data includes the number of adjacent word-grams and the dependency relationship between adjacent word-grams.

[0113] In some embodiments of the present disclosure, step 221 may include at least one step from step 2211 to step 2212 .

[0114] Step 2211, determining the loss function of the first encoder module of each layer according to the word-unit pruning data of the first encoder module of each layer.

[0115] In some embodiments of the present disclosure, step 2211 may include: calculating the loss function of the low-level Encoder based on the number of tokens N according to formula (1).

[0116]

[0117] In formula (1), d represents the dimension d*d of the Attention matrix, Att is the Attention matrix, i represents the i-th token, and j represents the tokens adjacent to the left and right of the token. The sum is calculated to calculate the dependency relationship of N adjacent tokens in this layer.

[0118] In some embodiments of the present disclosure, the dependency relationship is learned through weights, and the loss function is to calculate the sum of the weights of these interaction relationships to represent the importance of the token being associated with other tokens.

[0119] In some embodiments of the present disclosure, the loss function of the low-level Encoder based on the number of tokens N may adopt a regularization term calculation method or other loss function calculation methods (cross entropy loss function or logit loss function).

[0120] Step 2212, determining the total loss function of the network model according to the loss function of the first encoder module of each layer.

[0121] In some embodiments of the present disclosure, step 2212 may include: adding the loss function of the first encoder module of each layer to the total loss function of the network model.

[0122] In some embodiments of the present disclosure, step 2212 may include: adding the loss function of the first encoder module of each layer as a regularization term to the final loss function.

[0123] In some embodiments of the present disclosure, step 2212 may include: determining the final total loss function LOSS according to formula (2): 总 .

[0124] LOSS 总 = LOSS task + λLOS Stoken (2)

[0125] In formula (1), λ is used as a hyperparameter to adjust the downstream task effect and the overall pruning rate. task is the loss function for downstream tasks.

[0126] Figure 5 Schematic diagram of adding the underlying Encoder regularization term to the overall loss function in some embodiments of the present disclosure. Figure 5 As shown in , the loss function of the first encoder module of each layer is added to the total loss function Loss of the network model. Figure 5 As shown, each layer of encoder module can include Algo-unit & self Attention, FFN (Feed-forward networks) and two layers of LN (Layernorm layer normalization) & Add.

[0127] Step 222, determine the number of layers (loc or Layer_loc) of the first encoder module according to the total loss function of the network model.

[0128] In some embodiments of the present disclosure, step 221 to step 222 may include: another data Layer_loc determines the Layer_loc value by adding the low-layer pruning data to the loss function of the overall task to ensure the best overall model effect.

[0129] In some embodiments of the present disclosure, the number of layers of the first encoder module may range from 10 to 16. The initial value of the number of layers of the first encoder module may be 10 layers.

[0130] In some embodiments of the present disclosure, step 222 may include: adjusting at least one of the number of adjacent word units of the last layer of the first encoder module and the number of layers of the first encoder module according to the total loss function of the network model.

[0131] In some embodiments of the present disclosure, two parameters that need to be adjusted during training are the number of adjacent tokens N that need to be calculated for each layer, and the number of low-level Encoder layers Layer_loc.

[0132] In some embodiments of the present disclosure, step 222 may include at least one of steps 2221 to 2226.

[0133] Step 2221, during each iteration, if the total loss function of the network model increases, determine whether the number of adjacent word units of the first encoder module of the last layer is less than or equal to the first threshold.

[0134] Step 2222, when the number of adjacent word units of the last layer of the first encoder module is less than or equal to the first threshold, the last layer of the first encoder module is subtracted and the number of layers of the first encoder module is reduced by 1.

[0135] Step 2223, when the number of adjacent word units of the first encoder module of the last layer is greater than the first threshold, the number of adjacent word units of the first encoder module of the last layer is reduced by 1.

[0136] Step 2224, during each iteration, if the total loss function of the network model decreases, determine whether the number of adjacent word units of the first encoder module of the last layer is greater than or equal to a second threshold, wherein the second threshold is greater than the first threshold.

[0137] Step 2225, when the number of adjacent word units of the last layer of the first encoder module is greater than or equal to the second threshold, the number of layers of the first encoder module is increased by 1, and one layer of the first encoder module is expanded as the new last layer of the first encoder module.

[0138] Step 2226: When the number of adjacent word units of the first encoder module of the last layer is less than the second threshold, the number of adjacent word units of the first encoder module of the last layer is increased by 1.

[0139] In some embodiments of the present disclosure, the value range of the number of adjacent word units may be 2 to 6, the first threshold may be 2, and the second threshold may be 6.

[0140] In some embodiments of the present disclosure, step 222 may include: if the parameter N increases in the loss function during each iteration, the number of adjacent Tokens in the top layer N is adjusted to N-1, until the number of Tokens in the top layer N<2, then Layer_loc=Loyer-1 is subtracted from this layer; similarly, if the loss function of the model still decreases after the top layer increases to the number of adjacent Tokens N, then N is adjusted to N+1, until N>6 and then increases, Layer_loc=Loyer+1, and one layer is expanded.

[0141] Step 23, determining the number of layers of the second encoder module according to the total number of layers of the encoder modules in the network model and the number of layers of the first encoder module.

[0142] In some embodiments of the present disclosure, step 23 may include: determining the number of layers of the second encoder module according to the difference between the total number of layers of encoder modules in the network model and the number of layers of the first encoder module.

[0143] The above-mentioned embodiments of the present disclosure can dynamically adjust the boundary value between the low-level and high-level encoders.

[0144] Figure 6 Schematic diagram of module pruning of the second encoder module in some embodiments of the present disclosure. Preferably, this embodiment can be executed by the model training device of the present disclosure. Figure 6 As shown, the model training method disclosed in the present invention (e.g. Figure 1Step 3) of the embodiment may include at least one of steps 31 to 33.

[0145] Step 31, starting from the first layer of the second encoder module, for each layer of the second encoder module, calculate the similarity between the word element vector matrix output by the second encoder module of this layer and the word element vector matrix output by the second encoder module of the next layer.

[0146] In some embodiments of the present disclosure, the similarity may be Cosine Similarity (CSL) or other similarity calculation schemes.

[0147] Step 32: determine whether the similarity is greater than a predetermined similarity.

[0148] Step 33, when the similarity is greater than the predetermined similarity, the calculation of the subsequent second encoder module is exited, and the word element vector matrix output by the second encoder module of the next layer is used as the output result of the entire encoder structure.

[0149] In some embodiments of the present disclosure, the model training method of the present disclosure (e.g. Figure 1 Step 2) of the embodiment may include: for a high-level Encoder, such as Figure 2 In each layer after Layer_loc, the Token vector matrix S*d dimension (S is the number of tokens in the input sequence, d is the dimension of each token word vector, each word in the input sequence is a token, the vector dimension of each word is 1*d vector) output by each layer is calculated, and the similarity relationship between the output of the Encoder of the next layer (the Cosine distance of the tokens at the same position, and the values ​​of all tokens are added together); if the Encoder similarity of two adjacent layers exceeds a certain threshold, it will exit directly and no longer perform subsequent Encoder calculations. It will be used as the final output of the entire Encoder result and passed to the downstream task for calculation of the final task result.

[0150] In some embodiments of the present disclosure, the Cosine distance of the tokens at the same position, the step of adding the values ​​of all tokens may include: two S*d matrices, row by row vector multiplication to calculate a numerical result, and finally accumulating the results of each row as the final similarity result.

[0151] The above-mentioned embodiments of the present disclosure can target the association between the multi-layer encoders in the LLM, perform unstructured pruning inside the encoder and transformer structure, and reduce the number of parameters of the LLM model to the greatest extent, and finally speed up the reasoning efficiency in the reasoning stage. The transformer model (literally translated as "transformer") is a deep learning model that uses a self-attention mechanism, which can assign different weights according to the importance of each part of the input data. The model is mainly used in the fields of natural language processing (NLP Natural Language Process) and computer vision (CV, computer vision).

[0152] In order to maximize the network pruning rate, the above embodiments of the present disclosure propose to use token pruning for the low-level Encoder network and Early Exit structure for the high-level Encoder layer, and combine structured pruning and unstructured pruning to fully prune useless layers and parameters.

[0153] The low-level encoder module and the high-level encoder module in the above-mentioned embodiment of the present disclosure use different pruning granularities to perform pruning, thereby improving the pruning effect.

[0154] The above-mentioned embodiments of the present disclosure reduce the computational delay of LLM-like model reasoning and reduce the consumption of computing resources.

[0155] Figure 7 The following is a schematic diagram of some embodiments of the data processing method of the present disclosure. Preferably, the present embodiment can be executed by the data processing device of the present disclosure. The method includes at least one of the following steps 71-72.

[0156] Step 71 , determining a word sequence according to input text data, image data, video data or other data as an input sequence.

[0157] Step 72: Perform natural language processing on the input sequence using a natural language processing model, wherein the natural language processing model is a natural language processing model trained using the model training method of any of the above embodiments.

[0158] Figure 8 Schematic diagrams of some embodiments of the model training device disclosed in the present invention. Figure 8 As shown, the model training device disclosed herein may include a first pruning module 81 , a layer number determination module 82 , and a second pruning module 83 .

[0159] The first pruning module 81 is configured to perform word-gram pruning on the multi-layer first encoder module of the network model, wherein the word-gram is a word-gram in an input sequence, and the word-gram is determined based on input text data, image data, video data, or other data.

[0160] The layer number determination module 82 is configured to adjust the number of layers of the first encoder module and the number of layers of the second encoder module according to the word unit pruning data.

[0161] In some embodiments of the present disclosure, the layer number determination module 82 can be configured to obtain word unit pruning data; determine the number of layers of the first encoder module based on the word unit pruning data; and determine the number of layers of the second encoder module based on the total number of layers of the encoder modules in the network model and the number of layers of the first encoder module.

[0162] In some embodiments of the present disclosure, the word-gram pruning data may include the number of adjacent word-grams and the adjacent word-gram dependencies.

[0163] In some embodiments of the present disclosure, the layer number determination module 82 can be configured to determine the number of adjacent word units of the first encoder module of each layer by learning the training sample data of the task when acquiring word unit pruning data; in the process of fine-tuning the downstream tasks of the network model, the adjacent word unit dependency of the first encoder module of each layer is calculated according to the number of adjacent word units.

[0164] In some embodiments of the present disclosure, the layer number determination module 82 can be configured to determine the total loss function of the network model according to the word unit pruning data of each layer of the first encoder module, while determining the number of layers of the first encoder module according to the word unit pruning data; and determine the number of layers of the first encoder module according to the total loss function of the network model.

[0165] In some embodiments of the present disclosure, the layer number determination module 82 can be configured to determine the loss function of the first encoder module of each layer according to the word-unit pruning data of the first encoder module of each layer, while determining the total loss function of the network model according to the word-unit pruning data of the first encoder module of each layer; and determine the total loss function of the network model according to the loss function of the first encoder module of each layer.

[0166] The layer number determination module 82 can be configured to adjust at least one of the number of adjacent word units of the last layer of the first encoder module and the number of layers of the first encoder module according to the total loss function of the network model when determining the number of layers of the first encoder module according to the total loss function of the network model.

[0167] In some embodiments of the present disclosure, the layer number determination module 82 can be configured to adjust at least one of the number of adjacent word units of the last layer of the first encoder module and the number of layers of the first encoder module according to the total loss function of the network model. In each iteration, if the total loss function of the network model increases, it is determined whether the number of adjacent word units of the last layer of the first encoder module is less than or equal to a first threshold value; if the number of adjacent word units of the last layer of the first encoder module is less than or equal to the first threshold value, the last layer of the first encoder module is subtracted and the number of layers of the first encoder module is reduced by 1; if the number of adjacent word units of the last layer of the first encoder module is greater than the first threshold value, the number of adjacent word units of the last layer of the first encoder module is reduced by 1.

[0168] In some embodiments of the present disclosure, the layer number determination module 82 can be configured to adjust at least one of the number of adjacent word units of the last layer of the first encoder module and the number of layers of the first encoder module according to the total loss function of the network model. In each iteration, if the total loss function of the network model is reduced, it is determined whether the number of adjacent word units of the last layer of the first encoder module is greater than or equal to a second threshold, wherein the second threshold is greater than the first threshold; when the number of adjacent word units of the last layer of the first encoder module is greater than or equal to the second threshold, the number of layers of the first encoder module is increased by 1, and one layer of the first encoder module is expanded as the new last layer of the first encoder module; when the number of adjacent word units of the last layer of the first encoder module is less than the second threshold, the number of adjacent word units of the last layer of the first encoder module is increased by 1.

[0169] In some embodiments of the present disclosure, the value range of the number of adjacent word units is 2 to 6, the first threshold is 2, and the second threshold is 6.

[0170] In some embodiments of the present disclosure, the number of layers of the first encoder module ranges from 10 to 16.

[0171] The second pruning module 83 is configured to perform module pruning on a multi-layer second encoder module of a network model, wherein the network model includes a multi-layer first encoder module and a multi-layer second encoder module.

[0172] In some embodiments of the present disclosure, the second pruning module 83 may be configured to perform module pruning on the second encoder module by adopting an early exit method.

[0173] In some embodiments of the present disclosure, the second pruning module 83 may be configured to perform module pruning on the second encoder module according to the similarity of the outputs of adjacent second encoder modules when the early exit method is used to perform module pruning on the second encoder module.

[0174] In some embodiments of the present disclosure, the second pruning module 83 can be configured to, when performing module pruning on the second encoder modules based on the similarity output by adjacent second encoder modules, start from the first layer of second encoder modules, and for each layer of second encoder modules, calculate the similarity between the word element vector matrix output by the second encoder modules of that layer and the word element vector matrix output by the second encoder modules of the next layer; determine whether the similarity is greater than a predetermined similarity; and when the similarity is greater than the predetermined similarity, exit the calculation of subsequent second encoder modules, and use the word element vector matrix output by the second encoder modules of the next layer as the output result of the entire encoder structure.

[0175] In some embodiments of the present disclosure, the model training device is used to perform operations to implement the model training method described in any of the above embodiments.

[0176] Fig. 9 Schematic diagram of the structure of some other embodiments of the model training device disclosed in the present invention. Fig. 9 As shown, the model training device includes a memory 91 and a processor 92.

[0177] The memory 91 is used to store instructions. The processor 92 is coupled to the memory 91 . The processor 92 is configured to execute the method involved in the above embodiment based on the instructions stored in the memory.

[0178] like Fig. 9 As shown, the model training device also includes a communication interface 93 for information exchange with other devices. At the same time, the model training device also includes a bus 94, through which the processor 92, the communication interface 93, and the memory 91 communicate with each other.

[0179] The memory 91 may include a high-speed RAM memory, and may also include a non-volatile memory, such as at least one disk memory. The memory 91 may also be a memory array. The memory 91 may also be divided into blocks, and the blocks may be combined into virtual volumes according to certain rules.

[0180] In addition, the processor 92 may be a central processing unit (CPU), or may be an application specific integrated circuit (ASIC), or may be configured to implement one or more integrated circuits of the embodiments of the present disclosure.

[0181] Fig.10 Schematic diagram of some embodiments of the data processing device disclosed in the present invention. Fig.10 As shown, the data processing device disclosed herein may include a text acquisition device 101 , a model training device 102 and a natural language processing device 103 .

[0182] The text acquisition device 101 is configured to determine a word sequence according to input text data, image data, video data or other data as an input sequence.

[0183] The model training device 102 is configured to train the natural language processing model.

[0184] In some embodiments of the present disclosure, the model training device 102 may be a model training device as described in any of the above embodiments.

[0185] The natural language processing device 103 is used to perform natural language processing on the input sequence using the trained natural language processing model.

[0186] According to another aspect of the present disclosure, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores computer instructions, and when the instructions are executed by a processor, the model training method as described in any of the above embodiments, or the data processing method as described in any of the above embodiments is implemented.

[0187] According to another aspect of the present disclosure, a computer program product is provided, including a computer program, wherein when the computer program is executed by a processor, the model training method as described in any of the above embodiments, or the data processing method as described in any of the above embodiments is implemented.

[0188] According to another aspect of the present disclosure, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores computer instructions, and when the instructions are executed by a processor, the model training method as described in any of the above embodiments is implemented.

[0189] In some embodiments of the present disclosure, the computer-readable storage medium may be a non-transitory computer-readable storage medium.

[0190] The disclosure can comprehensively manage, optimize and improve the combination of performance such as power, economy, NVH, braking, emissions, thermal management, etc. The disclosure can obtain the dynamic comprehensive performance data of the whole vehicle, thereby better performing data analysis, achieving comprehensive consideration of various performances, and making the various performances of the product reach the optimal level.

[0191] Those skilled in the art will appreciate that the embodiments of the present disclosure may be provided as methods, devices, or computer program products. Therefore, the present disclosure may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present disclosure may take the form of a computer program product implemented on one or more computer-usable non-transient storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0192] The present disclosure is described with reference to flowcharts and / or block diagrams of methods, devices (systems) and computer program products according to embodiments of the present disclosure. It should be understood that each process and / or block in the flowchart and / or block diagram and the combination of processes and / or blocks in the flowchart and / or block diagram can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0193] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0194] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0195] The text acquisition device, model training device, natural language processing device, first pruning module, layer number determination module and second pruning module described above can be implemented as a general-purpose processor, a programmable logic controller (PLC), a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic device, discrete gate or transistor logic device, discrete hardware component or any appropriate combination thereof for performing the functions described in this application.

[0196] So far, the present disclosure has been described in detail. In order to avoid obscuring the concept of the present disclosure, some details known in the art are not described. Based on the above description, those skilled in the art can fully understand how to implement the technical solution disclosed here.

[0197] A person skilled in the art will understand that all or part of the steps to implement the above embodiments may be accomplished by hardware, or may be accomplished by instructing the relevant hardware through a program, and the program may be stored in a non-transitory computer-readable storage medium, and the above-mentioned storage medium may be a read-only memory, a disk or an optical disk, etc.

[0198] The description of the present disclosure is given for the purpose of illustration and description, and is not intended to be exhaustive or to limit the present disclosure to the disclosed form. Many modifications and variations will be apparent to those of ordinary skill in the art. The embodiments are selected and described in order to better illustrate the principles and practical applications of the present disclosure, and to enable those of ordinary skill in the art to understand the present disclosure and thereby design various embodiments with various modifications suitable for specific uses.

Claims

1. A model training method, comprising: Performing word-unit pruning on the multi-layer first encoder module of the network model, wherein the word-unit is a word-unit in the input sequence, and the word-unit is determined according to the input text data or image data; Adjusting the number of layers of the first encoder module and the number of layers of the second encoder module according to the word unit pruning data; Perform module pruning on the multi-layer second encoder module of the network model.

2. The model training method according to claim 1, wherein: The step of adjusting the number of layers of the first encoder module and the number of layers of the second encoder module according to the word-unit pruning data comprises: Get word pruning data; Determining the number of layers of the first encoder module according to the word-unit pruning data; The number of layers of the second encoder module is determined according to the total number of layers of the encoder modules in the network model and the number of layers of the first encoder module.

3. The model training method according to claim 2, wherein: The word-unit pruning data includes the number of adjacent word-units and the dependency relationship between adjacent word-units. The obtaining of word-unit pruning data comprises: Determine the number of adjacent word units of the first encoder module in each layer by learning the training sample data of the task; During the downstream task fine-tuning process of the network model, the adjacent word dependencies of the first encoder module of each layer are calculated according to the number of adjacent word units.

4. The model training method according to claim 3, wherein: Determining the number of layers of the first encoder module according to the word-unit pruning data includes: Determine the total loss function of the network model according to the word-unit pruning data of the first encoder module of each layer; The number of layers of the first encoder module is determined according to the total loss function of the network model.

5. The model training method according to claim 4, wherein: Determining the total loss function of the network model according to the word-unit pruning data of the first encoder module of each layer includes: Determine a loss function of the first encoder module of each layer according to the word-unit pruning data of the first encoder module of each layer; According to the loss function of the first encoder module of each layer, the total loss function of the network model is determined.

6. The model training method according to claim 4 or 5, wherein: Determining the number of layers of the first encoder module according to the total loss function of the network model includes: At least one of the number of adjacent word units of the last layer of the first encoder module and the number of layers of the first encoder module is adjusted according to the total loss function of the network model.

7. The model training method according to claim 6, wherein: The adjusting, according to the total loss function of the network model, at least one of the number of adjacent word units of the last layer of the first encoder module and the number of layers of the first encoder module comprises: In each iteration, if the total loss function of the network model increases, it is determined whether the number of adjacent word units of the first encoder module of the last layer is less than or equal to the first threshold; When the number of adjacent word units of the last layer of the first encoder module is less than or equal to the first threshold, the last layer of the first encoder module is removed, and the number of layers of the first encoder module is reduced by 1; When the number of adjacent word units of the first encoder module of the last layer is greater than the first threshold, the number of adjacent word units of the first encoder module of the last layer is reduced by 1.

8. The model training method according to claim 7, wherein: The adjusting, according to the total loss function of the network model, at least one of the number of adjacent word units of the last layer of the first encoder module and the number of layers of the first encoder module comprises: In each iteration process, if the total loss function of the network model decreases, determining whether the number of adjacent word units of the first encoder module of the last layer is greater than or equal to a second threshold, wherein the second threshold is greater than the first threshold; When the number of adjacent word units of the last layer of the first encoder module is greater than or equal to the second threshold, the number of layers of the first encoder module is increased by 1, and one layer of the first encoder module is expanded as the new last layer of the first encoder module; When the number of adjacent word units of the first encoder module of the last layer is less than the second threshold, the number of adjacent word units of the first encoder module of the last layer is increased by 1.

9. The model training method according to claim 8, wherein: The value range of the number of adjacent word units is 2 to 6, the first threshold is 2, and the second threshold is 6; and / or The number of layers of the first encoder module ranges from 10 to 16.

10. The model training method according to any one of claims 2 to 5, wherein: The performing module pruning on the second encoder module comprises: According to the similarity of the outputs of adjacent second encoder modules, module pruning is performed on the second encoder modules.

11. The model training method according to any one of claim 10, wherein: The performing module pruning on the second encoder module according to the similarity of the outputs of the adjacent second encoder modules includes: Starting from the second encoder module of the first layer, for each second encoder module, calculate the similarity between the word element vector matrix output by the second encoder module of this layer and the word element vector matrix output by the second encoder module of the next layer; Determining whether the similarity is greater than a predetermined similarity; When the similarity is greater than the predetermined similarity, the calculation of the subsequent second encoder module is exited, and the word element vector matrix output by the second encoder module of the next layer is used as the output result of the entire encoder structure.

12. A data processing method, comprising: Determine a word sequence according to text data or image data as an input sequence; A natural language processing model is used to perform natural language processing on the input sequence, wherein the natural language processing model is a natural language processing model trained by the model training method according to any one of claims 1 to 11.

13. A model training device, comprising: A first pruning module is configured to perform word-unit pruning on the multi-layer first encoder module of the network model, wherein the word-unit is a word-unit in an input sequence, and the word-unit is determined according to input text data or image data; A layer number determination module, configured to adjust the number of layers of the first encoder module and the number of layers of the second encoder module according to the word unit pruning data; The second pruning module is configured to perform module pruning on the multi-layer second encoder module of the network model.

14. A model training device, comprising: a memory configured to store instructions; The processor is configured to execute the instructions so that the model training device implements the model training method as described in any one of claims 1-11.

15. A data processing device comprising: A text acquisition device, configured to determine a word unit sequence according to text data or image data as an input sequence; The model training device as claimed in claim 13 or 14, configured to train a natural language processing model; The natural language processing device is configured to perform natural language processing on the input sequence using a trained natural language processing model.

16. A computer-readable storage medium, wherein: The computer-readable storage medium stores computer instructions, which, when executed by a processor, implement the model training method as described in any one of claims 1 to 11, or the data processing method as described in claim 12.

17. A computer program product comprising a computer program, wherein: When the computer program is executed by a processor, it implements the model training method as described in any one of claims 1 to 11, or the data processing method as described in claim 12.

Images