Task processing method and device, electronic equipment and storage medium

Abstract

The invention provides a task processing method and device, electronic equipment and a storage medium, and the method comprises the steps: in the training process of a Transform model, for any normalization layer in the Transform model, determining a first statistical value of the current batch of the normalization layer, and according to the first statistical value and the statistical value of the historical batch of the normalization layer, determining the first statistical value of the current batch of the normalization layer; smoothing the first statistical value by using a moving average strategy to obtain a second statistical value, and performing forward or reverse propagation on the second statistical value of the normalization layer; and in the process of performing task processing by using the trained Transform model, for any one normalization layer in the Transform model, fixing the statistical value of the normalization layer as a third statistical value, and performing reasoning calculation. According to the method, the processing performance can be ensured under the condition that the calculation memory consumption of the Transform model is reduced.

Description

Technical Field

[0001] The present application relates to the field of neural networks, and in particular to a task processing method, device, electronic device and storage medium. Background Art

[0002] In recent years, Transformer structure models have achieved surprising performance in natural language processing and computer vision tasks, becoming a research hotspot and focus in the current field of artificial intelligence.

[0003] However, the layer normalization operator included in the native Transformer will bring high computing memory consumption, and the layer normalization operator is not supported on many edge devices. If the layer normalization operator in the Transformer structure is replaced by the batch normalization operator with cheap computing memory, it will bring obvious performance degradation. Summary of the invention

[0004] In view of this, the present application provides a task processing method, device, electronic device and storage medium to optimize the task processing performance of the Transformer model.

[0005] Specifically, the present application is implemented through the following technical solutions:

[0006] According to a first aspect of an embodiment of the present application, a task processing method is provided, including:

[0007] In the process of training the Transformer model, for any normalization layer in the Transformer model, determine a first statistical value of a current batch of the normalization layer, smooth the first statistical value using a sliding average strategy based on the first statistical value and the statistical values ​​of historical batches of the normalization layer to obtain a second statistical value, and use the second statistical value of the normalization layer for forward or backward propagation;

[0008] In the process of using a trained Transformer model to process a task, for any normalization layer in the Transformer model, the statistical value of the normalization layer is fixed to a third statistical value for inference calculation; wherein, during the training process of the Transformer model, the third statistical value is determined based on the second statistical value using an exponential averaging strategy.

[0009] According to a second aspect of an embodiment of the present application, there is provided a task processing device, including:

[0010] A training unit is used to determine, for any normalization layer in the Transformer model, a first statistical value of a current batch of the normalization layer during training of the Transformer model, smooth the first statistical value using a sliding average strategy based on the first statistical value and the statistical values ​​of historical batches of the normalization layer to obtain a second statistical value, and use the second statistical value of the normalization layer for forward or backward propagation;

[0011] An inference unit is used to fix the statistical value of any normalization layer in the Transformer model to a third statistical value for inference calculation during task processing using a trained Transformer model; wherein the third statistical value is determined according to the second statistical value using an exponential averaging strategy during the training process of the Transformer model.

[0012] According to a third aspect of an embodiment of the present application, there is provided an electronic device, comprising a processor and a memory, wherein the memory stores machine executable instructions that can be executed by the processor, and the processor is used to execute the machine executable instructions to implement the method provided in the first aspect.

[0013] According to a fourth aspect of an embodiment of the present application, a machine-readable storage medium is provided, wherein the machine-readable storage medium stores machine-executable instructions, and when the machine-executable instructions are executed by a processor, the method provided in the first aspect is implemented.

[0014] The technical solution provided by this application can at least bring the following beneficial effects:

[0015] During the training of the Transformer model, for any normalization layer in the Transformer model, the first statistical value of the current batch of the normalization layer is determined, and based on the first statistical value and the statistical value of the historical batch of the normalization layer, the first statistical value is smoothed using a sliding average strategy to obtain a second statistical value, and the second statistical value of the normalization layer is used for forward or backward propagation, thereby alleviating the oscillation of the batch statistical value and improving the model performance. Furthermore, when using the trained Transformer model for task processing, the statistical value of each normalization layer is fixed to the third statistical value for inference calculation, thereby ensuring the processing performance while reducing the computational memory consumption of the Transformer model. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 is a flowchart of a task processing method shown in an exemplary embodiment of the present application;

[0017] Figure 2It is a schematic diagram of the architecture of a feature extraction network in a neural network with a Transformer structure shown in an exemplary embodiment of the present application;

[0018] Figure 3A It is a schematic diagram of calculation dimensions of an online normalization method shown in an exemplary embodiment of the present application;

[0019] Figure 3B It is a schematic diagram of calculation dimensions of an offline normalization method shown in an exemplary embodiment of the present application;

[0020] Figure 4 is a structural schematic diagram of a task processing device shown in an exemplary embodiment of the present application;

[0021] Figure 5 It is a schematic diagram of the hardware structure of an electronic device shown in an exemplary embodiment of the present application. DETAILED DESCRIPTION

[0022] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application. Instead, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the appended claims.

[0023] The terms used in this application are for the purpose of describing specific embodiments only and are not intended to limit this application. The singular forms "a", "said" and "the" used in this application and the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings.

[0024] In order to enable those skilled in the art to better understand the technical solutions provided by the embodiments of the present application and to make the above-mentioned purposes, features and advantages of the embodiments of the present application more obvious and understandable, the technical solutions in the embodiments of the present application are further described in detail below in conjunction with the accompanying drawings.

[0025] See also Figure 1 , is a flowchart of a task processing method provided in an embodiment of the present application, such as Figure 1 As shown, the task processing method may include the following steps:

[0026] It should be noted that the serial numbers of the steps in the embodiments of the present application do not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0027] Step S100: During the training of the Transformer model, for any normalization layer in the Transformer model, determine the first statistical value of the current batch of the normalization layer, smooth the first statistical value using a sliding average strategy based on the first statistical value and the statistical values ​​of the historical batches of the normalization layer to obtain a second statistical value, and use the second statistical value of the normalization layer for forward or backward propagation.

[0028] In an embodiment of the present application, in order to reduce the computational memory consumption of the Transformer model, the normalization operator of the Transformer model can be replaced with a batch normalization operator. In order to reduce the performance degradation caused by the batch normalization operator, in the process of training the Transformer model, when determining the statistical value of the normalization layer, the statistical value of a single batch is no longer used as the final statistical value. Instead, a sliding average strategy is used to smooth the statistical value of a single batch to alleviate the oscillation of the batch statistical value and improve model performance.

[0029] Accordingly, in the process of training the Transformer model, for any normalization layer in the Transformer model, the statistical value of the current batch of the normalization layer (also referred to as the statistical value at the current moment, recorded as the first statistical value in this article) can be determined, and based on the first statistical value and the statistical value of the historical batch of the normalization layer (also referred to as the statistical value at the historical moment), the first statistical value is smoothed using a sliding average strategy to obtain the smoothed statistical value (referred to as the second statistical value in this article), and the second statistical value of the normalization layer is used for forward or backward propagation.

[0030] Exemplarily, for any normalization layer in the Transformer model, on the one hand, the second statistical value determined by the normalization layer for the batch during the training process can be used to perform normalization calculation on the input data; on the other hand, a third statistical value can be determined by iterative optimization based on the second statistical value determined by the normalization layer during the training process, for example, by exponentially smoothing the second statistical value to obtain the third statistical value.

[0031] Step S110: In the process of performing task processing using the trained Transformer model, for any normalization layer in the Transformer model, the statistical value of the normalization layer is fixed to a third statistical value for inference calculation; wherein the third statistical value is determined according to the second statistical value using an exponential averaging strategy during the training process of the Transformer model.

[0032] In the embodiment of the present application, when the Transformer model training is completed in the manner described in step S100, the trained Transformer model can be used to perform task processing.

[0033] Exemplarily, for any normalization layer of the Transformer model, the third statistical value of the normalization layer can be determined during the training process of the Transformer model according to the second statistical value of the normalization layer using an exponential averaging strategy.

[0034] Exemplarily, the third statistical value of any normalization layer of the Transformer model is finally determined and fixed at the end of training. During the inference process, for any normalization layer of the trained Transformer model, the statistical value of the normalization layer can be fixed to the third statistical value for inference calculation. For example, the third statistical value of the normalization layer can be used to normalize the input data.

[0035] Exemplarily, the above-mentioned task processing includes but is not limited to NLP (Natural Language Processing) tasks (such as machine translation), or Speech (voice) tasks (such as speech recognition), or CV (Computer Vision) tasks (such as classification, target detection or instance segmentation tasks, etc.), etc.

[0036] It can be seen that in Figure 1 In the method flow shown, during the training of the Transformer model, for any normalization layer in the Transformer model, the first statistical value of the current batch of the normalization layer is determined, and based on the first statistical value and the statistical value of the historical batch of the normalization layer, the first statistical value is smoothed using a sliding average strategy to obtain a second statistical value, and the second statistical value of the normalization layer is used for forward or backward propagation, which alleviates the oscillation of the batch statistical value and improves the model performance. Furthermore, when using the trained Transformer model for task processing, the statistical value of each normalization layer is fixed to the third statistical value for inference calculation, thereby ensuring the processing performance while reducing the computational memory consumption of the Transformer model.

[0037] In some embodiments, the first statistical value of the current batch of the normalization layer is determined, and according to the first statistical value and the statistical value of the historical batch of the normalization layer, the first statistical value is smoothed by using a sliding average strategy to obtain a second statistical value, including:

[0038] During the forward propagation process, the square average operation is performed on the input activation values ​​of the current batch of the normalization layer to obtain the first activation value statistics of the current batch of the normalization layer;

[0039] According to the first activation value statistics of the current batch of the normalization layer and the activation value statistics of the historical batches of the normalization layer, the first activation value statistics of the current batch of the normalization layer are smoothed by using the geometric mean strategy to obtain the second activation value statistics.

[0040] Exemplarily, in order to improve the training stability when using the smoothing strategy, during the forward propagation process of training the Transformer model, for any normalization layer, when calculating the activation value statistics of the current batch of the normalization layer, the zero mean constraint can be removed, and the square average operation can be performed on the input activation values ​​of the current batch of the normalization layer to obtain the first statistical value of the current batch of the normalization layer (which can be called the first activation value statistical value).

[0041] Exemplarily, in order to suppress the oscillation of batch statistics, the first activation value statistics determined in the above manner may be smoothed using a sliding average strategy.

[0042] Exemplarily, during the forward propagation process, the geometric mean strategy can be used to calculate the geometric mean of the first activation value statistics of the current batch of the normalization layer and the activation value statistics of the historical batches of the normalization layer based on the first activation value statistics of the current batch of the normalization layer and the activation value statistics of the historical batches of the normalization layer to obtain the second activation value statistics.

[0043] Exemplarily, the number of activation value statistics of the historical batches participating in the geometric mean calculation in the above process can be determined by the size of a preset sliding window.

[0044] For example, assuming that the sliding window size is M (M ≥ 2), the activation value statistics of (M-1) historical batches need to be used to participate in the geometric mean calculation.

[0045] It should be noted that, in an embodiment of the present application, the above-mentioned third statistical value may be a third activation value statistical value, and the third activation value statistical value may be obtained by smoothing the second activation value statistical value of each batch during the training process of the Transformer model. For example, during the training process of the Transformer model, for any normalization layer, during the forward propagation process of any batch, when the second activation value statistical value of the current batch of the normalization layer is determined, the third activation value statistical value of the batch of the normalization layer may be determined based on the second activation value statistical value of each batch of the normalization layer using an exponential smoothing strategy; and further, when the training end condition is met, the third activation value statistical value determined for the last batch of the normalization layer may be used as the third activation value statistical value of the normalization layer used for final inference.

[0046] In some embodiments, the first statistical value of the current batch of the normalization layer is determined, and according to the first statistical value and the statistical value of the historical batch of the normalization layer, the first statistical value is smoothed by using a sliding average strategy to obtain a second statistical value, including:

[0047] During the back propagation process, the first gradient statistic of the current batch of the normalization layer is determined according to the second activation value statistic of the current batch of the normalization layer;

[0048] According to the first gradient statistic of the current batch of the normalization layer and the first gradient statistic of the historical batch of the normalization layer, the first gradient statistic of the current batch of the normalization layer is smoothed by using algorithm averaging and exponential averaging strategies to obtain a second gradient statistic.

[0049] Exemplarily, during the back propagation process of training the Transformer model, for any normalization layer, the second activation value statistic of the current batch of the normalization layer can be derived by the chain rule to obtain the first gradient statistic of the current batch of the normalization layer.

[0050] Exemplarily, since the calculation of the second activation value statistic brings in the historical activation value statistic information during forward propagation, the gradient brought in by the historical activation value statistic cannot be accurately derived according to the chain rule. Therefore, in order to estimate the gradient more accurately, the determined first gradient statistic can be smoothed using a sliding average strategy to obtain a second gradient statistic, and the gradient from the historical activation value statistic can be approximated by smoothing.

[0051] Exemplarily, during the back-propagation process, the arithmetic mean and exponential mean strategies can be used to calculate the arithmetic mean and exponential mean of the first gradient statistic of the current batch of the normalization layer and the gradient statistic of the historical batches of the normalization layer based on the first gradient statistic of the current batch of the normalization layer and the gradient statistic of the historical batches of the normalization layer to obtain the second gradient statistic.

[0052] Exemplarily, the number of gradient statistics of the historical batches involved in the calculation of the arithmetic mean and the exponential mean in the above process can be determined by the size of a preset sliding window.

[0053] For example, assuming that the sliding window size is M (M ≥ 2), the gradient statistics of (M-1) historical batches need to be used to participate in the calculation of the arithmetic mean and exponential mean.

[0054] In some embodiments, for any normalization layer in the Transformer model, after determining the first statistical value of the current batch of the normalization layer, and based on the first statistical value and the statistical values ​​of the historical batches of the normalization layer, using a sliding average strategy to smooth the first statistical value, before obtaining the second statistical value, the method may further include:

[0055] Determining whether the current batch of the normalization layer meets the smoothing processing condition according to the first statistical value of the current batch of the normalization layer;

[0056] When it is determined that the current batch of the normalization layer meets the smoothing conditions, determine to execute the above-mentioned operation of smoothing the first statistical value based on the first statistical value and the statistical values ​​of the historical batches of the normalization layer, and using the sliding average strategy to obtain the second statistical value.

[0057] For example, if there are large outliers in the batch statistics, using the sliding average strategy will make the Transformer model training unstable (the gradient approximation error becomes larger). Therefore, in order to improve the stability of Transformer model training and reduce the gradient approximation error introduced by the sliding average strategy in improving the performance of the Transformer model, an adaptive judgment strategy can be introduced to determine whether the sliding average strategy needs to be adopted to reduce the impact of outliers on the sliding average strategy.

[0058] Accordingly, in the process of training the Transformer model, for any normalization layer, after determining the first statistical value of the current batch of the normalization layer (such as the first activation value statistical value or the first gradient statistical value mentioned above), it is possible to determine whether the current batch of the normalization layer meets the smoothing processing condition based on the first statistical value of the current batch of the normalization layer, and when it is determined that the current batch of the normalization layer meets the smoothing processing condition, the first statistical value is smoothed using a sliding average strategy based on the first statistical value and the statistical values ​​of historical batches of the normalization layer to obtain the second statistical value.

[0059] It should be noted that, when it is determined that the current batch of the normalization layer does not meet the smoothing conditions, it is not necessary to use the sliding average strategy to smooth the first statistical value. Instead, the first statistical value of the current batch of the normalization layer can be determined as the second statistical value. That is, when it is determined that the current batch of the normalization layer does not meet the smoothing conditions, the statistical value of the normalization layer uses the first statistical value.

[0060] In an example, the current batch of the normalization layer satisfies the smoothing processing condition, which may include:

[0061] The absolute value of the difference between the arithmetic mean of the first statistical value of the current batch of the normalization layer and the first statistical value of the historical batch and the geometric mean of the first statistical value of the current batch of the normalization layer and the first statistical value of the historical batch is less than or equal to the first threshold;

[0062] The first threshold is determined based on the variance of the historical statistical values ​​in the previous batch of record windows.

[0063] For example, considering that when the first statistical value of the current batch of the normalization layer is smoothed by using the sliding average strategy, if the absolute value of the difference between the arithmetic mean of the first statistical values ​​of each batch in the sliding window and the geometric mean of the first statistical values ​​of each batch in the sliding window is too large, it indicates that there are outliers in the first statistical values ​​of the current batch. In this case, if the sliding average strategy is used for smoothing, the Transformer model training will be unstable.

[0064] For example, taking the sliding window size as M, the arithmetic mean of the first statistical value of the current batch and the first statistical value of the previous M-1 batches can be calculated, and the geometric mean of the first statistical value of the current batch and the first statistical value of the previous M-1 batches can be calculated. Based on the absolute value of the difference between the arithmetic mean and the geometric mean, it is determined whether the current batch of the normalization layer meets the smoothing processing conditions.

[0065] Exemplarily, when determining whether the current batch of the normalization layer meets the smoothing processing conditions, a determination threshold (which may be referred to as a first threshold) may be determined based on the variance of the first statistical value of the historical batches of the normalization layer, and is used to determine whether the absolute value of the difference between the arithmetic mean and the geometric mean is too large. When the absolute value of the difference between the arithmetic mean and the geometric mean is less than or equal to the first threshold, it is determined that the current batch of the normalization layer meets the smoothing processing conditions; otherwise, it is determined that the current batch of the normalization layer does not meet the smoothing processing conditions.

[0066] For example, assuming that the current moment is moment t and the sliding window size is M, the first threshold can be determined based on the first statistical value of the normalization layer at moment t-1 and the variance of the first statistical values ​​of the previous M-1 batches.

[0067] In another example, the current batch of the normalization layer satisfies the smoothing processing condition, which may include:

[0068] The absolute value of the first statistical value of the current batch of the normalization layer is less than or equal to the second threshold;

[0069] The second threshold is determined based on the standard deviation of the statistical value distribution of the historical batches of the normalization layer.

[0070] For example, considering that the first statistical value of the current batch of the normalization layer is smoothed by using the sliding average strategy, if the standard deviation of the first statistical value of the current batch of the normalization layer is too different from that of the first statistical value of the historical batch of the normalization layer, it indicates that there are outliers in the first statistical value of the current batch of the normalization layer. At this time, if the sliding average strategy is used for smoothing in this case, the Transformer model training will be unstable.

[0071] Exemplarily, since the statistical value is calculated according to the channel dimension when the batch normalization operator is used for normalization calculation, the calculated statistical value (such as the first statistical value mentioned above) can be in the form of a C-element one-dimensional vector, that is, there are a total of C elements (one element corresponds to one channel, and C is the number of channels).

[0072] Correspondingly, the absolute value of the first statistical value being less than or equal to the second threshold value may mean that the absolute value of any element of the first statistical value is less than or equal to the corresponding second threshold value.

[0073] Exemplarily, the second threshold may be determined based on a standard deviation of the statistical value distribution of historical batches of the normalization layer.

[0074] For example, assuming that the sliding window size is M, the second threshold corresponding to the first statistical value of the current batch of the normalization layer can be determined based on the standard deviation of the distribution of the first statistical values ​​of the M-1 batches before the current batch.

[0075] It should be noted that, in the embodiment of the present application, the above determination of whether the current batch of the normalization layer meets the smoothing processing conditions emphasizes whether it is necessary to determine the second statistical value based on the first statistical value using the sliding average strategy during the forward / backward propagation process.

[0076] Exemplarily, when the current batch of the normalization layer meets the smoothing processing conditions, it is necessary to determine the second statistical value based on the first statistical value using a sliding average strategy; when the current batch of the normalization layer does not meet the smoothing processing conditions, the second statistical value may not be determined using a sliding average strategy. For example, the first statistical value of the current batch of the normalization layer may be determined as the second statistical value of the current batch of the normalization layer.

[0077] Exemplarily, in the forward propagation process of any batch, it is necessary to update the third activation value statistics determined in the previous batch based on the second activation value statistics of the current batch using the exponential smoothing strategy to obtain the third activation value statistics of the current batch. That is, regardless of whether the second statistics need to be determined using the sliding average strategy, the third statistics need to be determined using the exponential average strategy. The above-mentioned smoothing processing conditions are constraints on the sliding average strategy, but do not constrain the exponential average strategy.

[0078] In order to enable those skilled in the art to better understand the technical solutions provided in the embodiments of the present application, the technical solutions provided in the embodiments of the present application are described below with reference to specific examples.

[0079] The following is a brief description of the Transformer neural network.

[0080] The Transformer structured neural network may include an encoder part (that is, a feature extraction network in the Transformer structured neural network) and a decoder part.

[0081] See also Figure 2 , is a schematic diagram of an architecture of a feature extraction network in a neural network with a Transformer structure provided in an embodiment of the present application. Figure 2As shown in the figure, the feature extraction network in the Transformer structure neural network includes an embedding layer and at least one Transformer layer, and one Transformer layer includes two basic modules, one of which includes a normalization layer, a multi-head attention layer, and an add layer; the other basic module includes a normalization layer, a feed forward neural network layer, and an add layer. That is, after the original sequence information to be processed is processed by the feature extraction network in the Transformer structure neural network, the feature information of the entire original sequence information to be processed can be obtained.

[0082] The characteristic information is characteristic information of the sequence information to be processed that is suitable for computer processing.

[0083] Exemplarily, the input to be processed by Transformer is sequence information, which may include but is not limited to text, speech or image blocks. Therefore, the Transformer structure can be used for natural language processing tasks such as text similarity, text classification, machine translation, etc.; it can also be applied to speech tasks such as speech recognition; it can also be used for visual tasks such as image classification, object detection and semantic segmentation.

[0084] Next, we will introduce the above embedding layer and multi-head attention layer in detail with specific examples.

[0085] Taking text as an input sequence as an example, after the embedding layer obtains the text to be processed, it can embed each word in the text to be processed to obtain the initial feature information of each word. The text to be processed can be a paragraph of text or a sentence. The text can be Chinese text, English text, or text in other languages.

[0086] Specifically, in some embodiments, Figure 2 As shown, the embedding layer includes an input embedding layer and a positional encoding layer.

[0087] Exemplarily, in the input embedding layer, each word in the processed text can be embedded to obtain a word embedding tensor for each word. The tensor can be specifically expressed as a one-dimensional vector, a two-dimensional matrix, three-dimensional or more dimensional data, and so on.

[0088] Exemplarily, in the position encoding layer, the position of each word in the text to be processed can be obtained, and then a position tensor is generated for the position of each word.

[0089] In some examples, the position of each word may be the absolute position of each word in the text to be processed.

[0090] Taking the text to be processed as "The weather is great today" as an example, the position of "今" can be represented as the first position, the position of "天" can be represented as the second position, and so on.

[0091] In some examples, the positions of each word can be the relative positions between each word.

[0092] Still taking the text to be processed as "The weather is great today" as an example, the position of "今" can be represented as before "天", the position of "天" can be represented as after "今" and before "气", and so on.

[0093] After obtaining the word embedding tensor and position tensor of each word in the text to be processed, the position tensor and word embedding tensor of each word can be combined to obtain the initial feature information of each word, thereby obtaining the initial feature information corresponding to the text to be processed.

[0094] Exemplarily, the multi-head attention mechanism can simply and effectively abstract the context dependence and capture syntactic and semantic features.

[0095] In the multi-head attention mechanism, the input features are linearly mapped to different information subspaces through different full-connected layers, and the same attention calculation is completed in each subspace to fully learn the potential structure and semantics of the text. The outputs of each information subspace will be combined in a concatenation manner and then fused through a full-connected layer.

[0096] Such as Figure 3A and Figure 3B As shown, the normalization methods can be divided into two categories: one is the "online normalization method", represented by the layer normalization operator (Layer Normalization). Such methods still need to dynamically calculate statistical values during inference and cannot be merged with adjacent linear operators for calculation. Therefore, the inference process cannot be accelerated; the other is the "offline normalization method", represented by the batch normalization operator (Batch Normalization). Such methods calculate statistical values in each corresponding channel dimension (Channel) and use fixed statistical values for forward calculation during inference. Therefore, they can be merged with adjacent linear layers to accelerate the model inference.

[0097] Considering that the layer normalization operator is inefficient in hardware deployment and consumes a large amount of computing memory, in order to improve the deployment efficiency of the Transformer model and reduce the computing memory consumption, the normalization operator in the Transformer model can be replaced with the batch normalization operator.

[0098] In order to reduce the performance degradation caused by the batch normalization operator, during the training of the Transformer model, the sliding average strategy is used to correct the inference statistics to improve the performance. At the same time, the "outlier filtering" strategy is applied to adaptively adjust the application of the sliding average strategy of each layer.

[0099] When using a trained Transformer model for inference, you can fix the statistical values ​​required for normalization and merge the Norm operator with the adjacent linear operator to speed up inference.

[0100] For example, considering that replacing the mid-layer normalization operator of Transformer with the batch normalization operator will lead to performance degradation, there are the following problems:

[0101] Problem 1: The distribution range of batch statistics is large (there is oscillation), which leads to inaccurate inference statistics and thus performance degradation.

[0102] Problem 2: There are large outliers in the distribution of batch statistics, which will cause instability in training.

[0103] For question 1, in the embodiment of the present application, the constraint of zero mean is removed, and the square mean is used as the statistical value of scaled normalization. It is proposed to use a sliding average strategy in both the forward and reverse normalization to suppress the oscillation of batch statistical values, thereby calibrating the inference statistical values ​​and improving model performance.

[0104] In response to question 2, in the embodiment of the present application, an adaptive judgment strategy is introduced to reduce the impact of outliers on the sliding average strategy, thereby stabilizing the training process.

[0105] The specific implementation method is as follows:

[0106] a) Using the sliding average strategy, the forward process uses the geometric mean strategy to smooth the impact of the activation value oscillation on the inference statistic during training, and the reverse process uses the arithmetic mean and exponential mean strategies to smooth the gradient statistic.

[0107] For example, assuming that the sliding window size is M, during the forward propagation process, the first activation value statistic of the current batch is the first activation value statistic at time t The following formula can be used to smooth the first activation value statistics to obtain the second activation value statistics:

[0108]

[0109] in, is the first activation value statistic at time (ti), that is, the first activation value statistic of the i-th batch between the current batch.

[0110] Assume that during the back propagation process, the first gradient statistic of the current batch is the first gradient statistic at time t The following formula can be used to smooth the first gradient statistic to obtain the second activation value statistic:

[0111]

[0112]

[0113] in, is the second gradient statistic at time t, is the second gradient statistic at time t-1, is the first gradient statistic at time ti.

[0114] b) Introduce the "outlier filtering" strategy and adaptively adjust the application of the sliding average strategy to reduce the impact of outliers on training stability.

[0115] Embodiment 1

[0116] The stored historical statistical values ​​are used to determine the arithmetic mean AM and the geometric mean GM, and then the inequality upper bound of |AM-GM| is used to select the adaptive threshold (at time t):

[0117]

[0118] in, is the arithmetic mean of the first statistical value in the sliding window M at time t, is the geometric mean of the first statistical value in the sliding window M at time t, Var[σ t-1 ] is the square root of the first statistical value in the sliding window M at time t-1, and M is the size of the sliding window.

[0119] Embodiment 2

[0120] Using stored historical statistics n times the standard deviation (std) of the distribution is used as the adaptive threshold:

[0121]

[0122] in, is the first statistical value at time t, is the standard deviation of the first statistical value from time (t-M+1) to time t-1, and n is an empirical value that can be set according to actual needs, such as 3, 5, or 11.

[0123] If the above inequality 1) or 2) does not hold, it is determined that the first statistical value of the current batch (time t) has an outlier, and the sliding average strategy in the above forward propagation process / backward propagation process will be skipped. In this case, the first statistical value can be used (there is no need to further calculate the second statistical value, or determine that the second statistical value is equal to the first statistical value).

[0124] It should be noted that for any batch of forward propagation, if the forward propagation of the batch (assuming it is time t) meets the smoothing condition, the first activation value / gradient statistic can be used to update window M; if the forward propagation of the batch does not meet the smoothing condition, for example, the activation value statistics of the batch are judged to contain outliers, that is, they are not representative statistics, then in order to avoid the influence of the activation value statistics of the batch on the sliding average strategy, the third activation value statistic determined by the previous batch (that is, the third activation value statistic at time t-1) can be used as the first activation value statistic at time t to update window M; and if the window M of the first gradient statistic is updated, the second gradient statistic at time t-1 is used as the first gradient statistic at time t to update window M.

[0125] It can be seen that by using the sliding average strategy to calibrate the inference statistics, the performance of the model can be avoided from being significantly reduced. However, using the sliding window strategy to approximate the reverse gradient will cause training instability due to the presence of outliers. Therefore, an adaptive judgment strategy is also needed to determine whether to use the sliding window strategy at the current moment to avoid training crashes.

[0126] The method provided by the present application is described above. The device provided by the present application is described below:

[0127] See also Figure 4 , is a structural diagram of a task processing device provided in an embodiment of the present application, such as Figure 4 As shown, the task processing device may include:

[0128] The training unit 410 is used to determine, for any normalization layer in the Transformer model, a first statistical value of a current batch of the normalization layer during training of the Transformer model, smooth the first statistical value using a sliding average strategy based on the first statistical value and the statistical values ​​of historical batches of the normalization layer to obtain a second statistical value, and use the second statistical value of the normalization layer for forward or backward propagation;

[0129] The inference unit 420 is used to fix the statistical value of any normalization layer in the Transformer model to a third statistical value for inference calculation during the process of using the trained Transformer model to process the task; wherein the third statistical value is determined according to the second statistical value using an exponential averaging strategy during the training process of the Transformer model.

[0130] In some embodiments, the training unit 410 determines a first statistical value of a current batch of the normalization layer, and smoothes the first statistical value using a sliding average strategy based on the first statistical value and the statistical value of a historical batch of the normalization layer to obtain a second statistical value, including:

[0131] During the forward propagation process, the square average operation is performed on the input activation values ​​of the current batch of the normalization layer to obtain the first activation value statistics of the current batch of the normalization layer;

[0132] According to the first activation value statistics of the current batch of the normalization layer and the activation value statistics of the historical batches of the normalization layer, the first activation value statistics of the current batch of the normalization layer are smoothed by using the geometric mean strategy to obtain the second activation value statistics.

[0133] In some embodiments, the training unit 410 determines a first statistical value of a current batch of the normalization layer, and smoothes the first statistical value using a sliding average strategy based on the first statistical value and the statistical value of a historical batch of the normalization layer to obtain a second statistical value, including:

[0134] During the back propagation process, the first gradient statistic of the current batch of the normalization layer is determined according to the second activation value statistic of the current batch of the normalization layer;

[0135] According to the first gradient statistic of the current batch of the normalization layer and the first gradient statistic of the historical batch of the normalization layer, the first gradient statistic of the current batch of the normalization layer is smoothed by using arithmetic mean and exponential mean strategies to obtain a second gradient statistic.

[0136] In some embodiments, the training unit 410, for any normalization layer in the Transformer model, after determining the first statistical value of the current batch of the normalization layer, and before smoothing the first statistical value using a sliding average strategy based on the first statistical value and the statistical values ​​of the historical batches of the normalization layer to obtain the second statistical value, further includes:

[0137] Determining whether the current batch of the normalization layer meets the smoothing processing condition according to the first statistical value of the current batch of the normalization layer;

[0138] When it is determined that the current batch of the normalization layer meets the smoothing conditions, determine to execute the operation of smoothing the first statistical value based on the first statistical value and the statistical values ​​of the historical batches of the normalization layer using a sliding average strategy to obtain a second statistical value.

[0139] In some embodiments, the current batch of the normalization layer satisfies the smoothing processing conditions, including:

[0140] The absolute value of the difference between the arithmetic mean of the first statistical value of the current batch of the normalization layer and the first statistical value of the historical batch and the geometric mean of the first statistical value of the current batch of the normalization layer and the first statistical value of the historical batch is less than or equal to the first threshold;

[0141] The first threshold is determined according to the variance of the first statistical value of the historical batch of the normalization layer.

[0142] In some embodiments, the current batch of the normalization layer satisfies the smoothing processing conditions, including:

[0143] The absolute value of the first statistical value of the current batch of the normalization layer is less than or equal to the second threshold;

[0144] The second threshold is determined according to the standard deviation of the first statistical value distribution of the historical batch of the normalization layer.

[0145] An embodiment of the present application provides an electronic device, including a processor and a memory, wherein the memory stores machine executable instructions that can be executed by the processor, and the processor is used to execute the machine executable instructions to implement the task processing method described above.

[0146] See also Figure 5 , is a schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present application. The electronic device may include a processor 501 and a memory 502 storing machine executable instructions. The processor 501 and the memory 502 may communicate via a system bus 503. Furthermore, by reading and executing the machine executable instructions corresponding to the task processing logic in the memory 502, the processor 501 may execute the task processing method described above.

[0147] The memory 502 mentioned herein may be any electronic, magnetic, optical or other physical storage device that may contain or store information, such as executable instructions, data, etc. For example, the machine-readable storage medium may be: RAM (RadomAccess Memory), volatile memory, non-volatile memory, flash memory, storage drive (such as hard disk drive), solid state drive, any type of storage disk (such as optical disk, DVD, etc.), or similar storage medium, or a combination thereof.

[0148] In some embodiments, a machine-readable storage medium is also provided. Figure 5 The memory 502 in the machine readable storage medium stores machine executable instructions, and when the machine executable instructions are executed by the processor, the task processing method described above is implemented. For example, the storage medium can be ROM, RAM, CD-ROM, magnetic tape, floppy disk, optical data storage device, etc.

[0149] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0150] The above description is only a preferred embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application shall be included in the scope of protection of the present application.

Claims

1. A task processing method, characterized in that: include: In the process of training the Transformer model, for any normalization layer in the Transformer model, determine the first statistical value of the current batch of the normalization layer, and use a sliding average strategy to smooth the first statistical value based on the first statistical value and the statistical value of the historical batch of the normalization layer to obtain a second statistical value, and use the second statistical value of the normalization layer to perform forward or backward propagation; wherein the sliding average strategy includes an arithmetic mean strategy and / or a geometric mean strategy; In the process of using a trained Transformer model to process a task, for any normalization layer in the Transformer model, the statistical value of the normalization layer is fixed to a third statistical value for inference calculation; wherein, during the training process of the Transformer model, the third statistical value is determined based on the second statistical value using an exponential averaging strategy.

2. The method according to claim 1, characterized in that: The determining of the first statistical value of the current batch of the normalization layer, and smoothing the first statistical value by using a sliding average strategy according to the first statistical value and the statistical value of the historical batch of the normalization layer to obtain the second statistical value, includes: In the forward propagation process, an arithmetic mean operation is performed on the input activation values ​​of the current batch of the normalization layer to obtain the first activation value statistics of the current batch of the normalization layer; wherein, in the process of determining the first activation value statistics of the current batch of the normalization layer, the zero mean constraint is removed; According to the first activation value statistics of the current batch of the normalization layer and the activation value statistics of the historical batches of the normalization layer, the first activation value statistics of the current batch of the normalization layer are smoothed by using the geometric mean strategy to obtain the second activation value statistics.

3. The method according to claim 2, characterized in that The number of activation value statistics of the historical batches involved in the geometric mean calculation is determined by the size of the preset sliding window.

4. The method according to claim 1, characterized in that: The determining of the first statistical value of the current batch of the normalization layer, and smoothing the first statistical value by using a sliding average strategy according to the first statistical value and the statistical value of the historical batch of the normalization layer to obtain the second statistical value, includes: During the back propagation process, the first gradient statistic of the current batch of the normalization layer is determined according to the second activation value statistic of the current batch of the normalization layer; According to the first gradient statistic of the current batch of the normalization layer and the first gradient statistic of the historical batch of the normalization layer, the first gradient statistic of the current batch of the normalization layer is smoothed by using arithmetic mean and exponential mean strategies to obtain a second gradient statistic.

5. The method according to claim 4, characterized in that The first gradient statistic of the current batch of the normalized layer and the first gradient statistic of the historical batch of the normalized layer are smoothed by using arithmetic mean and exponential mean strategies to obtain the second gradient statistic, including: According to the first gradient statistic value of the current batch of the normalization layer and the first gradient statistic value of the historical batch of the normalization layer, the arithmetic mean and the exponential mean of the first gradient statistic value of the current batch of the normalization layer and the first gradient statistic value of the historical batch of the normalization layer are calculated by using the arithmetic mean and exponential mean strategies to obtain a second gradient statistic value; The number of first gradient statistical values ​​of the historical batches participating in the calculation of the arithmetic mean and the exponential mean is determined by the size of the preset sliding window.

6. The method according to claim 5, characterized in that The gradient statistics of (M-1) historical batches are used to calculate the arithmetic mean and exponential mean; where M is the sliding window size, M≥2.

7. The method according to claim 1, characterized in that According to the first activation value statistics of the current batch of the normalized layer and the activation value statistics of the historical batches of the normalized layer, the first activation value statistics of the current batch of the normalized layer are smoothed by using the geometric mean strategy to obtain the second activation value statistics, which is implemented using the following formula: in, is the first activation value statistic at time t, is the first activation value statistic at time (ti), and M is the sliding window size.

8. The method according to claim 4, characterized in that According to the first gradient statistic of the current batch of the normalized layer and the first gradient statistic of the historical batch of the normalized layer, the first gradient statistic of the current batch of the normalized layer is smoothed by using the arithmetic mean and exponential mean strategies to obtain the second gradient statistic, which is implemented using the following formula: in, is the second gradient statistic at time t, is the second gradient statistic at time t-1, is the first gradient statistic at time ti.

9. A task processing device, characterized in that: include: A training unit is used to determine, for any normalization layer in the Transformer model, a first statistical value of a current batch of the normalization layer during training of the Transformer model, smooth the first statistical value using a sliding average strategy based on the first statistical value and the statistical values ​​of historical batches of the normalization layer to obtain a second statistical value, and use the second statistical value of the normalization layer for forward or backward propagation; An inference unit, used for fixing the statistical value of any normalization layer in the Transformer model to a third statistical value for inference calculation during task processing using the trained Transformer model; wherein the third statistical value is determined according to the second statistical value using an exponential averaging strategy during the training of the Transformer model; The training unit determines a first statistical value of a current batch of the normalization layer, and smoothes the first statistical value using a sliding average strategy based on the first statistical value and the statistical value of a historical batch of the normalization layer to obtain a second statistical value, including: In the forward propagation process, a square average operation is performed on the input activation values ​​of the current batch of the normalization layer to obtain the first activation value statistics of the current batch of the normalization layer; wherein, in the process of determining the first activation value statistics of the current batch of the normalization layer, the zero mean constraint is removed; According to the first activation value statistics of the current batch of the normalization layer and the activation value statistics of the historical batches of the normalization layer, the first activation value statistics of the current batch of the normalization layer are smoothed by using the geometric mean strategy to obtain the second activation value statistics.

10. An electronic device, characterized in that: The method comprises a processor and a memory, wherein the memory stores machine executable instructions that can be executed by the processor, and the processor is used to execute the machine executable instructions to implement the method according to any one of claims 1 to 8.

11. A machine-readable storage medium, characterized in that: The machine-readable storage medium stores machine-executable instructions, and when the machine-executable instructions are executed by a processor, the method according to any one of claims 1 to 8 is implemented.

Images