Translation method and device, electronic equipment and medium

By reducing the LayerNorm sublayer and truncation processing, the problem of quantization quality degradation caused by outliers when deploying the Transformer model on the terminal device is solved, and high-quality model quantization and translation effects are achieved.

CN119067131APending Publication Date: 2024-12-03BEIJING XIAOMI MOBILE SOFTWARE CO LTD +1
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202310628946.1
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2023-05-30
Publication Date
2024-12-03

AI Technical Summary

Technical Problem

The deployment of Transformer models on terminal devices such as mobile phones is limited. Due to the huge number of model parameters and a large number of outliers, it affects the quantization quality of the model, resulting in a decrease in the quantized model quality.

Method used

By reducing the number of LayerNorm sublayers in the Transformer model, truncating the model parameters, limiting the model parameter range, and reducing the impact of outliers, quantizing the model and converting the model into a fixed-point type.

Benefits of technology

It effectively reduces the errors of the model before and after quantization, improves the quality of the model after quantization, ensures the accuracy of the translation, and reduces the consumption of computing resources.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119067131A_ABST
    Figure CN119067131A_ABST
Patent Text Reader

Abstract

The invention provides a translation method and device, an electronic device and a medium, a Transform model is obtained, the model comprises a reference module comprising a plurality of reference layers, each reference layer comprises a plurality of sub-layers, one of the plurality of reference layers of the reference module comprises a preset sub-layer, an outlier generated by the preset sub-layer meets a preset condition, and the outlier is generated by the preset sub-layer; it shows that the outlier generated by the reference module where the preset sub-layer is located has small influence on model quantization; compared with the prior art, the method has the advantages that outliers generated in the Transform model are limited by reducing the number of preset sub-layers in the Transform model, so that the error of the model before and after quantification is reduced, the quality of the quantified model is guaranteed, and the method is simple and convenient to operate. The first preset statement is translated into the second preset statement through the quantized model, so that the translation accuracy is improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present disclosure relates to the technical field of machine learning, and in particular to a translation method, device, electronic device and medium. Background Art

[0002] At present, deep learning models based on the Transformer model structure have made breakthrough progress in many fields such as natural language processing, speech, and vision. However, due to the huge number of model parameters and the large memory occupied, the deployment of Transformer models on terminal devices such as mobile phones is limited. Model quantization is a method that can reduce memory usage, but there are usually a large number of outliers in the Transformer model, which will affect the quality of model quantization. Summary of the invention

[0003] In order to overcome the problems existing in the related art, the present disclosure provides a translation method, device, electronic device and medium.

[0004] According to a first aspect of an embodiment of the present disclosure, a translation method is provided, the translation method comprising: obtaining a Transformer model, wherein the Transformer model is a floating-point model, the Transformer model comprises a reference module, the reference module comprises a plurality of reference layers, each of the plurality of reference layers comprises a plurality of sub-layers, one reference layer of the plurality of reference layers of the reference module comprises a preset sub-layer, wherein an outlier generated by the preset sub-layer satisfies a preset condition; quantizing the Transformer model to obtain a quantized Transformer model, wherein the quantized Transformer model is a fixed-point model; and translating a first preset sentence into a second preset sentence by using the quantized Transformer model.

[0005] Optionally, the reference module includes a decoder and / or an encoder, and the preset sublayer includes a LayerNorm sublayer.

[0006] Optionally, the LayerNorm sublayer corresponds to the L1 norm.

[0007] Optionally, the method further includes: obtaining training samples, wherein the training samples include input samples and output samples, and the input samples and the output samples are both multiple; using the input samples as the input of the Transformer model to be trained, and using the output samples as the output of the Transformer model to be trained, training the Transformer model to be trained, and obtaining a trained Transformer model; obtaining model parameters of the trained Transformer model; if the model parameters of the trained Transformer model exceed a preset threshold, setting the model parameters to the preset threshold, and using the trained Transformer model as a new Transformer model to be trained, continuing to execute the steps of using the input samples as the input of the Transformer model to be trained, and using the output samples as the output of the Transformer model to be trained, training the Transformer model to be trained, and obtaining a trained Transformer model, thereby obtaining a Transformer model.

[0008] Optionally, the model parameters are multidimensional parameters. If the model parameters of the trained Transformer model exceed a preset threshold, before setting the model parameters to the preset threshold, the method further includes: obtaining the standard deviation of the multidimensional model parameters; and using a preset multiple of the standard deviation as the preset threshold.

[0009] Optionally, the LayerNorm sublayer is calculated as follows:

[0010]

[0011] Among them, L1LN is the output data of the LayerNorm sublayer, x is the input data of the LayerNorm sublayer, x is multidimensional data, μ is the mean of multidimensional x, n is the dimension of x, g, eps and b are all parameters of the LayerNorm sublayer, and g, eps and b are all learnable parameters.

[0012] Optionally, after obtaining the Transformer model, it also includes: if the Transformer model does not meet the preset requirements, adding the preset sublayer to the remaining reference layers of the Transformer model, wherein the remaining reference layers are reference layers among the multiple reference layers of the Transformer model except one reference layer including the preset sublayer.

[0013] Optionally, the Transformer model further includes an attention layer, and before obtaining the Transformer model, further includes:

[0014] Perform parameter modification processing on the attention layer of the Transformer model to obtain a modified Transformer model.

[0015] Optionally, the calculation method of the attention layer is: output1=input1+mask*(-inf), wherein input1 is the input of the attention layer, output1 is the output of the attention layer, mask is the mask, and -inf is a very large negative value; the calculation method of the modified attention layer is: output2=input2+mask*(max(abs(input2))+α)*(-1), wherein input2 is the input of the modified attention layer, output2 is the output of the modified attention layer, mask is the mask, max(abs(input2)) is the maximum absolute value of input2, and α is a preset parameter.

[0016] According to a second aspect of an embodiment of the present disclosure, a translation device is provided, comprising: an acquisition module, used to acquire a Transformer model, wherein the Transformer model is a floating-point model, the Transformer model comprises a reference module, the reference module comprises a plurality of reference layers, each of the plurality of reference layers comprises a plurality of sub-layers, one reference layer of the plurality of reference layers of the reference module comprises a preset sub-layer, wherein an outlier generated by the preset sub-layer satisfies a preset condition; a quantization module, used to quantize the Transformer model to obtain the quantized Transformer model, wherein the quantized Transformer model is a fixed-point model; and a translation module, used to translate a first preset sentence into a second preset sentence by means of the quantized Transformer model.

[0017] According to a third aspect of an embodiment of the present disclosure, an electronic device is provided, comprising: a processor; a memory for storing processor executable instructions; wherein the processor is configured to implement the steps of the method described in the first aspect when executed.

[0018] According to a fourth aspect of an embodiment of the present disclosure, a computer-readable storage medium is provided, on which computer program instructions are stored. When the program instructions are executed by a processor, the steps of the method provided in the first aspect of the present disclosure are implemented.

[0019] The present invention discloses a translation method, device, electronic device and medium, which obtain a Transformer model, wherein the Transformer model includes a reference module, the reference module includes multiple reference layers, each reference layer includes multiple sub-layers, one of the multiple reference layers of the reference module includes a preset sub-layer, and the outlier generated by the preset sub-layer meets the preset condition, indicating that the outlier generated by the reference module where the preset sub-layer is located has little influence on the model quantization; and then quantize the Transformer model to quantize the Transformer model from a floating point type to a fixed point type. Figure 1 The prior art in the invention limits the outliers generated in the Transformer model by reducing the number of preset sublayers in the Transformer model, thereby reducing the error of the Transformer model before and after quantization, ensuring the quality of the Transformer model after quantization, and translating the first preset sentence into the second preset sentence through the above-mentioned quantized model, thereby improving the accuracy of the translation.

[0020] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the present disclosure.

[0022] Figure 1 This is a schematic diagram of the model structure of a Transformer model;

[0023] Figure 2 is a flowchart of a translation method shown in an exemplary embodiment of the present disclosure;

[0024] Figure 3 is a schematic diagram of the model structure of a Transformer model shown in an exemplary embodiment of the present disclosure;

[0025] Figure 4 is a flowchart of a translation method shown in yet another exemplary embodiment of the present disclosure;

[0026] Figure 5 is a block diagram of a translation device according to an exemplary embodiment;

[0027] Figure 6 The invention is a block diagram of an electronic device for a translation method according to an exemplary embodiment. DETAILED DESCRIPTION

[0028] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. Instead, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the appended claims.

[0029] At present, deep learning models based on the Transformer model structure have made breakthrough progress in many fields such as natural language processing, speech, and vision. However, due to the huge number of model parameters and the large memory usage, the Transformer model is limited to be deployed on terminal devices such as mobile phones. Model quantization is a method that can reduce memory usage. At present, quantization is generally divided into two types: post-training quantization and quantization-aware training. Post-training quantization refers to directly quantizing the trained model. Quantization-aware training is to train the model during the quantization process. However, since there are usually a large number of outliers in the Transformer model, it is difficult to determine the range of the quantized numerical representation during the model quantization process due to the existence of outliers, which will lead to a large error between the quantized value and the pre-quantized value, and serious model quantization loss, resulting in a large loss of accuracy and thus a decrease in the quality of the quantized model.

[0030] After long-term research, the inventors found that in the model structure of the Transformer model, some network layers generate a large number of outliers. For example, the network layer including the LayerNorm (normalization) sublayer will output a large number of outliers, and the LayerNorm sublayer will amplify the accuracy loss caused by the outliers. However, the LayerNorm sublayer may exist in multiple network layers of the existing Transformer model. For example, see Figure 1 The existing Transformer model includes an encoder 110 and a decoder 120. The encoder 110 includes an Embedding layer, a Positional encoding layer, a Linear layer, and a Softmax layer. The encoder 110 also includes a network layer 111 and a network layer 112. Each network layer includes multiple sublayers. Figure 1 , the network layer 111 includes: Add&L2LayerNorm (normalization sublayer), FFN (Feed-forward Neural Network, feed-forward neural network sublayer) and Multi Head Attention (attention sublayer). In the network layer, "Add&L2 LayerNorm" represents the LayerNorm sublayer. Figure 1 The LayerNorm sublayer corresponds to the L2 norm. It can be understood that Figure 1 The LayerNorm sublayer in is calculated using the L2 norm. The network layer 111 and the network layer 112 of the encoder 110 each include two LayerNorm sublayers, and the decoder 120 includes one LayerNorm sublayer. Figure 1 There are LayerNorm sublayers in multiple network layers, which will aggravate the accuracy loss of the Transformer model and lead to poor quality of the quantized model. As a result, the poor quality quantized Transformer model is difficult to complete the translation task.

[0031] To solve the above problems, the present disclosure provides a translation method, device, electronic device and medium. Under the condition of ensuring the function of the Transformer model, part of the LayerNorm sublayer is removed on the basis of the existing Transformer model, so that during the model quantization process, the influence of the LayerNorm sublayer on the accuracy of the Transformer model can be reduced to ensure the quality of the quantized Transformer model.

[0032] The present disclosure provides a translation method. Figure 2 , Figure 2 is a flowchart of a translation method shown in an exemplary embodiment of the present disclosure, and the translation method can be applied to Figure 5 The translation device 500 shown, Figure 6 The electronic device 600 shown, Figure 6 The processor 620 and the computer readable storage medium in this embodiment are used as an example of an electronic device, where the electronic device can be a terminal device such as a smart phone, a tablet computer, a smart watch, a laptop computer, etc. Figure 2 The process shown is described in detail, and the translation method may specifically include the following steps:

[0033] Step S210, obtaining a Transformer model, wherein the Transformer model is a floating-point model, the Transformer model includes a reference module, the reference module includes multiple reference layers, each of the multiple reference layers includes multiple sub-layers, one reference layer of the multiple reference layers of the reference module includes a preset sub-layer, wherein the outlier generated by the preset sub-layer meets a preset condition.

[0034] The Transformer model may be a pre-trained model. In one embodiment, the Transformer model is stored locally in the electronic device, and when the Transformer model needs to be obtained, the electronic device can directly obtain the Transformer model from the local storage location.

[0035] In another embodiment, the Transformer model may be stored in a server connected to the electronic device. When the electronic device needs to obtain the Transformer model, the Transformer model is exported from the storage location of the Transformer model in the server, thereby obtaining the Transformer model.

[0036] The Transformer model includes multiple network modules, for example, the network module includes an input module, an encoder, a decoder, an output module, etc. In the network module of the existing Transformer model, multiple network layers contain preset sublayers. The network module after removing the preset sublayers from some network layers in the aforementioned network module containing preset sublayers is the reference module in this embodiment. The preset sublayer may output outliers. In this embodiment, only one reference layer in the reference module contains a preset sublayer. The outliers generated by the preset sublayer meet the preset conditions. Overall, the outliers generated by the reference module have little effect on the model quantization.

[0037] The outliers generated by the preset sublayer meet the preset conditions, indicating that the outliers have little effect on the model quantization. In one embodiment, the preset condition may be the number of preset outliers. The outliers generated by the preset sublayer meet the preset conditions, which can be understood as the number of outliers generated by the preset sublayer is less than or equal to the number of preset outliers. Exemplarily, the number of preset outliers is 100, and the outliers generated by the preset sublayer are less than or equal to 100.

[0038] In another embodiment, the preset condition may be a preset range, for example, the preset range is from a to b, and a is greater than b. Within the range of a and b are normal values ​​generated by the preset sublayer or outliers that have little impact on the quality of the quantized model. If the outliers generated by the preset sublayer are within the preset range, it means that the outliers generated by the preset sublayer meet the preset condition. On the contrary, if the outliers generated by the preset sublayer are not within the preset range, for example, the outliers are less than a or the outliers are greater than b, it means that such outliers are outliers that seriously affect the quality of the quantized model.

[0039] Step S220: quantize the Transformer model to obtain a quantized Transformer model, wherein the quantized Transformer model is a fixed-point model.

[0040] Exemplarily, the Transformer model is quantized by a quantization tool to obtain a quantized Transformer model. After the quantization, the Transformer model is changed from a floating point type to a fixed point type.

[0041] Optionally, this embodiment may be applied in an offline scenario or in a real-time scenario.

[0042] Step S230: translating the first preset sentence into a second preset sentence through the quantized Transformer model.

[0043] The quantized Transformer model is deployed on an electronic device, and the electronic device can implement a translation function through the model. For example, a user inputs a first preset sentence through an input interface on a display screen of an electronic device, and the electronic device translates the first sentence into a second sentence through a quantized Transformer model, and displays the second sentence on the display screen of the electronic device. For another example, the display screen on the electronic device displays a first preset sentence, for example, the first preset sentence is a sentence sent by the account of the other user on the instant messaging interface. The user clicks on a preset control on the display screen, and the electronic device responds to the click operation based on the preset control, and translates the first sentence into a second sentence through the quantized Transformer model. Exemplarily, the preset control can be a user clicking, long pressing, or other operations on the position of the first preset sentence on the display screen, and the preset control is displayed on the display screen.

[0044] Optionally, the first preset statement and the second preset statement may be in the same language. For example, the second preset statement may be an explanation or description of the first preset statement. For example, the first preset statement includes network vocabulary, and the second preset statement explains the first preset statement including the network vocabulary. The first preset statement and the second preset statement may be in different languages. For example, the first preset statement is in Chinese, and the second preset statement is in English.

[0045] The translation method provided in this embodiment obtains a Transformer model, wherein the Transformer model includes a reference module, the reference module includes multiple reference layers, each reference layer includes multiple sub-layers, one reference layer among the multiple reference layers of the reference module includes a preset sub-layer, and the outlier generated by the preset sub-layer meets the preset condition, indicating that the outlier generated by the reference module where the preset sub-layer is located has little influence on the model quantization; and then quantizes the Transformer model, quantizing the Transformer model from a floating point type to a fixed point type. Compared with Figure 1Compared with the prior art in the prior art, this embodiment adjusts the model structure of the Transformer model and reduces the number of preset sub-layers. By reducing the number of preset sub-layers in the Transformer model, the number of outliers generated in the Transformer model is limited to reduce the error of the Transformer model before and after quantization, and the quality of the Transformer model after quantization is ensured. The first preset sentence is translated into the second preset sentence through the above-mentioned quantized model, thereby improving the accuracy of the translation.

[0046] In one embodiment, the reference module includes a decoder and / or an encoder, and the preset sublayer includes a LayerNorm sublayer.

[0047] For example, taking the reference module including the encoder as an example, see Figure 3 The Transformer model shown in the embodiment of the present disclosure includes at least an encoder 310 and a decoder 320. The encoder 310 includes a network layer 311 and a network layer 312. "Add&L1 LayerNorm" in the network layer indicates the LayerNorm sublayer. The encoder 310 is used as a reference module, and its network layer 311 does not include a LayerNorm sublayer, while the network layer 312 includes a LayerNorm sublayer. Please continue to refer to Figure 3 , the decoder 320 does not include the LayerNorm sublayer. Figure 1 Compared with the Transformer model of the prior art, the Transformer model in this embodiment has fewer LayerNorm sub-layers, and the LayerNorm sub-layers that generate discrete values ​​are reduced. The discrete values ​​generated by the Transformer model are also reduced accordingly, which can reduce the accuracy loss in the subsequent model quantization process.

[0048] In the Transformer model of this embodiment, the LayerNorm sublayer corresponds to the L1 norm, that is, the LayerNorm sublayer of this embodiment is calculated using the L1 norm. Figure 1 In the Transformer model of the prior art, adjusting the L2 norm to the L1 norm in this embodiment can reduce the complexity of the model, reduce the amount of calculation, and facilitate the subsequent deployment of the model to an electronic device, thereby reducing the consumption of computing resources (such as processor resources, storage resources, etc.) of the electronic device.

[0049] It should be noted that the preset sublayer is not limited to the LayerNorm sublayer, but may also include other sublayers that can generate a large number of discrete values.

[0050] It should also be noted that Figure 3This is only an example of the Transformer model disclosed in the present invention. The layout of the LayerNorm sublayer in the reference module is not limited to Figure 3 As shown in , when the reference module includes a decoder and an encoder, a reference layer in each of the decoder and the encoder may have a LayerNorm sublayer set in it. When the reference module includes a decoder, a reference layer in the decoder may have a LayerNorm sublayer set in it. Alternatively, the number of reference modules is multiple, while ensuring the function of the Transformer model, such as Figure 3 As shown, among the multiple reference modules, only the first reference layer of the multiple reference layers of one reference module contains the LayerNorm sublayer.

[0051] Optionally, before exporting the Transformer model, the Transformer model needs to be trained in advance. The translation method provided by the present disclosure also includes: first obtaining a training sample, wherein the training sample includes an input sample and an output sample, and both the input sample and the output sample are multiple. When the Transformer model is applied in a translation scenario, the input sample can be a sentence corresponding to a first preset language, for example, the first preset language is Chinese, English or French, and the output sample can be a sentence corresponding to a second preset language, for example, the second preset language is German, Chinese or Korean. In order to enable the Transformer model to be trained to recognize the training sample, the training sample can be in the form of a sequence, wherein the sequence can be composed of corresponding words and sentences mapped to ID (Identification) in the vocabulary. Then the input sample is used as the input of the Transformer model to be trained, and the output sample is used as the output of the Transformer model to be trained, and the Transformer model to be trained is iteratively trained to obtain a trained Transformer model.

[0052] In this embodiment, the Transformer model is pre-trained, and when the Transformer model is needed in the later stage, the model can be directly exported without performing a training process each time the model is used.

[0053] The calculation method of the LayerNorm sublayer of the Transformer model provided in the embodiment of the present disclosure is as follows:

[0054]

[0055] Among them, L1LN is the output data of the LayerNorm sublayer, x is the input data of the LayerNorm sublayer, x is multidimensional data, μ is the mean of multidimensional x, n is the dimension of x, g, eps and b are all parameters of the LayerNorm sublayer, and g, eps and b are all learnable parameters.

[0056] In this embodiment, since the LayerNorm sublayer can output outliers, the eps parameter of the LayerNorm sublayer is set to a learnable parameter. The learnable eps parameter can suppress the output of outliers by the LayerNorm sublayer, further reduce the outliers generated by the Transformer model, and improve the quality of the quantized model.

[0057] The process of iteratively training the Transformer model using training samples may include:

[0058] The input sample is used as the input of the Transformer model to be trained, and the output sample is used as the output of the Transformer model to be trained, and the Transformer model to be trained is trained to obtain a trained Transformer model. The model parameters of the trained Transformer model are obtained. A preset threshold is used as a screening criterion for whether the model parameters are abnormal. If the model parameters of the trained Transformer model do not exceed the preset threshold, there are no abnormal values ​​in the model parameters. The trained Transformer model is used as a new Transformer model to be trained, and the remaining training samples are used to continue training the new Transformer model to be trained. If the model parameter of the trained Transformer model exceeds the preset threshold, it means that the model parameter has an abnormal value, and the model parameter is truncated, that is, the model parameter is set to the preset threshold, and the trained Transformer model is used as a new Transformer model to be trained, and the input sample is used as the input of the Transformer model to be trained, and the output sample is used as the output of the Transformer model to be trained, the Transformer model to be trained is trained, and the trained Transformer model is obtained to obtain the Transformer model.

[0059] The preset threshold may be a fixed value set based on experience. Alternatively, the model parameters of different Transformer models may be adapted to different Transformer models. The preset threshold may also be set according to each specific Transformer model. For example, the model parameters of the Transformer model are multidimensional parameters. The standard deviation of the multidimensional model parameters is obtained, and the preset multiple standard deviation is used as the preset threshold. The preset multiple may be 1.2 times, 1.5 times, 2 times, 2.1 times, etc. In the actual training process, the preset multiple may also be adjusted according to the predictive ability and stability of the Transformer model.

[0060] Exemplarily, taking the preset multiple of 2 as an example, the truncation process can be expressed by the following formula: W2=clip(W1, -2σ, 2σ), wherein W2 is the model parameter after truncation, W1 is the model parameter before truncation, clip represents the truncation process, and σ represents the above standard deviation.

[0061] It should be noted that during the iterative training process, the model parameters need to be exported after each iterative training, and the above screening is performed through the preset threshold. If the model parameters do not exceed the preset threshold, the iterative training continues. If the model parameters exceed the preset threshold, the model parameters need to be truncated before the truncated model is iteratively trained.

[0062] In this embodiment, the model parameters are truncated during the iterative training of the Transformer model, and the outliers in the model parameters can be truncated. The model is optimized during the training process, and the outliers output by the Transformer model are further reduced, thereby reducing the accuracy loss of the quantized model.

[0063] Optionally, the LayerNorm sublayer belongs to the Transformer model, and therefore, the parameters of the LayerNorm sublayer can also be used as model parameters. Exemplarily, the model parameters include g parameters, b parameters, and the like.

[0064] Optionally, after step S210, if the Transformer model obtained by training in this embodiment does not meet the preset requirements, for example, the Transformer model is unstable or even cannot realize the corresponding function of the model, the preset sublayer is added to the remaining reference layers of the Transformer model, wherein the remaining reference layers are reference layers among the multiple reference layers of the Transformer model except one reference layer including the preset sublayer.

[0065] For example, in Figure 1 On the basis of Figure 3The LayerNorm sublayer in the network layer 311 in the Transformer model of the present invention has been deleted. In order to ensure the performance of the model, the LayerNorm sublayer in the network layer 311 is restored. For example, a LayerNorm sublayer is restored in the network layer 311. If the restored Transformer model still does not meet the preset requirements, another LayerNorm sublayer is restored.

[0066] Optionally, the attention layer of the Transformer model is subjected to parameter modification processing to obtain a modified Transformer model. After the above modification, the model is exported to obtain a Transformer model, which can be used for subsequent quantization processing.

[0067] The calculation method of the attention layer is: output1=input1+mask*(-inf), where input1 is the input of the attention layer, which can be understood as the input that needs to be masked, output1 is the output of the attention layer, mask is the mask, and mask is a matrix with a value of 0 or 1, where the value of mask is 0, indicating that input1 does not need to be masked, and the value of mask is 1, indicating that input1 needs to be masked, and -inf is a very large negative value.

[0068] Since the inf value is too large, this part will cause errors when calculating the quantization parameters. Therefore, this part needs to be adjusted. The calculation method of the modified attention layer is: output2=input2+mask*(max(abs(input2))+α)*(-1), wherein input2 is the input of the modified attention layer, output2 is the output of the modified attention layer, mask is the mask, max(abs(input2)) is the maximum value of the absolute value of input2, and α is the preset parameter. α can be adjusted according to input2. For example, if the value of input2 is -10, α can be any value between -3 and -2. For another example, if the value of input2 is -11, α can be any value between -4 and -3.

[0069] Based on the comparison between the above two formulas, this solution modifies the original -inf of the attention layer to (max(abs(input2))+α)*(-1). Since α can suppress the error caused by -inf and reduce the generation of outliers, the accuracy loss of the quantized model can be further reduced.

[0070] Based on the above embodiment, this embodiment provides a translation method. Figure 4, the translation method may include the following steps:

[0071] Step S410: adjust the number of LayerNorm layers to limit the model parameter range for training.

[0072] Step S420: Export the Transformer model using the dynamic mask strategy.

[0073] Step S430: The quantization tool converts the exported floating-point Transformer model into a fixed-point Transformer model.

[0074] Step S440: terminal-side reasoning.

[0075] It means that the quantized Transformer model is deployed on an electronic device for inference and use. Compared with before quantization, the quantized Transformer model occupies less memory in the electronic device. The electronic device can implement functions such as translation through the quantized Transformer model. For example, the electronic device translates a sentence in a first preset language into a sentence in a second preset language through the quantized Transformer model.

[0076] The specific description of steps S410 to S440 can be found in the above embodiments and will not be repeated here.

[0077] The translation method provided by the embodiment of the present disclosure uses a post-training quantization method. Compared with quantization-aware training, the quantization quality of the Transformer structure model can be improved without quantization-aware training. There is no need to insert quantization and dequantization nodes during model training, which greatly improves the efficiency of quantization training. In addition, the model structure (i.e., the deletion of the LayerNorm sublayer) and the training method of limiting the parameter range (i.e., the aforementioned truncation processing method) given in the embodiment of the present disclosure can further reduce the number of outliers in the model, greatly reduce the error in the model quantization process, and improve the accuracy of the quantized model. The Transformer model obtained by the present disclosure can perform full INT8 quantization reasoning, and all operators can be quantized during reasoning on the terminal. For operators involved in Softmax, LayerNorm, etc., there is no need to retain floating-point calculations. The model can be inferred on hardware that only supports fixed-point calculations. In order to ensure the normal use of the Transformer model, the position and number of layers of LayerNorm can be adjusted according to actual conditions, and the generation of outliers can be reduced while ensuring the normal operation of the model.

[0078] Based on the same concept, the present disclosure also provides a translation device, see Figure 5, the translation device 500 includes: an acquisition module 510, a quantification module 520 and a translation module 530;

[0079] An acquisition module 510 is used to acquire a Transformer model, wherein the Transformer model is a floating-point model, the Transformer model includes a reference module, the reference module includes multiple reference layers, each of the multiple reference layers includes multiple sub-layers, one reference layer of the multiple reference layers of the reference module includes a preset sub-layer, and the outlier generated by the preset sub-layer meets a preset condition;

[0080] A quantization module 520 is used to quantize the Transformer model to obtain a quantized Transformer model, wherein the quantized Transformer model is a fixed-point model;

[0081] The translation module 530 is used to translate the first preset sentence into a second preset sentence through the quantized Transformer model.

[0082] Optionally, the reference module includes a decoder and / or an encoder, and the preset sublayer includes a LayerNorm sublayer.

[0083] Optionally, the LayerNorm sublayer corresponds to the L1 norm.

[0084] Optionally, the translation device 500 further includes: a sample collection module, a first training module, a parameter acquisition module and a second training module;

[0085] A sample collection module, used to obtain training samples, wherein the training samples include input samples and output samples, and both the input samples and the output samples are multiple;

[0086] A first training module is used to use the input sample as the input of the Transformer model to be trained, and use the output sample as the output of the Transformer model to be trained, to train the Transformer model to be trained, and obtain a trained Transformer model;

[0087] A parameter acquisition module, used to obtain model parameters of the trained Transformer model;

[0088] The second training module is used to set the model parameters to the preset threshold if the model parameters of the trained Transformer model exceed the preset threshold, and use the trained Transformer model as a new Transformer model to be trained, continue to execute the steps of using the input samples as the input of the Transformer model to be trained, and using the output samples as the output of the Transformer model to be trained, training the Transformer model to be trained, and obtaining the trained Transformer model, thereby obtaining the Transformer model.

[0089] Optionally, the model parameters are multi-dimensional parameters, and the translation device 500 further includes: a standard deviation acquisition module and a preset threshold module;

[0090] A standard deviation acquisition module, used to acquire the standard deviation of the multi-dimensional model parameters;

[0091] The preset threshold module is used to use the preset multiple standard deviation as the preset threshold.

[0092] Optionally, the LayerNorm sublayer is calculated as follows:

[0093]

[0094] Among them, L1LN is the output data of the LayerNorm sublayer, x is the input data of the LayerNorm sublayer, x is multidimensional data, μ is the mean of multidimensional x, n is the dimension of x, g, eps and b are all parameters of the LayerNorm sublayer, and g, eps and b are all learnable parameters.

[0095] Optionally, the translation device 500 further includes: a new module;

[0096] A new module is added, which is used to add the preset sublayer to the remaining reference layers of the Transformer model if the Transformer model does not meet the preset requirements, wherein the remaining reference layers are reference layers among the multiple reference layers of the Transformer model except one reference layer containing the preset sublayer.

[0097] Optionally, the translation device 500 further includes: a modification module;

[0098] A modification module is used to perform parameter modification processing on the attention layer of the Transformer model to obtain a modified Transformer model.

[0099] Optionally, the calculation method of the attention layer is: output1=input1+mask*(-inf), wherein input1 is the input of the attention layer, output1 is the output of the attention layer, mask is the mask, and -inf is a very large negative value;

[0100] The calculation method of the modified attention layer is: output2=input2+mask*(max(abs(input2))+α)*(-1), wherein input2 is the input of the modified attention layer, output2 is the output of the modified attention layer, mask is the mask, max(abs(input2)) is the maximum absolute value of input2, and α is a preset parameter.

[0101] Regarding the device in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0102] The present disclosure also provides a computer-readable storage medium having computer program instructions stored thereon, and the program instructions, when executed by a processor, implement the steps of the translation method provided by the present disclosure.

[0103] Figure 6 It is a block diagram of an electronic device for a translation method according to an exemplary embodiment. For example, the electronic device 600 may be a mobile phone, a computer, a digital broadcast terminal, a messaging device, a game console, a tablet device, a medical device, a fitness device, a personal digital assistant, etc.

[0104] Reference Figure 6 , the electronic device 600 may include one or more of the following components: a processing component 602 , a memory 604 , a power component 606 , a multimedia component 608 , an audio component 610 , an input / output interface 612 , a sensor component 614 , and a communication component 616 .

[0105] The processing component 602 generally controls the overall operation of the electronic device 600, such as operations associated with display, phone calls, data communications, camera operations, and recording operations. The processing component 602 may include one or more processors 620 to execute instructions to complete all or part of the steps of the above-mentioned method. In addition, the processing component 602 may include one or more modules to facilitate the interaction between the processing component 602 and other components. For example, the processing component 602 may include a multimedia module to facilitate the interaction between the multimedia component 608 and the processing component 602.

[0106] The memory 604 is configured to store various types of data to support operations on the electronic device 600. Examples of such data include instructions for any application or method operating on the electronic device 600, contact data, phone book data, messages, pictures, videos, etc. The memory 604 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0107] The power supply component 606 provides power to the various components of the electronic device 600. The power supply component 606 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to the electronic device 600.

[0108] The multimedia component 608 includes a screen that provides an output interface between the electronic device 600 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touch, slide, and gestures on the touch panel. The touch sensor may not only sense the boundaries of the touch or slide action, but also detect the duration and pressure associated with the touch or slide operation. In some embodiments, the multimedia component 608 includes a front camera and / or a rear camera. When the electronic device 600 is in an operating mode, such as a shooting mode or a video mode, the front camera and / or the rear camera may receive external multimedia data. Each front camera and the rear camera may be a fixed optical lens system or have a focal length and optical zoom capability.

[0109] The audio component 610 is configured to output and / or input audio signals. For example, the audio component 610 includes a microphone (MIC), and when the electronic device 600 is in an operating mode, such as a call mode, a recording mode, and a voice recognition mode, the microphone is configured to receive an external audio signal. The received audio signal can be further stored in the memory 604 or sent via the communication component 616. In some embodiments, the audio component 610 also includes a speaker for outputting audio signals.

[0110] The input / output interface 612 provides an interface between the processing component 602 and the peripheral interface modules, which may be keyboards, click wheels, buttons, etc. These buttons may include but are not limited to: a home button, a volume button, a start button, and a lock button.

[0111] The sensor assembly 614 includes one or more sensors for providing various aspects of status assessment for the electronic device 600. For example, the sensor assembly 614 can detect the open / closed state of the electronic device 600, the relative positioning of the components, such as the display and keypad of the electronic device 600, and the sensor assembly 614 can also detect the position change of the electronic device 600 or a component of the electronic device 600, the presence or absence of contact between the user and the electronic device 600, the orientation or acceleration / deceleration of the electronic device 600, and the temperature change of the electronic device 600. The sensor assembly 614 may include a proximity sensor configured to detect the presence of a nearby object without any physical contact. The sensor assembly 614 may also include an optical sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, the sensor assembly 614 may also include an acceleration sensor, a gyroscope sensor, a magnetic sensor, a pressure sensor, or a temperature sensor.

[0112] The communication component 616 is configured to facilitate wired or wireless communication between the electronic device 600 and other devices. The electronic device 600 can access a wireless network based on a communication standard, such as WiFi, 2G or 3G, or a combination thereof. In an exemplary embodiment, the communication component 616 receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 616 also includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology and other technologies.

[0113] In an exemplary embodiment, the electronic device 600 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the above methods.

[0114] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 604 including instructions, and the instructions can be executed by a processor 620 of an electronic device 600 to perform the above method. For example, the non-transitory computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.

[0115] In addition to being an independent electronic device, the electronic device 600 can also be a part of an independent electronic device. For example, in one embodiment, the device can be an integrated circuit (IC) or a chip, wherein the integrated circuit can be an IC or a collection of multiple ICs; the chip can include but is not limited to the following types: GPU (Graphics Processing Unit), CPU (Central Processing Unit), FPGA (Field Programmable Gate Array), DSP (Digital Signal Processor), ASIC (Application Specific Integrated Circuit), SOC (System on Chip, SoC), etc. The above-mentioned integrated circuit or chip can be used to execute executable instructions (or codes) to implement the above-mentioned translation method. The executable instructions can be stored in the integrated circuit or chip, or can be obtained from other devices or equipment, such as the integrated circuit or chip including a processor, a memory, and an interface for communicating with other devices. The executable instruction may be stored in the memory, and when the executable instruction is executed by the processor, the above-mentioned translation method is implemented; alternatively, the integrated circuit or chip may receive the executable instruction through the interface and transmit it to the processor for execution, so as to implement the above-mentioned translation method.

[0116] In another exemplary embodiment, a computer program product is further provided. The computer program product includes a computer program executable by a programmable device. The computer program has a code portion for executing the above translation method when executed by the programmable device.

[0117] In summary, the translation method, device, electronic device and medium provided by the present disclosure obtain a Transformer model, wherein the Transformer model includes a reference module, the reference module includes multiple reference layers, each reference layer includes multiple sub-layers, one of the multiple reference layers of the reference module includes a preset sub-layer, and the outlier generated by the preset sub-layer meets the preset condition, indicating that the outlier generated by the reference module where the preset sub-layer is located has little effect on the model quantization; then the Transformer model is quantized to quantize the Transformer model from a floating point type to a fixed point type, compared with Figure 1The prior art in the invention limits the outliers generated in the Transformer model by reducing the number of preset sublayers in the Transformer model, thereby reducing the error of the Transformer model before and after quantization, ensuring the quality of the Transformer model after quantization, and translating the first preset sentence into the second preset sentence through the above-mentioned quantized model, thereby improving the accuracy of the translation.

[0118] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the present disclosure. The present disclosure is intended to cover any variations, uses or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art that are not disclosed in the present disclosure. The description and examples are to be considered as exemplary only, and the true scope and spirit of the present disclosure are indicated by the following claims.

[0119] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. A translation method, characterized in that: The method comprises: Acquire a Transformer model, wherein the Transformer model is a floating-point model, the Transformer model includes a reference module, the reference module includes multiple reference layers, each of the multiple reference layers includes multiple sublayers, one reference layer of the multiple reference layers of the reference module includes a preset sublayer, and an outlier generated by the preset sublayer satisfies a preset condition; Quantizing the Transformer model to obtain a quantized Transformer model, wherein the quantized Transformer model is a fixed-point model; The first preset sentence is translated into the second preset sentence by using the quantized Transformer model.

2. The method according to claim 1, characterized in that The reference module includes a decoder and / or an encoder, and the preset sublayer includes a LayerNorm sublayer.

3. The method according to claim 2, characterized in that The LayerNorm sublayer corresponds to the L1 norm.

4. The method according to claim 2, characterized in that: The method further comprises: Acquire training samples, wherein the training samples include input samples and output samples, and both the input samples and the output samples are multiple; Using the input sample as the input of the Transformer model to be trained, and using the output sample as the output of the Transformer model to be trained, training the Transformer model to be trained, and obtaining a trained Transformer model; Obtaining model parameters of the trained Transformer model; If the model parameters of the trained Transformer model exceed the preset threshold, the model parameters are set to the preset threshold, and the trained Transformer model is used as a new Transformer model to be trained, and the steps of using the input sample as the input of the Transformer model to be trained and the output sample as the output of the Transformer model to be trained, training the Transformer model to be trained, and obtaining the trained Transformer model are continued to obtain the Transformer model.

5. The method according to claim 4, characterized in that The model parameter is a multi-dimensional parameter. If the model parameter of the trained Transformer model exceeds a preset threshold, the model parameter is set to the preset threshold, and the method further includes: Obtaining multidimensional standard deviations of the model parameters; The preset multiple of the standard deviation is used as the preset threshold.

6. The method according to claim 5, characterized in that The LayerNorm sublayer is calculated as follows: Among them, L1LN is the output data of the LayerNorm sublayer, x is the input data of the LayerNorm sublayer, x is multidimensional data, μ is the mean of multidimensional x, n is the dimension of x, g, eps and b are all parameters of the LayerNorm sublayer, and g, eps and b are all learnable parameters.

7. The method according to claim 4, characterized in that After obtaining the Transformer model, the method further includes: If the Transformer model does not meet the preset requirements, the preset sublayer is added to the remaining reference layers of the Transformer model, wherein the remaining reference layers are reference layers among the multiple reference layers of the Transformer model except one reference layer including the preset sublayer.

8. The method according to any one of claims 1 to 7, characterized in that: The Transformer model further includes an attention layer, and before obtaining the Transformer model, further includes: Perform parameter modification processing on the attention layer of the Transformer model to obtain a modified Transformer model.

9. The method according to claim 8, characterized in that The calculation method of the attention layer is: output1=input1+mask*(-inf), where input1 is the input of the attention layer, output1 is the output of the attention layer, mask is the mask, and -inf is a very large negative value; The calculation method of the modified attention layer is: output2=input2+mask*(max(abs(input2))+α)*(-1), wherein input2 is the input of the modified attention layer, output2 is the output of the modified attention layer, mask is the mask, max(abs(input2)) is the maximum absolute value of input2, and α is a preset parameter.

10. A translation device, characterized in that: The device comprises: An acquisition module, used to acquire a Transformer model, wherein the Transformer model is a floating-point model, the Transformer model includes a reference module, the reference module includes multiple reference layers, each of the multiple reference layers includes multiple sub-layers, one reference layer of the multiple reference layers of the reference module includes a preset sub-layer, wherein the outlier generated by the preset sub-layer meets a preset condition; A quantization module, used to quantize the Transformer model to obtain a quantized Transformer model, wherein the quantized Transformer model is a fixed-point model; A translation module is used to translate the first preset sentence into a second preset sentence through the quantized Transformer model.

11. An electronic device, characterized in that: include: processor; a memory for storing processor-executable instructions; The processor is configured to implement the steps of any one of the methods of claims 1 to 9 when executing.

12. A computer-readable storage medium having computer program instructions stored thereon, characterized in that: When the program instructions are executed by a processor, the steps of the method described in any one of claims 1 to 9 are implemented.