Binary quantization translation model generation method, machine translation method and related equipment

Abstract

The invention discloses a binary quantization translation model generation method, a machine translation method and related equipment, and the generation method comprises the steps: collecting sample data of multiple languages, and constructing a plurality of bilingual sentence pairs according to the sample data of the multiple languages, each bilingual sentence pair comprising two mutually translated sentence texts; the bilingual sentence pairs serve as training samples, a pre-trained original translation model and a pre-trained compression translation model are trained through knowledge distillation, and the compression translation model is obtained by conducting binary quantization processing on the original translation model; and taking the trained compression translation model as a binary quantization translation model capable of being deployed to edge equipment, wherein the binary quantization translation model is used for converting a given source language statement into a target language statement. The scale of the translation model is effectively reduced through binary quantization, the model is trained through knowledge distillation, and the binary quantization translation model which can adapt to edge equipment configuration and is good in translation performance is obtained, so that the translation model can be successfully deployed at an edge computing end, and the translation efficiency is improved.

Description

Technical Field

[0001] The present invention relates to the field of financial technology, and in particular to a method for generating a binary quantization translation model, a machine translation method and related equipment. Background Art

[0002] Machine translation, also known as automatic translation, is the process of using computers to convert one natural language (source language) into another natural language (target language). With the rapid development of economic globalization and the Internet, machine translation technology plays an increasingly important role in many fields. In the financial field, machine translation models can reduce the time and cost of manual translation and improve translation speed and quality.

[0003] At the same time, with the development of edge computing, the use of AI technology on the terminal side can process the collected data instantly without uploading it to the cloud processing center, thereby significantly improving the system response speed and reducing processing delays. Therefore, in the financial field, more and more AI models are deployed on the edge computing side to process valuable key data more efficiently, alleviate network bandwidth limitations and reduce central storage pressure.

[0004] However, for the edge deployment of machine translation, traditional machine translation models are usually very large. The computing resources and memory capacity of edge devices are limited and cannot cope with complex machine translation models. Traditional int8 quantization can only convert floating point numbers into integers between -128 and 127, which is difficult to meet the requirements for complex models. Therefore, it is difficult to compress large machine translation models to a configuration that is suitable for edge devices, making it very difficult to deploy machine translation models on edge devices. Summary of the invention

[0005] In view of the above-mentioned deficiencies in the prior art, the purpose of the present invention is to provide a method for generating a binary quantization translation model, a machine translation method and related equipment that can be applied to financial technology or other related fields. Its main purpose is to improve the adaptability of the machine translation model to the edge device, so that the translation model can be deployed on the edge computing end to improve translation efficiency.

[0006] The technical solution of the present invention is as follows:

[0007] A first aspect of the present invention provides a method for generating a binary quantized translation model, comprising:

[0008] Collecting sample data in multiple languages, and constructing a plurality of bilingual sentence pairs according to the sample data in the multiple languages, wherein the bilingual sentence pairs include two sentence texts translated from each other;

[0009] The bilingual sentence pairs are used as training samples, and a pre-trained original translation model and a compressed translation model are trained through knowledge distillation, wherein the compressed translation model is obtained by performing binary quantization processing on the original translation model;

[0010] The trained compression translation model is used as a binary quantization translation model that can be deployed to an edge device. The binary quantization translation model is used to convert a given source language sentence into a target language sentence.

[0011] In one embodiment, the method of using the bilingual sentence pairs as training samples and training the pre-trained original translation model and the compressed translation model through knowledge distillation includes:

[0012] Inputting the first language text in the bilingual sentence pair into a pre-trained original translation model and a compressed translation model simultaneously, obtaining a first prediction output of the pre-trained original translation model and a second prediction output of the compressed translation model;

[0013] According to the first prediction output, the second prediction output and the second language text corresponding to the first language text in the bilingual sentence pair, the parameters of the compression translation model are adjusted according to a preset knowledge distillation strategy until a preset convergence condition is met, and the training is completed.

[0014] In one embodiment, adjusting the parameters of the compression translation model according to a preset knowledge distillation strategy based on the first prediction output, the second prediction output, and the second language text corresponding to the first language text in the bilingual sentence pair includes:

[0015] Calculating a soft label loss of the compression translation model according to the first prediction output and the second prediction output;

[0016] Calculating a hard label loss of the compression translation model based on the second prediction output and the second language text;

[0017] A total loss is calculated according to the soft label loss and the hard label loss, and parameters of the compression translation model are adjusted by back-propagation according to the total loss.

[0018] In one embodiment, the binary quantization processing of the original translation model includes:

[0019] Binarizing the floating point numbers in the original translation model;

[0020] Scaling the binary floating point numbers by a scaling factor or layer normalization function;

[0021] Adding a residual operation to the binary attention layer results in a binary quantized compressed translation model.

[0022] In one embodiment, the binarization of the floating point numbers in the original translation model specifically includes:

[0023] By the formula clip(x,x min ,x max ):=min(x max ,max(x min ,x))

[0024]

[0025] Binarize the floating point numbers in the original translation model, where x is the input floating point number, x min is the minimum value in x, x max is the maximum value in x; x b is the result of binarization of x, ε is a floating point number, B is the binarization limit, clip is the clipping function, and floor is the rounding down operation.

[0026] A second aspect of the present invention provides a machine translation method, comprising:

[0027] Obtain source language sentences to be translated;

[0028] Inputting the source language sentence to be translated into a binary quantization translation model, and outputting a target language sentence corresponding to the source language sentence;

[0029] The binary quantization translation model is obtained by using the binary quantization translation model generation method as described above.

[0030] A third aspect of the present invention provides a device for generating a binary quantization translation model, comprising:

[0031] A collection and construction module, used for collecting sample data in multiple languages, and constructing a plurality of bilingual sentence pairs according to the sample data in the multiple languages, wherein the bilingual sentence pairs include two sentence texts translated from each other;

[0032] A training module, taking the bilingual sentence pairs as training samples, and training the pre-trained original translation model and the compressed translation model through knowledge distillation, wherein the compressed translation model is obtained by performing binary quantization processing on the original translation model;

[0033] A deployment module is generated, which is used to use the trained compression translation model as a binary quantization translation model that can be deployed to an edge device, wherein the binary quantization translation model is used to convert a given source language sentence into a target language sentence.

[0034] A fourth aspect of the present invention provides a machine translation device, comprising:

[0035] An acquisition module, used to acquire source language sentences to be translated;

[0036] A translation module, used for inputting the source language sentence to be translated into a binary quantization translation model, and outputting a target language sentence corresponding to the source language sentence;

[0037] The binary quantization translation model is obtained by using the binary quantization translation model generation method as described above.

[0038] A fifth aspect of the present invention provides an electronic device, comprising at least one processor; and

[0039] a memory communicatively connected to the at least one processor; wherein,

[0040] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the above-mentioned method for generating a binary quantization translation model or execute the above-mentioned machine translation method.

[0041] A sixth aspect of the present invention provides a non-volatile computer-readable storage medium, which stores computer-executable instructions. When the computer-executable instructions are executed by one or more processors, the one or more processors can execute the above-mentioned method for generating a binary quantization translation model or execute the above-mentioned machine translation method.

[0042] Beneficial effects: The present invention discloses a method for generating a binary quantization translation model, a machine translation method and related equipment. Compared with the prior art, the embodiment of the present invention collects sample data in multiple languages, constructs multiple bilingual sentence pairs based on the sample data in multiple languages, and the bilingual sentence pairs include two sentence texts that are translated into each other; the bilingual sentence pairs are used as training samples, and the pre-trained original translation model and the compressed translation model are trained through knowledge distillation, and the compressed translation model is obtained by binary quantization of the original translation model; the trained compressed translation model is used as a binary quantization translation model that can be deployed to the edge device, and the binary quantization translation model is used to convert a given source language sentence into a target language sentence. The scale of the translation model is effectively reduced through binary quantization, and the model is trained through knowledge distillation to obtain a binary quantization translation model that can adapt to the edge device configuration and has good translation performance, so that the translation model can be successfully deployed on the edge computing end, improving translation efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] The present invention will be further described below with reference to the accompanying drawings and embodiments, in which:

[0044] Figure 1 A flow chart of a method for generating a binary quantization translation model provided by an embodiment of the present invention;

[0045] Figure 2 A flowchart of step S200 in the method for generating a binary quantization translation model provided in an embodiment of the present invention;

[0046] Figure 3 A flowchart of step S202 in the method for generating a binary quantization translation model provided in an embodiment of the present invention;

[0047] Figure 4 A flowchart of a binarization process in a method for generating a binary quantized translation model provided by an embodiment of the present invention;

[0048] Figure 5 A schematic diagram of adding a residual operation to an attention layer in a method for generating a binary quantization translation model provided in an embodiment of the present invention;

[0049] Figure 6 A flow chart of a machine translation method provided by an embodiment of the present invention;

[0050] Figure 7 A schematic diagram of functional modules of a device for generating a binary quantized translation model provided by an embodiment of the present invention;

[0051] Figure 8 A schematic diagram of the functional modules of a machine translation device provided by an embodiment of the present invention;

[0052] Fig. 9 A schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0053] In order to make the purpose, technical solution and effect of the present invention clearer and more specific, the present invention is further described in detail below. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention. The embodiments of the present invention are described below in conjunction with the accompanying drawings.

[0054] Machine translation, also known as automatic translation, is the process of using computers to convert one natural language (source language) into another natural language (target language). With the rapid development of economic globalization and the Internet, machine translation technology plays an increasingly important role in many fields. In the financial field, machine translation models can reduce the time and cost of manual translation and improve translation speed and quality.

[0055] At the same time, with the development of edge computing, the use of AI technology on the terminal side can process the collected data instantly without uploading it to the cloud processing center, thereby significantly improving the system response speed and reducing processing delays. Therefore, in the financial field, more and more AI models are deployed on the edge computing side to process valuable key data more efficiently, alleviate network bandwidth limitations and reduce central storage pressure.

[0056] However, for the edge deployment of machine translation, traditional machine translation models are usually very large. The computing resources and memory capacity of edge devices are limited and cannot cope with complex machine translation models. Traditional int8 quantization can only convert floating point numbers into integers between -128 and 127, which is difficult to meet the requirements for complex models. Therefore, it is difficult to compress large machine translation models to a configuration that is suitable for edge devices, making it very difficult to deploy machine translation models on edge devices.

[0057] In order to solve the above problems, the present invention proposes a method for generating a binary quantization translation model and a machine translation method. The system architecture of the method for generating a binary quantization translation model or the machine translation method provided in the present embodiment may include a first terminal device, a second terminal device, a third terminal device, a network and a server. The network is used to provide a medium for a communication link between the first terminal device, the second terminal device, the third terminal device and the server. The network may include various connection types, such as wired and / or wireless communication links, etc.

[0058] The user may use the first terminal device, the second terminal device, and the third terminal device to interact with the server through the network to receive or send messages, etc. Various communication client applications may be installed on the first terminal device, the second terminal device, and the third terminal device, such as knowledge reading applications, web browser applications, search applications, instant messaging tools, email clients, and / or social platform software, etc. (only as examples).

[0059] The first terminal device, the second terminal device, and the third terminal device may be various electronic devices having display screens and supporting web browsing, including but not limited to smart phones, tablet computers, laptop computers, desktop computers, and the like.

[0060] The server may be a server that provides various services, such as a background management server that provides support for the content browsed by the user using the first terminal device, the second terminal device, and the third terminal device (for example only). The background management server may analyze and process the received data such as the user request, and feed back the processing results (such as web pages, information, or data obtained or generated according to the user request) to the terminal device. The server may be a cloud server, also known as a cloud computing server or a cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services ("Virtual Private Server", or "VPS" for short). The server may also be a server for a distributed system, or a server combined with a blockchain.

[0061] It should be noted that the method for generating a binary quantized translation model or the machine translation method provided in the embodiment of the present invention can generally be executed by the first terminal device, the second terminal device or the third terminal device. Correspondingly, the device for generating a binary quantized translation model or the machine translation device provided in the embodiment of the present invention can also be arranged in the first terminal device, the second terminal device or the third terminal device.

[0062] Alternatively, the generation method or machine translation method of the binary quantization translation model provided by the embodiment of the present invention may also be generally performed by a server. Accordingly, the generation device or machine translation device of the binary quantization translation model provided by the embodiment of the present invention may generally be arranged in a server. The generation method or machine translation method of the binary quantization translation model provided by the embodiment of the present invention may also be performed by a server or server cluster that is different from the server and can communicate with the first terminal device, the second terminal device, the third terminal device and / or the server. Accordingly, the generation device or machine translation device of the binary quantization translation model provided by the embodiment of the present invention may also be arranged in a server or server cluster that is different from the server and can communicate with the first terminal device, the second terminal device, the third terminal device and / or the server.

[0063] It should be understood that the numbers of the above terminal devices, networks and servers are only illustrative. Any number of terminal devices, networks and servers may be provided according to implementation requirements.

[0064] See also Figure 1 , Figure 1 The flowchart of one embodiment of the method for generating a binary quantized translation model provided by the present invention. Figure 1 As shown, the method specifically comprises the following steps:

[0065] S100: Collect sample data in multiple languages, and construct multiple bilingual sentence pairs according to the sample data in multiple languages, wherein the bilingual sentence pairs include two sentence texts that are translated into each other.

[0066] In this embodiment, sample data in multiple languages ​​are collected to construct a plurality of bilingual sentence pairs for subsequent model training. The bilingual sentence pairs include two sentence texts that are translated into each other, that is, the bilingual sentence pairs include a first language text and a second language text, and the first language text and the second language text are in a mutual translation relationship. Specifically, data in multiple languages ​​can be collected to construct bilingual sentence pairs with mutual translation relationships between different languages. For example, the sample data includes sample data in Chinese, English, French, Korean, German and other languages. The constructed bilingual sentence pairs can be [Chinese text, English text], [Chinese text, French text] and the like. After the translation model is trained using the constructed bilingual sentence pairs in multiple languages, the model can complete the translation task between any two languages ​​in the multiple languages.

[0067] S200, using the bilingual sentence pairs as training samples, and training a pre-trained original translation model and a compressed translation model through knowledge distillation, wherein the compressed translation model is obtained by performing binary quantization processing on the original translation model.

[0068] In this embodiment, in order to realize the deployment of the translation model on the edge device, the original translation model is pre-built and the original translation model is binary quantized to obtain the corresponding compressed translation model. Binarization can greatly reduce the storage requirements of the model parameters by quantizing the weights and activations of the model into a single bit. For the neural machine translation model using int8 quantization, the binary quantization process can reduce the model size by 16 times, making it suitable for use in edge computing application scenarios.

[0069] For the compressed translation model obtained by binary quantization, in order to ensure the model performance, the constructed bilingual sentence pairs are used as training samples, and the pre-trained original translation model and the compressed translation model are trained through knowledge distillation. Knowledge distillation is to transfer the knowledge of a complex model (teacher model) to a simplified model (student model). Through knowledge distillation, the complexity and computational overhead of the model can be reduced while maintaining high performance. The teacher model is usually a larger and more complex model with a higher accuracy on the training set. In this embodiment, the original translation model based on the Transformer network is used as the teacher model. For example, the teacher model uses a Transformer with a 12-layer attention mechanism (six layers for encoding and six layers for decoding), the size of the hidden layer unit is 512, the attention mechanism is single-headed, and the hidden layer unit size of the fully connected layer is also 512; while the student model is a smaller and simpler model. In this embodiment, the compressed translation model obtained by binary quantization is used as the student model. Its goal is to achieve performance close to or exceeding that of the teacher model by learning from the teacher model. Specifically, a Transformer with a 4-layer attention mechanism, two layers for encoding (Encoder) and the other two layers for decoding (Decoder), the size of the hidden layer unit is 128, the attention mechanism is single-headed, and the hidden layer unit size of the fully connected layer is also 128.

[0070] Specifically, the teacher model is first trained on translation tasks using a large-scale dataset to achieve a high accuracy rate. During the knowledge distillation training process, the pre-trained teacher model passes the mapping relationship between different languages ​​it has learned to the student model in a corresponding form, such as the output probability distribution, attention weights, or compressed intermediate layer representations. The student model can imitate the behavior of the teacher model and adjust its own learning goals, so as to make its output as close to the output of the teacher model as possible, thereby compressing the model size while maintaining model performance.

[0071] S300: Use the trained compression translation model as a binary quantization translation model that can be deployed to an edge device, where the binary quantization translation model is used to convert a given source language sentence into a target language sentence.

[0072] In this embodiment, the compressed translation model trained under knowledge distillation not only adapts to the configuration of the edge device but also maintains the translation performance. It is used as a binary quantization translation model that can be deployed to the edge device. By deploying the binary quantization model on the edge device, the source language sentences can be directly received at the edge and converted into target language sentences without uploading to the cloud processing center for translation processing, which improves the system response speed and reduces processing delays, effectively improves translation efficiency, reduces the model scale and reduces deployment costs.

[0073] In one embodiment, Figure 2 As shown, step S200 includes:

[0074] S201, inputting the first language text in the bilingual sentence pair into a pre-trained original translation model and a compressed translation model at the same time, obtaining a first prediction output of the pre-trained original translation model and a second prediction output of the compressed translation model;

[0075] S202: According to the first prediction output, the second prediction output, and the second language text corresponding to the first language text in the bilingual sentence pair, adjust the parameters of the compression translation model according to a preset knowledge distillation strategy until a preset convergence condition is met, thereby completing the training.

[0076] In this embodiment, taking the bilingual sentence pair in which the first language text is English text and the second language text is English text as an example, the English text is simultaneously input into the pre-trained original translation model and the compressed translation model during training to obtain the first predicted Chinese output by the pre-trained original translation model and the second predicted Chinese output by the compressed model. Based on the prediction results of the pre-trained original translation model and the compressed translation model and the real Chinese text, the parameters of the compressed translation model are adjusted according to the preset knowledge distillation strategy, that is, the teacher model transfers its own knowledge to the student model according to the corresponding knowledge distillation strategy and adjusts the learning objectives of the student model, so that it continuously optimizes the parameters of the model until the preset convergence conditions are met, such as the number of training times reaches the preset number of times or the model loss is less than the preset value, etc., and the training of the compressed translation model is completed.

[0077] The specific knowledge distillation strategy can be flexibly selected according to the needs, such as knowledge distillation based on soft labels: in the training of the student model, the teacher model is used to predict the training data, and the prediction results are used as soft labels. The goal of the student model is to minimize the prediction error between its prediction results and the real labeled data and the difference between the prediction results of the teacher model. Attention-based knowledge distillation: by introducing the attention mechanism, the student model can focus on the key features in the teacher model's prediction. By allowing the student model to learn the attention weights of the teacher model, the performance of the student model can be improved. Knowledge distillation based on model structure: in addition to transferring prediction results, knowledge distillation can also be performed by transferring model structure. Certain layers or part of the network structure of the teacher model can be directly copied to the student model, so that the student model can better simulate the behavior of the teacher model.

[0078] In one embodiment, Figure 3 As shown, step S202 includes:

[0079] S2021. Calculate the soft label loss of the compression translation model according to the first prediction output and the second prediction output;

[0080] S2022. Calculate the hard label loss of the compression translation model according to the second prediction output and the second language text;

[0081] S2023. Calculate a total loss according to the soft label loss and the hard label loss, and adjust the parameters of the compression translation model by back-propagation according to the total loss.

[0082] In this embodiment, the compression translation model is trained by knowledge distillation based on soft labels to adjust its parameters. Specifically, the soft label loss of the compression translation model is calculated according to the first prediction output and the second prediction output. The difference between the prediction result output by the compression model and the prediction result output by the original translation model is measured by the soft label loss. Specifically, the formula The Kullback-Leibler (KL) divergence is calculated as the soft label loss, where is the first predicted output of the pre-trained original translation model for the i-th word, is the second predicted output of the compression translation model for the i-th word; and the hard label loss of the compression translation model is calculated based on the second predicted output and the second language text, and the difference between the predicted output of the compression model and the real text is measured by the hard label loss, for example, by the formula Calculate the L2 loss as the hard label loss, where x c For second language texts, is the second predicted output of the compressed translation model; the total loss is the sum of these two parts, i.e. The parameters of the compression translation model are adjusted by back-propagation of the total loss value until the performance of the compression model is optimal, so that the compression model after binary quantization can achieve performance close to that of the original translation model under limited computing resources.

[0083] In one embodiment, Figure 4 As shown, the binary quantization processing of the original translation model includes:

[0084] S400, performing binarization processing on the floating point numbers in the original translation model;

[0085] S500, performing a scaling operation on the binary floating point number by using a scaling factor or a layer normalization function;

[0086] S600, adding a residual operation to the binary attention layer to obtain a binary quantized compressed translation model.

[0087] In this embodiment, the original translation model is binary quantized to compress the model size as much as possible, reduce the memory usage of the model parameters, and also reduce the communication overhead of the model, so that it can be used in edge computing application scenarios.

[0088] First, the floating-point numbers in the original translation model are binarized, and the floating-point numbers in the original translation model, including weights and activation values, are converted into binary form, such as {-1, +1} or {0, 1}, etc. Specifically, the following formula is used:

[0089] clip(x,x min ,x max ):=min(x max ,max(x min ,x))

[0090]

[0091] Binarize the floating point numbers in the original translation model, where x is the input floating point number and x min is the minimum value in x, x max is the maximum value in x; x b is the result of binarization of x, ε is a floating point number, B is the binarization limit, clip is the clipping function, and floor is the floor operation. That is, clipping is performed through the clip function to limit the floating point value to the range of (-1+ε,1-∈), ε is a very small floating point number to prevent overflow during rounding, and the clipped value is mapped to the set of {-1,+1} through the floor operation, thereby mapping the floating point number to the binary space.

[0092] Then, the binarized weights and activation values ​​are scaled to solve the problem of dot product variance expansion caused by binarization. Specifically, the scaling process can be achieved through a fixed scaling factor or layer normalization function. For the score calculation in the dot product attention mechanism, the Dense Layer can be scaled and adjusted by a fixed scaling factor. The adjustment formula is: The scaling factor s is a hyperparameter used to scale the result after binarization to compensate for the variance change caused by binarization. Dense(Ab) is a dense layer that uses binarized weights and activation values ​​for calculations. b is the binary activation value in this layer, the activation function is ReLU, W b And the binary weight parameters that need to be trained in this layer; for the fully connected layer (Feed Forward Network, FFN), the scaling operation can be performed through the LayerNorm function, that is, the layer normalization function, specifically:

[0093] FFN(A)=LN(LN(max(0,A b W 1b +b1)) b ·W 2b +b2)

[0094] The input of this layer

[0095] Where γ and β are learnable parameters. LN(z) is the LayerNorm function, W 1b , b1, W 2b and b2 are the parameters that need to be trained for this layer. Ε[z] and Var(z) are the mean and variance of the input z of the LN function. ε is a specific small constant, generally set to 0.0000001, to avoid the denominator being 0.

[0096] The LayerNorm function is a more dynamic scaling solution that normalizes and scales the output of each layer through learnable parameters γ and β. Since FFN contains nonlinear activations, its output distribution is more complex. The LayerNorm function can automatically adapt to changes in the training process and learn the best scaling factor without manually adjusting the scaling factor, so that LayerNorm can dynamically adjust the output of FFN to have a stable mean and variance. Of course, in other embodiments, the LayerNorm function can be used to replace the fixed scaling factor to scale all layers, which can recover losses faster after binarization, reduce the workload of hyperparameter adjustment, and improve the performance of the model.

[0097] Then the binarized model architecture is adjusted, specifically by adding residual operations in the binarized attention layer to improve the optimization problem in deep network training and help the gradient flow more efficiently, thereby improving the training stability and performance of the model. Figure 5 As shown, the residual operation is added to the attention layer through the following formula:

[0098] K b =LN(max(0,xW 11b +b 11 )) b

[0099] Q b =LN(max(0,xW 12b +b 12 )) b

[0100] V b =LN(max(0,xW 13b +b 13 )) b

[0101]

[0102] P=LN(max(0,UW 2b +b2)) b +U

[0103] Among them, Q b , K b 、V b They are query, key, and value matrices respectively. is the dimension of the key, U is the output of the attention layer (Self-Attention), and the output of the attention layer is used as input. After Linear (matrix multiplication) and LayerNorm operations are performed on the input, the obtained output is added to the original input. By adding a residual connection after the output linear projection layer of the attention layer and combining the use of LayerNorm, the compressed translation model after binary quantization can more effectively handle the optimization problem in the binarization process, thereby maintaining the model performance and improving the convergence speed.

[0104] It should be noted that there is not necessarily a certain order between the above-mentioned steps. A person skilled in the art can understand, based on the description of the embodiments of the present invention, that in different embodiments, the above-mentioned steps may have different execution orders, that is, they may be executed in parallel, may be executed interchangeably, and so on.

[0105] Another embodiment of the present invention provides a machine translation method, such as Figure 6 As shown, the method comprises the following steps:

[0106] S700, obtaining a source language sentence to be translated;

[0107] S800, inputting the source language sentence to be translated into a binary quantization translation model, and outputting a target language sentence corresponding to the source language sentence;

[0108] The binary quantization translation model is obtained by using the binary quantization translation model generation method as described above.

[0109] In this embodiment, after the original translation model is binarized by the generation method as described above and the binary quantization translation model is trained by knowledge distillation, the binary quantization translation model can be deployed in the edge device in the edge computing application scenario. The user can directly input the source language sentence to be translated into the edge device, and the binary quantization translation model deployed in the edge device outputs the target language sentence corresponding to the source language sentence, thereby realizing an efficient and low-deployment cost edge translation application.

[0110] Another embodiment of the present invention provides a device for generating a binary quantized translation model, such as Figure 7 As shown, the training device 1 comprises:

[0111] A third aspect of the present invention provides a device for generating a binary quantization translation model, comprising:

[0112] A collection and construction module 11 is used to collect sample data in multiple languages ​​and construct a plurality of bilingual sentence pairs according to the sample data in multiple languages, wherein the bilingual sentence pairs include two sentence texts translated from each other;

[0113] A training module 12 uses the bilingual sentence pairs as training samples and trains the pre-trained original translation model and the compressed translation model through knowledge distillation, wherein the compressed translation model is obtained by performing binary quantization processing on the original translation model;

[0114] A deployment module 13 is generated, which is used to use the trained compression translation model as a binary quantization translation model that can be deployed to an edge device, wherein the binary quantization translation model is used to convert a given source language sentence into a target language sentence.

[0115] The module referred to in the present invention refers to a series of computer program instruction segments that can complete specific functions, which is more suitable for describing the generation and execution process of the binary quantization translation model than a program. For the specific implementation of each module, please refer to the corresponding method embodiment above, which will not be repeated here.

[0116] In one embodiment, the training module 12 includes:

[0117] An input unit, used to input the first language text in the bilingual sentence pair into the pre-trained original translation model and the compressed translation model at the same time, to obtain a first prediction output of the pre-trained original translation model and a second prediction output of the compressed translation model;

[0118] A training unit is used to adjust the parameters of the compression translation model according to the first prediction output, the second prediction output and the second language text corresponding to the first language text in the bilingual sentence pair according to a preset knowledge distillation strategy until the training is completed when a preset convergence condition is met.

[0119] In one embodiment, the training unit comprises:

[0120] A first loss unit, configured to calculate a soft label loss of the compression translation model according to the first prediction output and the second prediction output;

[0121] A second loss unit, configured to calculate a hard label loss of the compression translation model according to the second prediction output and the second language text;

[0122] A parameter updating unit is used to calculate a total loss according to the soft label loss and the hard label loss, and adjust the parameters of the compression translation model according to the total loss by back-propagation.

[0123] In one embodiment, the apparatus 1 further comprises:

[0124] A binarization module, used for binarizing the floating point numbers in the original translation model;

[0125] A scaling module is used to scale the binary floating point numbers by a scaling factor or a layer normalization function;

[0126] The residual module is used to add a residual operation to the binary attention layer to obtain a binary quantized compressed translation model.

[0127] In one embodiment, the binarization module is specifically used for:

[0128] By the formula clip(x,x min ,x max ):=min(x max ,max(x min ,x))

[0129]

[0130] Binarize the floating point numbers in the original translation model, where x is the input floating point number, x min is the minimum value in x, x max is the maximum value in x; x b is the result of binarization of x, ε is a floating point number, B is the binarization limit, clip is the clipping function, and floor is the rounding down operation.

[0131] A second aspect of the present invention provides a machine translation method, comprising:

[0132] Obtain source language sentences to be translated;

[0133] Inputting the source language sentence to be translated into a binary quantization translation model, and outputting a target language sentence corresponding to the source language sentence;

[0134] The binary quantization translation model is obtained by using the binary quantization translation model generation method as described above.

[0135] Another embodiment of the present invention provides a machine translation device, such as Figure 8 As shown, the device 2 comprises:

[0136] An acquisition module 21 is used to acquire a source language sentence to be translated;

[0137] A translation module 22, configured to input the source language sentence to be translated into a binary quantization translation model, and output a target language sentence corresponding to the source language sentence;

[0138] The binary quantization translation model is obtained by using the binary quantization translation model generation method as described above.

[0139] The acquisition module 21 is connected to the translation module 22. The module referred to in the present invention refers to a series of computer program instruction segments that can complete specific functions, which is more suitable for describing the address recognition execution process than the program. For the specific implementation of each module, please refer to the corresponding method embodiment above, which will not be repeated here.

[0140] Another embodiment of the present invention provides an electronic device, such as Fig. 9 As shown, the electronic device 10 includes:

[0141] One or more processors 110 and memory 120, Fig. 9 A processor 110 is used as an example for description. The processor 110 and the memory 120 may be connected via a bus or other means. Fig. 9 The example of connecting through bus is taken in the following.

[0142] The processor 110 is used to complete various control logics of the electronic device 10. It can be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA), a single-chip microcomputer, an ARM (Acorn RISC Machine) or other programmable logic device, a discrete gate or transistor logic, a discrete hardware component, or any combination of these components. In addition, the processor 110 can also be any traditional processor, microprocessor or state machine. The processor 110 can also be implemented as a combination of computing devices, for example, a combination of a DSP and a microprocessor, a plurality of microprocessors, one or more microprocessors combined with a DSP and / or any other such configuration.

[0143] The memory 120, as a non-volatile computer-readable storage medium, can be used to store non-volatile software programs, non-volatile computer executable programs and modules, such as program instructions corresponding to the method for generating a binary quantization translation model in the embodiment of the present invention. The processor 110 executes various functional applications and data processing of the electronic device 10 by running the non-volatile software programs, instructions and units stored in the memory 120, that is, implements the method for generating a binary quantization translation model or the machine translation method in the above method embodiment.

[0144] The memory 120 may include a program storage area and a data storage area, wherein the program storage area may store an operating system and applications required for at least one function; the data storage area may store data created according to the use of the electronic device 10, etc. In addition, the memory 120 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage device. In some embodiments, the memory 120 may optionally include a memory remotely arranged relative to the processor 110, and these remote memories may be connected to the electronic device 10 via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0145] One or more units are stored in the memory 120, and when executed by one or more processors 110, the method for generating a binary quantization translation model in any of the above method embodiments is executed, for example, the following steps are executed:

[0146] Collecting sample data in multiple languages, and constructing a plurality of bilingual sentence pairs according to the sample data in the multiple languages, wherein the bilingual sentence pairs include two sentence texts translated from each other;

[0147] The bilingual sentence pairs are used as training samples, and a pre-trained original translation model and a compressed translation model are trained through knowledge distillation, wherein the compressed translation model is obtained by performing binary quantization processing on the original translation model;

[0148] The trained compression translation model is used as a binary quantization translation model that can be deployed to an edge device. The binary quantization translation model is used to convert a given source language sentence into a target language sentence.

[0149] Alternatively, the machine translation method in any of the above embodiments is executed, for example, the following steps are executed:

[0150] Obtain source language sentences to be translated;

[0151] Inputting the source language sentence to be translated into a binary quantization translation model, and outputting a target language sentence corresponding to the source language sentence;

[0152] The binary quantization translation model is obtained by using the binary quantization translation model generation method as described above.

[0153] The embodiment of the present invention further provides a non-volatile computer-readable storage medium, which stores computer-executable instructions. When the computer-executable instructions are executed by one or more processors, the method for generating a binary quantization translation model in any of the above method embodiments is executed, for example, the following steps are performed:

[0154] Collecting sample data in multiple languages, and constructing a plurality of bilingual sentence pairs according to the sample data in the multiple languages, wherein the bilingual sentence pairs include two sentence texts translated from each other;

[0155] The bilingual sentence pairs are used as training samples, and a pre-trained original translation model and a compressed translation model are trained through knowledge distillation, wherein the compressed translation model is obtained by performing binary quantization processing on the original translation model;

[0156] The trained compression translation model is used as a binary quantization translation model that can be deployed to an edge device. The binary quantization translation model is used to convert a given source language sentence into a target language sentence.

[0157] Alternatively, the machine translation method in any of the above embodiments is executed, for example, the following steps are executed:

[0158] Obtain source language sentences to be translated;

[0159] Inputting the source language sentence to be translated into a binary quantization translation model, and outputting a target language sentence corresponding to the source language sentence;

[0160] The binary quantization translation model is obtained by using the binary quantization translation model generation method as described above.

[0161] As an example, non-volatile storage media can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) as external cache memory. By way of illustration and not limitation, RAM can be obtained in many forms such as synchronous RAM (SRAM), dynamic RAM, (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), Synchlink DRAM (SLDRAM), and direct Rambus RAM (DRRAM). The disclosed memory components or memories of the operating environment described herein are intended to include one or more of these and / or any other suitable types of memory.

[0162] In summary, the present invention discloses a method for generating a binary quantization translation model, a machine translation method, and related equipment. The generation method includes: collecting sample data in multiple languages, constructing multiple bilingual sentence pairs based on the sample data in multiple languages, wherein the bilingual sentence pairs include two sentence texts that are translated into each other; using the bilingual sentence pairs as training samples, and training the pre-trained original translation model and the compressed translation model through knowledge distillation, wherein the compressed translation model is obtained by binary quantization of the original translation model; using the trained compressed translation model as a binary quantization translation model that can be deployed to an edge device, wherein the binary quantization translation model is used to convert a given source language sentence into a target language sentence. The size of the translation model is effectively reduced through binary quantization, and the model is trained through knowledge distillation to obtain a binary quantization translation model that can adapt to the edge device configuration and has good translation performance, so that the translation model can be successfully deployed on the edge computing end, thereby improving translation efficiency.

[0163] Of course, those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing related hardware (such as a processor, a controller, etc.) through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium, and the computer program can include the processes of the above-mentioned method embodiments when executed. The storage medium can be a memory, a disk, a floppy disk, a flash memory, an optical storage device, etc.

[0164] It should be understood that the application of the present invention is not limited to the above examples. For ordinary technicians in this field, improvements or changes can be made based on the above description. All these improvements and changes should fall within the scope of protection of the claims attached to the present invention.

Claims

1. A method for generating a binary quantized translation model, characterized in that: include: Collecting sample data in multiple languages, and constructing a plurality of bilingual sentence pairs according to the sample data in the multiple languages, wherein the bilingual sentence pairs include two sentence texts translated from each other; The bilingual sentence pairs are used as training samples, and a pre-trained original translation model and a compressed translation model are trained through knowledge distillation, wherein the compressed translation model is obtained by performing binary quantization processing on the original translation model; The trained compression translation model is used as a binary quantization translation model that can be deployed to an edge device. The binary quantization translation model is used to convert a given source language sentence into a target language sentence.

2. The method for generating a binary quantization translation model according to claim 1, characterized in that: The method of using the bilingual sentence pairs as training samples and training the pre-trained original translation model and the compressed translation model through knowledge distillation includes: Inputting the first language text in the bilingual sentence pair into a pre-trained original translation model and a compressed translation model simultaneously, obtaining a first prediction output of the pre-trained original translation model and a second prediction output of the compressed translation model; According to the first prediction output, the second prediction output and the second language text corresponding to the first language text in the bilingual sentence pair, the parameters of the compression translation model are adjusted according to a preset knowledge distillation strategy until a preset convergence condition is met, and the training is completed.

3. The method for generating a binary quantized translation model according to claim 2, characterized in that: The adjusting the parameters of the compression translation model according to the first prediction output, the second prediction output, and the second language text corresponding to the first language text in the bilingual sentence pair according to a preset knowledge distillation strategy includes: Calculating a soft label loss of the compression translation model according to the first prediction output and the second prediction output; Calculating a hard label loss of the compression translation model based on the second prediction output and the second language text; A total loss is calculated according to the soft label loss and the hard label loss, and parameters of the compression translation model are adjusted by back-propagation according to the total loss.

4. The method for generating a binary quantized translation model according to any one of claims 1 to 3, characterized in that: The binary quantization processing of the original translation model includes: Binarizing the floating point numbers in the original translation model; Scaling the binary floating point numbers by a scaling factor or layer normalization function; Adding a residual operation to the binary attention layer results in a binary quantized compressed translation model.

5. The method for generating a binary quantization translation model according to claim 4, characterized in that: The binarization of the floating point numbers in the original translation model specifically includes: By the formula clip(x,x min ,x max ):=min(x max ,max(x min ,x)) Binarize the floating point numbers in the original translation model, where x is the input floating point number, x min is the minimum value in x, x max is the maximum value in x; x b is the result of binarization of x, ε is a floating point number, B is the binarization limit, clip is the clipping function, and floor is the rounding down operation.

6. A machine translation method, characterized in that: include: Obtain source language sentences to be translated; Inputting the source language sentence to be translated into a binary quantization translation model, and outputting a target language sentence corresponding to the source language sentence; The binary quantization translation model is obtained by using the method for generating a binary quantization translation model as described in any one of claims 1 to 5.

7. A device for generating a binary quantization translation model, characterized in that: include: A collection and construction module, used for collecting sample data in multiple languages, and constructing a plurality of bilingual sentence pairs according to the sample data in the multiple languages, wherein the bilingual sentence pairs include two sentence texts translated from each other; A training module, taking the bilingual sentence pairs as training samples, and training the pre-trained original translation model and the compressed translation model through knowledge distillation, wherein the compressed translation model is obtained by performing binary quantization processing on the original translation model; A deployment module is generated, which is used to use the trained compression translation model as a binary quantization translation model that can be deployed to an edge device, wherein the binary quantization translation model is used to convert a given source language sentence into a target language sentence.

8. A machine translation device, characterized in that: include: An acquisition module, used to acquire source language sentences to be translated; A translation module, used for inputting the source language sentence to be translated into a binary quantization translation model, and outputting a target language sentence corresponding to the source language sentence; The binary quantization translation model is obtained by using the method for generating a binary quantization translation model as described in any one of claims 1 to 5.

9. An electronic device, characterized in that: comprising at least one processor; and, a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the method for generating a binary quantization translation model as described in any one of claims 1 to 5, or execute the machine translation method as described in claim 6.

10. A non-volatile computer-readable storage medium, characterized in that: The non-volatile computer-readable storage medium stores computer-executable instructions, which, when executed by one or more processors, enable the one or more processors to execute the method for generating a binary quantization translation model as described in any one of claims 1 to 5, or to execute the machine translation method as described in claim 6.

Images