Method and Storage Medium for Automatically Judging the Timing of Retrieving a Machine Translation Database

Abstract

The present invention discloses a method for automatically determining the timing of retrieving a machine translation database. The method comprises the following steps: dividing the knowledge in the database into mastered knowledge and unmastered knowledge; determining knowledge boundary values for different knowledge according to the mastery of knowledge within the local space in the database; using the knowledge boundary value situation of the knowledge in the database as training data to train a lightweight multi-layer perceptron network model, and predicting the knowledge boundary value situation corresponding to the hidden layer representation through the multi-layer perceptron network model; when decoding the text in the target translation domain, after obtaining the hidden layer representation of the translation context using a general domain translation model, determining whether the general domain translation model needs to retrieve the database at each decoding step according to the knowledge boundary value situation. The present invention avoids unnecessary retrieval actions of the general domain translation model at each decoding step and can automatically determine the retrieval timing when applying databases in different languages and domains.

Description

Technical Field

[0001] The present invention relates to the technical field of natural language processing, and particularly to a method and a storage medium for automatically determining the timing of retrieving a machine translation database. Background Art

[0002] Domain adaptation is an important topic in Neural Machine Translation (NMT). Its purpose is to adapt a general domain translation model to the target domain so that it can handle translation tasks in the target domain.

[0003] Recently, the retrieval-based kNN-MT method has become a new paradigm for domain adaptation. Different from the traditional fine-tune method, this framework can quickly complete domain adaptation without updating the parameters of the general domain translation model, greatly alleviating the problem of catastrophic forgetting.

[0004] Specifically, the kNN-MT method is divided into two steps: building a database and applying the database. In the database building stage, the method extracts translation knowledge from the bilingual data in the target domain and saves this knowledge into a database in the form of key-value pairs. Each entry in the database is a key-value pair, where the key is the hidden layer representation of the translation context, and the value is the target word that should be generated according to the translation context. In the database application stage, the hidden layer representation of the translation context at each decoding step is used to retrieve the database. The output distribution p kNN (y t |x,y ＜t ) generated by the retrieval result and the output distribution p MT (y t |x,y ＜t ) of the general domain translation model are aggregated according to the aggregation coefficient λ:

[0005] p kNNMT (y t |x,y ＜t )=λp kNN (y t |x,y ＜t )+(1-λ)p MT (y t |x,y ＜t )

[0006] , that is, a final output distribution p kNNMT (y t |x,y ＜t ) that may be closer to the target domain is obtained.

[0007] However, the existing process of applying the database is very unreasonable. The general domain translation model is required to retrieve the database at each decoding step, which will cause huge computational overhead. In addition, the existing process of applying the database also ignores the interpretability of the retrieval behavior. Ideally, the relevant knowledge in the database needs to be retrieved only when the general domain translation model is insufficient.

[0008] To automatically determine the retrieval timing, researchers have made some preliminary attempts. Currently, there is one method for determining the retrieval timing: this method proposes to dynamically determine the aggregation parameter λ and use the aggregation parameter to judge whether it is the retrieval timing. Specifically, this paper no longer treats λ as a fixed hyperparameter, but trains a multi-layer perceptron network to predict the aggregation parameter λ, and determines different aggregation parameters for different decoding positions. After predicting the aggregation parameter, the retrieval timing is judged by comparing the size relationship between λ and the preset threshold. The database is retrieved only when λ is greater than the threshold; otherwise, the retrieval behavior will be abandoned.

[0009] However, the problem with this solution is that the generalization effect of the prediction network obtained by this training method is poor, and it cannot effectively judge the retrieval timing during testing, and the retrieval timing judgment effect is very poor. Summary of the Invention

[0010] To overcome the shortcomings in the prior art of the above background technology, namely, the generalization ability of the prediction network is insufficient, it cannot effectively judge the retrieval timing during testing, and the retrieval timing judgment effect is very poor. The method and storage medium for automatically judging the retrieval timing of the machine translation database of the present invention can automatically judge the retrieval timing, abandon unnecessary retrieval behaviors, and make the retrieval behavior more interpretable.

[0011] To achieve the above objectives, the present invention adopts the following technical solutions:

[0012] In the first aspect of the present invention, a method for automatically judging the retrieval timing of a machine translation database is provided, including the following steps:

[0013] S1: Divide the knowledge in the database into mastered knowledge and unmastered knowledge;

[0014] S2: Determine knowledge boundary values for different knowledge according to the mastery of knowledge in the local space of the database;

[0015] S3: Use the knowledge boundary value situation of the knowledge in the database as training data to train a lightweight multi-layer perceptron network model, and predict the knowledge boundary value situation corresponding to the hidden layer representation through the multi-layer perceptron network model;

[0016] S4: When decoding the text in the target domain of translation, after obtaining the hidden layer representation of the translation context using the general domain translation model, predict the knowledge boundary value situation of the hidden layer representation according to the multi-layer perceptron network model, and then determine whether the general domain translation model needs to retrieve the database at each decoding step.

[0017] In some possible implementation manners, during the construction process of the database, the translation knowledge extracted from the parallel data is saved. A piece of translation knowledge is a key-value pair. The key is the hidden layer representation h(x ＜t ,y ＜t ) of the translation context (x ＜t ,y ＜t ), and the value is the target word y t that should be generated according to the translation context; during the construction process of the database, the predicted word output by the general domain translation model according to the translation context is also saved.

[0018] In some possible implementation manners, in S1, "dividing the knowledge in the database into mastered knowledge and unmastered knowledge" is specifically as follows:

[0019] By comparing the consistency between the target word y t and the predicted word to determine whether the knowledge is mastered. When the target word y t is consistent with the predicted word , it is considered that the corresponding piece of knowledge is the knowledge known mastered by the general domain translation model; when the target word y t is inconsistent with the predicted word , it is considered that the corresponding piece of knowledge is the knowledge unknown not mastered by the general domain translation model, that is:

[0020]

[0021] In some possible implementation manners, in S2, "determining the knowledge boundary value for different knowledge according to the mastery situation of the knowledge in the local space of the database" specifically includes the following steps:

[0022] S21: Determine the local space. The local space N k is the k-nearest neighbor of a certain knowledge (key, val) in the database, specifically expressed as:

[0023]

[0024] where j represents the j-th nearest neighbor knowledge, and the value of j is between 1 and k, its key is key j , and its value is val j ;

[0025] S22: Determine the knowledge boundary value km(key) of the knowledge with the hidden layer representation as key according to the mastery of the knowledge in the local space. The knowledge boundary value is specifically expressed as:

[0026]

[0027] where i represents the number of knowledge in the local space.

[0028] In some possible implementation manners, "predict the knowledge boundary value situation corresponding to the hidden layer representation through the multi-layer perceptron network model" in S3 is specifically as follows: The multi-layer perceptron network model predicts the size situation of the knowledge boundary value corresponding to the hidden layer representation according to the hidden layer representation, that is, whether the knowledge boundary value is greater than a certain preset threshold.

[0029] In some possible implementation manners, "when decoding and translating the text in the target domain, after obtaining the hidden layer representation of the translation context using the general domain translation model, predict the knowledge boundary value situation of the hidden layer representation according to the multi-layer perceptron network model, and then determine whether the general domain translation model needs to retrieve the database at each decoding step" in S4 is specifically as follows: At each decoding step when translating the text in the target domain, after obtaining the hidden layer representation of the translation context using the general domain translation model, the multi-layer perceptron network model will judge the corresponding knowledge boundary value situation according to the hidden layer representation. If the knowledge boundary value is greater than a certain preset threshold, it means that the general domain translation model has strong translation ability at the hidden layer representation, and the retrieval behavior can be cancelled; if the knowledge boundary value is less than a certain preset threshold, it means that the general domain translation model has weak translation ability at the hidden layer representation, and the retrieval behavior needs to be retained.

[0030] In a second aspect of the present invention, there is provided a computer-readable storage medium for storing program codes, and when the program codes are executed by a processor, the steps of the method for automatically determining the timing of retrieving the machine translation database are implemented.

[0031] In a third aspect of the present invention, there is provided an electronic device, and the electronic device includes a processor and a memory: The memory is used for storing program codes and transmitting the program codes to the processor; The processor is used for executing the steps of the method for automatically determining the timing of retrieving the machine translation database according to the instructions in the program codes.

[0032] The beneficial effects of the present invention are as follows:

[0033] 1. Technically, the present invention divides the knowledge in the database into mastered knowledge and unmastered knowledge; determines knowledge boundary values for different knowledge according to the mastery of knowledge in the local space of the database; uses the knowledge boundary value situation of the knowledge in the database as training data to train a lightweight multi-layer perceptron network model, and predicts the knowledge boundary value situation corresponding to the hidden layer representation through the multi-layer perceptron network model; when decoding and translating the text in the target domain, after obtaining the hidden layer representation of the translation context using the general domain translation model, according to the knowledge boundary value situation, it is determined whether the general domain translation model needs to retrieve the database at each decoding step. This avoids unnecessary retrieval behaviors of the general domain translation model at each decoding step, saves computational overhead, and improves the decoding speed.

[0034] 2. From the application level, the method for automatically determining the timing of retrieving the machine translation database in the present invention is easy to reproduce and has a simple process; it has high compatibility, and the multi-layer perceptron network model used to determine the retrieval timing can be used in any kNN-MT framework; it has a wide range of adaptability, and can automatically determine the retrieval timing when applying databases in different languages and different fields. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 is the overall step flow chart of the method for automatically determining the timing of retrieving the machine translation database in the embodiment of the present invention.

[0036] Figure 2 is an example diagram of the database in the embodiment of the present invention;

[0037] Figure 3 is an example diagram for determining whether the knowledge in the database is mastered by the general domain translation model in the embodiment of the present invention;

[0038] Figure 4 is the working flow chart of the multi-layer perceptron network model being used as the retrieval timing judgment module in the embodiment of the present invention;

[0039] Figure 5 is an example diagram for giving up part of the retrieval behavior according to the retrieval timing judgment situation in the embodiment of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0040] The following elaborates on the preferred embodiments of the present invention in detail with reference to the accompanying drawings, so that the advantages and features of the present invention can be more easily understood by those skilled in the art, thereby making the protection scope of the present invention more clearly defined.

[0041] The present invention provides an embodiment of a method for automatically determining the timing of retrieving a machine translation database. Referring to the attached Figure 1 figures, the method includes the following steps:

[0042] S1: Divide the knowledge in the database into mastered knowledge and unmastered knowledge.

[0043] Refer to the appendix Figure 2 as shown Figure 2 in the figure, which is an example diagram of the database; during the construction process, the database stores the translation knowledge extracted from the parallel data. One piece of translation knowledge is a key-value pair, where the key is the hidden layer representation h(x ＜t , y ＜t ) of the translation context (x ＜t , y ＜t ), and the value is the target word y that should be generated according to the translation context t . For example, from the German-English parallel data: (Wie wirkt Penizillin?, How does Penicillin work?), such a piece of translation knowledge can be extracted: among them, the key is the hidden layer representation h(Wie wirkt Penizillin?, <bos>How), the value is the target word does.

[0044] Refer to the appendix Figure 3 As shown, in order to determine whether each piece of knowledge in the database is mastered by the general domain translation model (NMT model), during the process of constructing the database, the predicted words output by the general domain translation model (NMT model) according to the translation context are also saved For example, for the translation context: (Wie wirkt Penizillin?, <bos>How?) The predicted word of the general domain translation model (NMT model) is "does". By comparing the target word y t and the predicted word consistency can be used to determine whether the knowledge is mastered. When the two are consistent, this piece of knowledge can be considered as the knowledge known by the general domain translation model (NMT model); when the two are inconsistent, this piece of knowledge can be considered as the knowledge unknown by the general domain translation model (NMT model):

[0045]

[0046] For example, for the translation knowledge (h(Wie wirkt Penizillin? <bos>How),does),If the target word and the predicted word are both "does", it is considered that this translation knowledge is mastered by the general-domain translation model (NMT model).

[0047] S2: Determine knowledge boundary values for different pieces of knowledge according to the mastery of knowledge within the local space in the database; in this embodiment, knowledge boundary values are proposed to describe the mastery of knowledge within the local space in the database, and the knowledge boundary values are used to measure the translation ability of the general-domain translation model (NMT model). The mastery of knowledge within the local space refers to the mastery of knowledge within the local space. The step of "determining knowledge boundary values for different pieces of knowledge according to the mastery of knowledge within the local space in the database" specifically includes the following steps:

[0048] S21: Determine the local space, where the local space N k is the k-nearest neighbor of a piece of knowledge (key, val) in the database, specifically represented as:

[0049]

[0050] where j represents the j-th nearest neighboring knowledge, the value of j ranges from 1 to k, and its key is key j and its value is val j .

[0051] S22: Determine the knowledge boundary value km(key) of the knowledge with the hidden layer representation as key according to the mastery of knowledge within the local space. The knowledge boundary value is specifically represented as:

[0052]

[0053] where i represents the number of pieces of knowledge in the local space.

[0054] For example, the translation knowledge (h(Wie wirkt Penizillin?, <bos>How),does)If the four nearest neighboring pieces of knowledge around are all knowledge mastered by the general domain translation model (NMT model), and there is unmastered knowledge among the farther neighbors, then the knowledge boundary value of this knowledge is 4.

[0055] S3: Use the knowledge boundary value situation of the knowledge in the database as training data to train a lightweight multi-layer perceptron network model. The multi-layer perceptron network model predicts the size situation of the corresponding knowledge boundary value at this hidden layer representation according to the hidden layer representation, that is, whether the knowledge boundary value is greater than a certain preset threshold.

[0056] Refer to the appendix Figure 4 As shown, the multi-layer perceptron network model predicts the knowledge boundary value situation of the general domain translation model (NMT model) at the hidden layer representation. The multi-layer perceptron network model sequentially includes from the input end to the output end: a normalization layer (Norm), a forward propagation layer (Feed Forward), a linear layer (Linear), and a softmax layer (Softmax). The normalization layer is used to perform a normalization operation on the input, the forward propagation layer and the linear layer are responsible for performing non-linear transformations, and the softmax layer is used to calculate the probability distribution as the output. Applied to this embodiment, the specific calculation process of this multi-layer perceptron network model is: input the hidden layer representation h(x,y ＜t )(For simplicity, represented by h in the following formulas), the multi-layer perceptron network model will respectively output the probability that the knowledge boundary is greater than the preset threshold t ＜t ) at this hidden layer representation h(x,y p and the probability that the knowledge boundary is less than the preset threshold t p , specifically expressed as:

[0057] [p(km≥t p ), p(km<t p )] =

[0058] softmaax(W3·ReLU(W2·ReLU(W1·h)))

[0059] Where W1 and W2 are the projection matrices in the forward propagation layer, W3 is the projection matrix in the linear layer, and ReLU is the activation function used in the forward propagation layer and the linear layer.

[0060] After training, the multi-layer perceptron network model can predict the size situation of the knowledge boundary at this hidden layer representation of the knowledge in the database, that is, whether the knowledge boundary value is greater than the preset threshold. When it is greater than the preset threshold, it indicates that the translation ability of the general domain translation model (NMT model) at this hidden layer representation is strong; when it is less than the preset threshold, it indicates that the translation ability of the general domain translation model (NMT model) at this hidden layer representation is weak.

[0061] For example, the hidden layer representation of the translation context h(Wie wirkt Penizillin?, <bos>Input the multi-layer perceptron network, and the multi-layer perceptron network will predict that the knowledge boundary value corresponding to the hidden layer representation is greater than the threshold value of 4 (for example, the preset threshold value is 4).

[0062] S4: When decoding the text in the target translation domain, after obtaining the hidden layer representation of the translation context using the general domain translation model, according to the knowledge boundary value situation predicted by the multi-layer perceptron network model for this hidden layer representation, further determine whether the general domain translation model needs to retrieve the database at each decoding step, specifically as follows: At each decoding step when translating the text in the target translation domain, after obtaining the hidden layer representation of the translation context using the general domain translation model, the multi-layer perceptron network model will judge the knowledge boundary value situation corresponding to this hidden layer representation. If the knowledge boundary value is greater than a certain preset threshold, it means that the general domain translation model has strong translation ability at this hidden layer representation, and the retrieval behavior can be cancelled; if the knowledge boundary value is less than a certain preset threshold, it means that the general domain translation model has weak translation ability at this hidden layer representation, and the retrieval behavior needs to be retained.

[0063] Refer to the appendix Figure 4 As shown, during the process of the general domain translation model (NMT model) decoding the text in the target translation domain, the multi-layer perceptron network model can be used as a retrieval timing judgment module. At decoding step t, the general domain translation model (NMT model) will first encode the translation context (x, y ＜t ) to obtain the hidden layer representation h(x, y ＜t ) and input it into the multi-layer perceptron network model, then it can predict whether the knowledge boundary value at this hidden layer representation is greater than the preset threshold. When the probability p(km≥t p ) of being greater than the preset threshold is larger, it indicates that the general domain translation model (NMT model) has strong translation ability at this hidden layer representation and can give up retrieval in advance; when the probability of being less than the preset threshold p(km<t p ) is larger, it indicates that the general domain translation model (NMT model) has weak translation ability at this hidden layer representation and needs to continue to perform the retrieval.

[0064] Refer to the appendix Figure 5 As shown, at the second decoding step of translating the German sentence Wie wirkt Cynokit into an English text, the multi-layer perceptron network model will be based on the hidden layer representation of the translation context:

[0065] h(Wie wirkt Cynokit?, <bos>How)

[0066] Predict that the probability p(km≥t p ) that the corresponding knowledge boundary value of this hidden layer representation is greater than 4 is 0.9 (assuming the set threshold is 4). This indicates that the general domain translation model has strong translation ability at this hidden layer representation, and the retrieval behavior at this decoding step will be cancelled.

[0067] The above general domain translation model is an NMT model.

[0068] This embodiment also provides a computer-readable storage medium, which is used to store program code, and the program code is used to execute the above method for reducing the scale of the machine translation database.

[0069] The storage medium stores program instructions that can implement all the above methods. Among them, the program instructions can be stored in the above storage medium in the form of a software product, including several instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor to execute all or part of the steps of the methods described in various embodiments of the present invention. The foregoing storage medium includes: various media that can store program code such as USB flash drives, mobile hard disks, read-only memories (ROMs), random access memories (RAMs), magnetic disks, or optical discs, or terminal devices such as computers, servers, mobile phones, and tablets.

[0070] This embodiment further provides an electronic device, which includes a processor and a memory: the memory is used to store program code and transmit the program code to the processor; the processor is used to execute the above method for reducing the scale of the machine translation database according to the instructions in the program code.

[0071] The processor can also be called a CPU (Central Processing Unit). The processor may be an integrated circuit chip with signal processing capabilities. The processor can also be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc.

[0072] The above embodiments are only for illustrating the technical concept and features of the present invention, and the purpose is to enable those who are familiar with this technology to understand the content of the present invention and implement it. It should not be used to limit the protection scope of the present invention. Any equivalent changes or modifications made according to the spirit and essence of the present invention should be covered within the protection scope of the present invention.< / bos> < / bos> < / bos> < / bos> < / bos> < / bos>

Claims

1. A method for automatically determining the timing of retrieving a machine translation database, characterized in that, It includes the following steps: S1: Divide the knowledge in the database into mastered knowledge and unmastered knowledge; S2: Determine knowledge boundary values for different knowledge according to the mastery of knowledge in the local space of the database; S3: Use the knowledge boundary value situation of the knowledge in the database as training data to train a lightweight multi-layer perceptron network model, and predict the knowledge boundary value situation corresponding to the hidden layer representation through the multi-layer perceptron network model; S4: When decoding and translating the text in the target domain, after obtaining the hidden layer representation of the translation context using the general domain translation model, predict the knowledge boundary value situation of the hidden layer representation according to the multi-layer perceptron network model, and then judge whether the general domain translation model needs to retrieve the database at each decoding step; The knowledge boundary value is specifically expressed as: Among them, i represents the number of knowledge in the local space, j represents the j-th nearest neighbor knowledge, and the value of j ranges from 1 to k, with its key being key j , and its value is val j .

2. The method for automatically determining the timing of retrieving a machine translation database according to claim 1, characterized in that The database stores the translation knowledge extracted from parallel data during the construction process. One piece of translation knowledge is a key-value pair. The key is the hidden layer representation h(x<t, y<t) of the translation context (x<t, y<t), and the value is the target word y that should be generated according to the translation context. t ; The database also stores the predicted words output by the general domain translation model according to the translation context during the construction process.

3. The method for automatically determining the timing of retrieving a machine translation database according to claim 2, wherein In S1, "divide the knowledge in the database into mastered knowledge and unmastered knowledge" is specifically as follows: By comparing the target word y t and the predicted word to determine whether the knowledge is mastered. When the target word y t is consistent with the predicted word , it is considered that the corresponding piece of knowledge is the knowledge known mastered by the general domain translation model; when the target word y t is inconsistent with the predicted word , it is considered that the corresponding piece of knowledge is the knowledge unknown not mastered by the general domain translation model, that is:

4. The method for automatically determining the timing of retrieving a machine translation database according to claim 3, characterized in that, In S2, "determine knowledge boundary values for different knowledge according to the mastery of knowledge in the local space of the database" specifically includes the following steps: S21: Determine the local space, where the local space N k is the k-nearest neighbor of a piece of knowledge (key, val) in the database, specifically expressed as: Among them, j represents the j-th nearest neighbor knowledge, where the value of j ranges from 1 to k, and its key is key j , and its value is val j ; S22: Determine the knowledge boundary value km(key) of the knowledge with the hidden layer representation as key according to the mastery of the knowledge in the local space.

5. The method for automatically determining the timing of retrieving a machine translation database according to claim 4, wherein In S3, "predict the knowledge boundary value situation corresponding to the hidden layer representation through the multi-layer perceptron network model" is specifically as follows: The multi-layer perceptron network model predicts the size situation of the knowledge boundary value corresponding to the hidden layer representation according to the hidden layer representation, that is, whether the knowledge boundary value is greater than a certain preset threshold.

6. The method for automatically determining the timing of retrieving a machine translation database according to claim 5, characterized in that, In S4, "when decoding and translating the text in the target domain, after obtaining the hidden layer representation of the translation context using the general domain translation model, predict the knowledge boundary value situation of the hidden layer representation according to the multi-layer perceptron network model, and then judge whether the general domain translation model needs to retrieve the database at each decoding step" is specifically as follows: At each decoding step when translating the text in the target domain, after obtaining the hidden layer representation of the translation context using the general domain translation model, the multi-layer perceptron network model will judge the corresponding knowledge boundary value situation according to the hidden layer representation. If the knowledge boundary value is greater than a certain preset threshold, it means that the general domain translation model has strong translation ability at this hidden layer representation and the retrieval behavior can be cancelled; if the knowledge boundary value is less than a certain preset threshold, it means that the general domain translation model has weak translation ability at this hidden layer representation and the retrieval behavior needs to be retained.

7. A computer-readable storage medium, characterized in that, The computer-readable storage medium is used to store program code, and when the program code is executed by a processor, it implements the steps of the method for automatically judging the timing of retrieving the machine translation database according to any one of claims 1-6.

8. An electronic device, characterized in that, The electronic device includes a processor and a memory: The memory is used to store program code and transmit the program code to the processor; The processor is used to execute the steps of the method for automatically judging the timing of retrieving the machine translation database according to any one of claims 1-6 according to the instructions in the program code.

Images