A neural network machine translation method and device

By fusion processing of encoding and decoding multiple source-end sequences, the problem of inconsistency in translation results caused by individual processing of source statements in the prior art is solved, and the accuracy and coherence of neural network machine translation are improved.

CN114065776BActive Publication Date: 2024-11-12ALIBABA GROUP HOLDING LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202010750886.7
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2020-07-30
Publication Date
2024-11-12
Estimated Expiration
2040-07-30

AI Technical Summary

Technical Problem

In the existing neural network machine translation methods, when using the Transformer model, processing each source statement alone results in low accuracy of translation results and lack of contextual correlation between multiple source statements, resulting in inconsistent translation results.

Method used

A neural network machine translation method is proposed. By encoding multiple source-end sequences, the global encoding results and local encoding results of the current word are obtained, and the translated target statements are combined to integrate local and global decoding intermediate states. , generate more accurate translation results.

Benefits of technology

Improves the accuracy and coherence of translation results, ensures context consistency between multiple source statements, and improves the overall quality of translation.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN114065776B_ABST
    Figure CN114065776B_ABST
Patent Text Reader

Abstract

The present invention discloses a neural network machine translation method and device, the neural network machine translation method comprising: encoding multiple acquired source end sequences to obtain a global encoding result of the current word, encoding the current sequence where the current word is located in the multiple source end sequences to obtain a local encoding result of the current word; decoding according to the target end sequence corresponding to the translated target sentence, the local encoding result of the current word and the global encoding result to obtain a local decoding intermediate state and a global decoding intermediate state of the current word; fusing the local decoding intermediate state and the global decoding intermediate state to obtain a decoding result of the current word. The present invention solves the problem of low accuracy of translation results in the prior art.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to a neural network machine translation method and device. Background Art

[0002] With the rise of neural networks, people have been trying to apply neural networks to natural language processing (NLP), for example, using neural machine translation to convert source language into target language. Among them, the transformer model is the current benchmark model for neural machine translation. The source sentence is encoded and decoded using the transformer model based on the self-attention neural network to obtain the target sentence, and the translated target sentence is accurate in meaning and easy to understand. However, in the translation method using the transformer model in the prior art, when performing language processing, each source sentence is generally processed separately, which will cause the problem of low accuracy of the translation result. Summary of the invention

[0003] In view of the above problems, the present invention is proposed to provide a neural network machine translation method and device that overcomes the above problems or at least partially solves the above problems.

[0004] In a first aspect, an embodiment of the present invention provides a neural network machine translation method, comprising the following steps:

[0005] Performing encoding processing on the obtained multiple source end sequences to obtain a global encoding result of the current word, and performing encoding processing on the current sequence where the current word is located in the multiple source end sequences to obtain a local encoding result of the current word;

[0006] Decoding is performed according to the target end sequence corresponding to the translated target sentence, the local encoding result and the global encoding result of the current word to obtain the local decoding intermediate state and the global decoding intermediate state of the current word;

[0007] The local decoding intermediate state and the global decoding intermediate state are fused to obtain a decoding result of the current word.

[0008] In a second aspect, an embodiment of the present invention provides a method for translating a text, comprising:

[0009] For each paragraph of the article, a plurality of source sentence segments are obtained according to the reading order of the sentences; the source sentence segments include one or more source sentences;

[0010] For each source sentence fragment, the above neural network machine translation method is executed to obtain the corresponding target sentence.

[0011] In a third aspect, an embodiment of the present invention provides a method for translating text from a picture, including:

[0012] Recognize the text information in the acquired image to obtain the source sentence corresponding to the text information in the image;

[0013] The above-mentioned neural network machine translation method is executed for the source sentence corresponding to the text information in the picture to obtain the target sentence corresponding to the text information in the picture.

[0014] In a fourth aspect, an embodiment of the present invention provides a speech translation method, comprising:

[0015] Acquire voice information, and perform recognition to obtain source sentences corresponding to the voice information;

[0016] The above-mentioned neural network machine translation method is executed for the source sentence corresponding to the voice information to obtain the target sentence corresponding to the voice information.

[0017] In a fifth aspect, an embodiment of the present invention provides a neural network machine translation device, including: an encoding end and a decoding end;

[0018] The encoding end includes at least one encoder;

[0019] The decoding end has at least one decoder;

[0020] The encoder is used to encode the multiple source end sequences obtained to obtain a global encoding result of the current word, and encode the current sequence where the current word is located in the multiple source end sequences to obtain a local encoding result of the current word;

[0021] The decoder is used to decode according to the target end sequence corresponding to the translated target sentence, the local encoding result and the global encoding result corresponding to the current word, to obtain the local decoding intermediate state and the global decoding intermediate state corresponding to the current word; and to fuse the local decoding intermediate state and the global decoding intermediate state of the current word to obtain the decoding result of the current word.

[0022] In a sixth aspect, an embodiment of the present invention provides a computer-readable storage medium having computer instructions stored thereon, which implement the above method when executed by a processor.

[0023] In a seventh aspect, an embodiment of the present invention provides a translation device, comprising: a processor, and a memory for storing processor executable commands; wherein the processor is configured to execute the above method.

[0024] The beneficial effects of the above technical solution provided by the embodiment of the present invention include at least:

[0025] The neural network machine translation method provided by the embodiment of the present invention, when translating multiple source end sequences, obtains the local encoding result of the current word corresponding to the current sequence and the global encoding result of the current word corresponding to multiple source end sequences respectively during encoding; during decoding, decoding is performed according to the target end sequence corresponding to the translated target sentence, the local encoding result of the current word and the global encoding result, to obtain the local decoding intermediate state and the global decoding intermediate state of the current word, and finally the decoding result of the current word is obtained by fusion processing. In the encoding and decoding process, not only the encoding and decoding result of the current word corresponding to the current sequence is paid attention to, but also the encoding and decoding result of the current word corresponding to multiple source end sequences is paid attention to, so that the decoding result obtained by fusion of the local decoding intermediate state and the global decoding intermediate state corresponding to the current word is more in line with the consistency and coherence requirements in the context, and the final translation result is more accurate.

[0026] Other features and advantages of the present invention will be described in the following description, and partly become apparent from the description, or understood by practicing the present invention. The purpose and other advantages of the present invention can be realized and obtained by the structures particularly pointed out in the written description, claims, and drawings.

[0027] The technical solution of the present invention is further described in detail below through the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] The accompanying drawings are used to provide a further understanding of the present invention and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the present invention and do not constitute a limitation of the present invention. In the accompanying drawings:

[0029] Figure 1 A flowchart of a neural network machine translation method according to an embodiment of the present invention;

[0030] Figure 2 A schematic diagram of the structure of a neural network machine translation device in an embodiment of the present invention;

[0031] Figure 3 A flowchart of another neural network machine translation method according to an embodiment of the present invention;

[0032] Figure 4 A schematic diagram of a masking matrix of a local encoding module of a neural network machine translation device according to an embodiment of the present invention;

[0033] Figure 5 Schematic diagram of calculation of three features of a local encoding module in an embodiment of the present invention;

[0034] Figure 6Schematic diagram of a matrix of a global encoding module of a neural network machine translation device according to an embodiment of the present invention;

[0035] Figure 7 A schematic diagram of a mask matrix of a local intermediate module of a neural network machine translation device according to an embodiment of the present invention;

[0036] Figure 8 Schematic diagram of a mask matrix of a local self-attention processing module of a neural network machine translation device in an embodiment of the present invention;

[0037] Fig. 9 Schematic diagram of a mask matrix of a global intermediate module of a neural network machine translation device according to an embodiment of the present invention;

[0038] Fig.10 Schematic diagram of a matrix of a global self-attention processing module of a neural network machine translation device according to an embodiment of the present invention;

[0039] Fig.11 Schematic diagram of local encoding and global encoding process in an embodiment of the present invention;

[0040] Fig.12 Schematic diagram of local target encoding and global target encoding process in an embodiment of the present invention;

[0041] Fig.13 It is a schematic diagram of the local intermediate decoding and global intermediate decoding process in an embodiment of the present invention. DETAILED DESCRIPTION

[0042] The exemplary embodiments of the present invention will be described in more detail below with reference to the accompanying drawings. Although the exemplary embodiments of the present invention are shown in the accompanying drawings, it should be understood that the present invention can be implemented in various forms and should not be limited by the embodiments set forth herein. On the contrary, these embodiments are provided in order to enable a more thorough understanding of the present invention and to enable the scope of the present invention to be fully communicated to those skilled in the art.

[0043] In view of the problems existing in the above-mentioned prior art, the embodiment of the present invention provides a neural network machine translation method, the process of which is as follows Figure 1 As shown, including:

[0044] S101: performing encoding processing on the acquired multiple source end sequences to obtain a global encoding result of the current word, and performing encoding processing on the current sequence where the current word is located in the multiple source end sequences to obtain a local encoding result of the current word;

[0045] S102: Decoding is performed according to the target end sequence corresponding to the translated target sentence, the local encoding result and the global encoding result of the current word to obtain a local decoding intermediate state and a global decoding intermediate state of the current word;

[0046] S103: Fusing the local decoding intermediate state of the current word and the global decoding intermediate state to obtain a decoding result of the current word.

[0047] The neural network machine translation method provided in the embodiment of the present invention can be achieved by Figure 2 The neural network machine translation device shown in the figure is used to implement the translation, so as to be applied in the scenario of translating multiple source sentences. Figure 2 As shown, the neural network machine translation device includes an encoding end 1 and a decoding end 2. When executing the above method, the encoding end 1 performs word encoding processing on multiple input source sentences to obtain multiple source end sequences described in the above step S101; the decoding end 2 performs word encoding processing on the target sentence that has been translated during the translation process to obtain the target end sequence corresponding to the target sentence that has been translated described in the above step S102.

[0048] In the above step S101, encoding is performed on the multiple source sequences obtained to obtain a global encoding result of the current word, and encoding is performed on the current sequence where the current word is located in the multiple source sequences to obtain a local encoding result of the current word, which may specifically include the following steps:

[0049] Obtaining a local encoding result of the current word according to the context information of the current word in the current sequence of the multiple source end sequences;

[0050] A global encoding result of the current word is obtained according to the context information of the current word in the multiple source sequences and the local encoding results of each word in the multiple source sequences.

[0051] In the above step S102, decoding is performed according to the target end sequence corresponding to the translated target sentence, the local encoding result and the global encoding result of the current word to obtain the local decoding intermediate state and the global decoding intermediate state of the current word, which may specifically include the following steps:

[0052] Obtaining a local decoding intermediate state of the current word according to a target-side sequence corresponding to the current sequence in a target-side sequence corresponding to the translated target sentence and a local encoding result of each word in the current sequence;

[0053] According to the global encoding results of the target-end sequence corresponding to the translated target sentence and each word in the multiple source-end sequences, a global decoding intermediate state of the current word is obtained.

[0054] Wherein, according to the target end sequence corresponding to the current sequence in the target end sequence corresponding to the translated target sentence and the local encoding results of each word in the current sequence, obtaining the local decoding intermediate state of the current word may specifically include the following steps:

[0055] Determine a target word corresponding to the current word in a target end sequence corresponding to the current sequence, and perform self-attention encoding processing on the target word corresponding to the current word according to previous context information of the target word in the target end sequence corresponding to the current sequence to obtain a local target encoding result;

[0056] According to the local encoding results of each word in the current sequence and the local target encoding result, the current word is subjected to self-attention decoding processing to obtain the local decoding intermediate state of the current word.

[0057] Wherein, obtaining the global decoding intermediate state of the current word according to the target end sequence corresponding to the translated target sentence and the global encoding results of each word in the multiple source end sequences may specifically include the following steps:

[0058] Determine a target word corresponding to the current word in the target-side sequence corresponding to the translated target sentence, and perform self-attention encoding processing on the target word corresponding to the current word according to the previous context information of the target word in the target-side sequence corresponding to the translated target sentence to obtain a global target encoding result;

[0059] According to the global encoding results of each word in the multiple source sequences and the global target encoding result, the current word is subjected to self-attention decoding processing to obtain a global decoding intermediate state of the current word.

[0060] Reference Figure 2 As shown, the encoding end 1 of the neural network machine translation device includes N encoders 11, each encoder 11 includes a local encoding module 111 and a global encoding module 112; the decoding end 2 corresponding to the encoding end 1 also includes N decoders 21, each decoder 21 includes a local target encoding module 211, a local self-attention processing module 212, a global target encoding module 213, a global self-attention processing module 214 and a fusion module 215, where N is a positive integer, for example, N=6.

[0061] When encoding the current word, the local encoding module 111 of the encoder 11 not only focuses on the feature information of the current word, but also can obtain the context information of the current word in the current sequence, so that the current word can be self-attentionally encoded according to the context information of the current word in the current sequence to obtain the local encoding result of the current word. When encoding the current word, the global encoding module 112 of the encoder 11 not only focuses on the feature information of the current word, but also can obtain the context information of the current word in multiple source sequences, so that the current word can be self-attentionally encoded according to the context information of the current word in multiple source sequences to obtain the global encoding result of the current word.

[0062] When encoding the target end sequence corresponding to the current sequence, the local target encoding module 211 of the decoder 21 not only focuses on the feature information of the target word corresponding to the current word, but also can obtain the previous context information of the target word corresponding to the current word in the target end sequence corresponding to the current sequence, so that the target word corresponding to the current word can be self-attentionally encoded according to the previous context information in the target end sequence where the target word corresponding to the current word is located to obtain the local target encoding result. When decoding the current word, the local self-attention processing module 212 of the decoder 21 not only focuses on the local encoding result corresponding to the current word, but also can obtain the local encoding result of each word in the current sequence, that is, the context information of the current word in the current sequence, so that the current word can be decoded according to the previous and next context information of the current word in the current sequence and the previous context information in the target end sequence where the target word corresponding to the current word is located to obtain the local decoding intermediate state of the current word.

[0063] When encoding the target end sequence corresponding to the translated target sentence, the global target encoding module 213 of the decoder 21 not only focuses on the feature information of the target word corresponding to the current word, but also can obtain the previous context information of the target word corresponding to the current word in the target end sequence corresponding to the translated target sentence, so that the target word corresponding to the current word can be self-attentionally encoded according to the previous context information of the target word corresponding to the current word in the target end sequence corresponding to the translated target sentence to obtain the global target encoding result. When decoding the current word, the global self-attention processing module 214 of the decoder 21 not only focuses on the global encoding result corresponding to the current word, but also can obtain the global encoding results of each word in multiple source end sequences, that is, the context information of the current word in multiple source end sequences, so that the current word can be decoded according to the previous and next context information of the current word in multiple source end sequences and the previous context information of the target word corresponding to the current word in the target end sequence corresponding to the translated target sentence, and obtain the global decoding intermediate state of the current word.

[0064] Reference Figure 2 As shown, the encoding end 1 of the neural network machine translation device further includes a word encoder 12, wherein the word encoder 12 includes a word embedding encoding module 121 and a position encoding module 122. For any source language segment including multiple source sentences, it can be encoded by the word embedding encoding module 121 and the position encoding module 122 of the encoding end 1, the word embedding encoding module 121 encodes the input multiple source sentences, and the position encoding module 122 performs position encoding on each word of the input multiple source sentences, and finally merges to obtain multiple source end sequences corresponding to the multiple source sentences.

[0065] Reference Figure 2As shown, the decoding end 2 of the neural network machine translation device also includes a target word encoder 22, wherein the target word encoder 22 includes a target word embedding encoding module 221 and a target word position encoding module 222. For target sentences translated from multiple source sentences, they can be encoded by the target word embedding encoding module 221 and the target word position encoding module 222 of the decoding end 2. The target word embedding encoding module 221 encodes the translated target sentence, and the target word position encoding module 222 performs position encoding on each target word of the input translated target sentence, and finally fuses them to obtain a target end sequence of the translated target sentence.

[0066] In the embodiment of the present invention, since the source-end sequence is translated according to the reading order of the words during the translation process, each target word in the target-end sequence corresponding to the translated target sentence will correspond to a word in multiple source-end sequences; for the target-end sequence corresponding to the translated target sentence, since the target word encoder 22 of the decoding end 2 performs a right shift (Shift Right) operation when encoding the translated target sentence, the current word of the currently translated source-end sequence corresponds to the last translated target word in the translated target sentence, that is, the target word corresponding to the current word is the translation result corresponding to the previous word of the current word in the source-end sequence.

[0067] The neural network machine translation method provided by the embodiment of the present invention, when translating multiple source end sequences, obtains the local encoding result of the current word corresponding to the current sequence and the global encoding result of the current word corresponding to multiple source end sequences respectively during encoding; during decoding, decoding is performed according to the target end sequence corresponding to the translated target sentence, the local encoding result of the current word and the global encoding result, to obtain the local decoding intermediate state and the global decoding intermediate state of the current word, and finally the decoding result of the current word is obtained by fusion processing. In the encoding and decoding process, not only the encoding and decoding result of the current word corresponding to the current sequence is paid attention to, but also the encoding and decoding result of the current word corresponding to multiple source end sequences is paid attention to, so that the decoding result obtained by fusion of the local decoding intermediate state and the global decoding intermediate state corresponding to the current word is more in line with the consistency and coherence requirements in the context, and the final translation result is more accurate.

[0068] Embodiment 2:

[0069] In order to describe the technical solution of the present invention in more detail, the neural network machine translation method provided by the second embodiment of the present invention is referred to Figure 3 As shown, the specific steps include:

[0070] S301: Obtaining a local encoding result of the current word according to the context information of the current word in the current sequence of the multiple source end sequences;

[0071] S302: Obtain a global encoding result of the current word according to the context information of the current word in the multiple source sequences and the local encoding results of each word in the multiple source sequences;

[0072] S303: Obtaining a local decoding intermediate state of the current word according to the target-side sequence corresponding to the current sequence in the target-side sequence corresponding to the translated target sentence and the local encoding results of each word in the current sequence;

[0073] S304: Obtaining a global decoding intermediate state of the current word according to the target-end sequence corresponding to the translated target sentence and the global encoding results of each word in the multiple source-end sequences;

[0074] S305: Fusing the local decoding intermediate state and the global decoding intermediate state to obtain a decoding result of the current word.

[0075] The neural network machine translation method provided by the embodiment of the present invention, when translating multiple source end sequences, obtains a local encoding result according to the context information of the current word in the current sequence during encoding, and obtains a global encoding result according to the context information of the current word in multiple source end sequences; when decoding, obtains a local decoding intermediate state of the current word according to the target end sequence corresponding to the current sequence and the local encoding result obtained by encoding, obtains a global decoding intermediate state of the current word according to the target end sequence corresponding to the translated target sentence and the global encoding result obtained by encoding, and finally obtains the decoding result of the current word by fusion processing. In the encoding and decoding process, not only the context information of the current word in the current sequence is paid attention to, but also the context information of the current word in multiple source end sequences is paid attention to, so that the decoding result obtained by fusion is more in line with the consistency and coherence requirements in the context, and the final translation result is more accurate, avoiding the problem of inconsistent full-text reference in the target sentence after the same word in multiple source sentences is translated when translating multiple source sentences.

[0076] The inventors of the present invention have found in experiments that the Transformer model in the prior art, when translating a source language segment including multiple source sentences, processes each source sentence separately, and only focuses on the context information of the current sentence during the translation process, but cannot perform encoding and decoding processing in combination with the context information of multiple source sentences. As a result, the words in the translated target sentence do not meet the requirements of coherence and consistency in the context, resulting in the phenomenon that the same word in multiple source sentences has inconsistent full-text references in the target sentence. Based on this, the inventors of the present invention provide a Figure 2 The neural network machine translation device shown is used to implement the above-mentioned neural network machine translation method, which is suitable for the scenario of translating multiple source sentences.

[0077] The following pairs Figure 2 The process of implementing the above method by the neural network machine translation device shown is described in detail:

[0078] <About self-attention encoding processing during local encoding>

[0079] In one embodiment, based on the neural network machine translation device, the local encoding result of the current word is obtained according to the context information of the current word in the current sequence in the step S301, which may specifically include the following steps:

[0080] According to the preset first parameter feature and the context information in the current sequence, a first query feature, a first value feature and a first key feature are obtained; according to the first query feature, the first value feature and the first key feature, self-attention encoding processing is performed on the current word to obtain a local encoding result of the current word.

[0081] In the embodiment of the present invention, the local encoding module 111 includes a local masked self-attention network. Multiple source end sequences are input into the local encoding module 111 of the encoder 11. When the local encoding module 111 performs self-attention encoding processing on the current word, the masking matrix M1 of the local masked self-attention network can mask the feature information of the context sequence of the current sequence where the current word is located in the multiple source end sequences. Therefore, only the current sequence can be processed. In the local masked self-attention network, the word encoding feature of each word in the current sequence is compared with the preset first query parameter feature W. Q1 Perform the operation to obtain the first query feature Q1, and compare the word encoding feature of each word in the current sequence with the preset first value parameter feature W V1Perform the operation to obtain the first value feature V1, and compare the word encoding feature of each word in the current sequence with the preset first key parameter feature W K1 Perform the operation to obtain the first key feature K1.

[0082] In the embodiment of the present invention, three preset parameter features W are used when calculating Q1, V1 and K1. Q1 , W V1 and W K1 It can be in the form of a matrix, such as a matrix with a dimension of (64,512). The above combines the word encoding feature of each word in the current sequence with the three preset parameter features W Q1 , W V1 and W K1 The operation can be performed by combining the word encoding features of each word in the current sequence with the matrix W Q1 , W V1 and W K1 Multiply them to obtain the query feature matrix, value feature matrix and key feature matrix of each word, combine the query feature matrix of each word to obtain the first query feature Q1, combine the value feature matrix of each word to obtain the first value feature V1, and combine the key feature matrix of each word to obtain the first key feature K1. The specific method of realizing matrix multiplication and combination can refer to the description in the prior art, and is not limited in the embodiments of the present invention.

[0083] In the embodiment of the present invention, the above-mentioned preset first parameter feature, such as the preset first query parameter feature W Q1 , the preset first value parameter characteristic W V1 and the preset first key parameter characteristic W K1 All of them can be obtained in the process of training the local masked self-attention network. The specific training process can refer to the description in the prior art and will not be repeated here.

[0084] <About self-attention encoding processing during global encoding>

[0085] In one embodiment, based on the neural network machine translation device, the step S302 above obtains the global encoding result of the current word according to the context information of the current word in the multiple source sequences and the local encoding results of each word in the multiple source sequences, which may specifically include the following steps:

[0086] Obtaining a second query feature, a second value feature, and a second key feature according to a preset second parameter feature and a local encoding result of each word in the plurality of source end sequences;

[0087] The current word is subjected to self-attention encoding processing according to the second query feature, the second value feature and the second key feature to obtain a global encoding result of the current word.

[0088] In the embodiment of the present invention, in the global self-attention network, multiple source end sequences are input into the global encoding module 112 of the encoder 11. When the global encoding module 112 performs self-attention encoding processing on the current word, the word encoding features of each word in the multiple source end sequences are compared with the preset second query parameter feature W. Q2 The local encoding results of each word in the multiple source end sequences are compared with the preset second value parameter feature W V2 The local encoding results of each word in the multiple source end sequences are compared with the preset second key parameter feature W K2 Perform the operation to obtain the second key feature K2.

[0089] In the embodiment of the present invention, three preset parameter features W are used when calculating Q2, V2 and K2. Q2 , W V2 and W K2 It can be in the form of a matrix, for example, a matrix with a dimension of (64,512). The specific operation process of obtaining the second query feature Q2, the second value feature V2, and the second key feature K2 by the above operation can refer to the specific description of the self-attention encoding process during local encoding in the above implementation, which will not be repeated here.

[0090] In the embodiment of the present invention, the above-mentioned preset second parameter feature, such as the preset second query parameter feature W Q2 , the preset second value parameter characteristic W V2 and the preset second key parameter characteristic W K2 It can also be obtained during the training of the global self-attention network.

[0091] In the embodiment of the present invention, when encoding the current word, the local encoding module 111 of the encoder 11 of the neural network machine translation device not only focuses on the feature information of the current word, but also can obtain the context information of the current word in the current sequence, so that the current word can be self-attentionally encoded according to the context information of the current word in the current sequence to obtain the local encoding result of the current word. When encoding the current word, the global encoding module 112 of the encoder 11 not only focuses on the feature information of the current word, but also can obtain the context information of the current word in multiple source sequences, so that the current word can be self-attentionally encoded according to the context information of the current word in multiple source sequences to obtain the global encoding result of the current word.

[0092] <About self-attention encoding processing during local decoding>

[0093] In one embodiment, based on the neural network machine translation device, the local decoding intermediate state of the current word is obtained according to the target end sequence corresponding to the current sequence in the target end sequence corresponding to the translated target sentence and the local encoding results of each word in the current sequence in the step S303, which may specifically include the following steps:

[0094] Determine a target word corresponding to the current word in the target end sequence corresponding to the current sequence, and perform self-attention encoding processing on the target word corresponding to the current word according to the previous context information of the target word in the target end sequence corresponding to the current sequence to obtain a local target encoding result;

[0095] According to the local encoding results of each word in the current sequence and the local target encoding result, the current word is subjected to self-attention decoding processing to obtain the local decoding intermediate state of the current word.

[0096] Reference Figure 2 As shown, when encoding the target end sequence corresponding to the current sequence, the local target encoding module 211 of the decoder 21 not only pays attention to the feature information of the target word corresponding to the current word, but also can obtain the previous context information of the target word corresponding to the current word in the target end sequence corresponding to the current sequence, so that the target word corresponding to the current word can be self-attentionally encoded according to the previous context information in the target end sequence where the target word corresponding to the current word is located to obtain the local target encoding result. When decoding the current word, the local self-attention processing module 212 of the decoder 21 not only pays attention to the local encoding result corresponding to the current word, but also can obtain the local encoding result of each word in the current sequence, that is, the context information of the current word in the current sequence, so that the current word can be decoded according to the previous and next context information of the current word in the current sequence and the previous context information in the target end sequence where the target word corresponding to the current word is located to obtain the local decoding intermediate state of the current word.

[0097] In a specific embodiment, the above-mentioned determination of the target word corresponding to the current word in the target end sequence corresponding to the current sequence, and performing self-attention encoding processing on the target word corresponding to the current word according to the previous context information of the target word in the target end sequence corresponding to the current sequence to obtain a local target encoding result, specifically includes the following steps:

[0098] Determine a target end sequence corresponding to the current sequence, and search and determine a target word corresponding to the current word in the target end sequence corresponding to the current sequence;

[0099] Obtaining a third query feature, a third value feature, and a third key feature according to a preset third parameter feature and previous text information of a target word corresponding to the current word in a target end sequence corresponding to the current sequence;

[0100] The target word corresponding to the current word is subjected to self-attention encoding processing according to the third query feature, the third value feature and the third key feature to obtain the local target encoding result.

[0101] In the embodiment of the present invention, the local target encoding module 211 includes a local masked self-attention network. The target end sequence of the translated target sentence is input into the local target encoding module 211 of the decoder 21. When the local target encoding module 211 performs self-attention encoding processing on the target word corresponding to the current word, the masking matrix M3 of the local masked self-attention network can mask the feature information of the previous sequence of the target end sequence where the target word corresponding to the current word in the target end sequence is located and all feature information after the target word corresponding to the current word. Therefore, only the target end sequence where the target word is located can be processed. In the local masked self-attention network, the target word encoding feature of each target word in the target end sequence where the target word is located is compared with the preset third query parameter feature W. Q3 The third query feature Q3 is obtained by performing an operation, and the target word encoding feature of each target word in the target end sequence where the target word is located is compared with the preset third value parameter feature W V3 The third value feature V3 is obtained by performing the operation, and the target word encoding feature of each target word in the target end sequence where the target word is located is compared with the preset third key parameter feature W K3 Perform the operation to obtain the third key feature K3.

[0102] In the embodiment of the present invention, three preset parameter features W are used when calculating Q3, V3 and K3. Q3 , W V3 and W K3 It can be in the form of a matrix, for example, a matrix with a dimension of (64,512). The specific operation process of performing the above operation to obtain the third query feature Q3, the third value feature V3, and the third key feature K3 can refer to the specific description of the self-attention encoding process during local encoding in the above implementation, which will not be repeated here.

[0103] In the embodiment of the present invention, the above-mentioned preset third parameter feature, such as the preset third query parameter feature W Q3 , the preset third value parameter characteristic W V3 and the preset third key parameter characteristic W K3 It can also be obtained during the process of training the local masked self-attention network of the local target encoding module 211.

[0104] In a specific embodiment, the above-mentioned self-attention decoding process of the current word according to the local encoding results of each word in the current sequence and the local target encoding result to obtain the local decoding intermediate state of the current word includes:

[0105] Obtaining a fourth query feature, a fourth value feature and a fourth key feature according to a preset fourth parameter feature, the local target encoding result and the local encoding results of each word in the current sequence;

[0106] The current word is subjected to self-attention decoding processing according to the fourth query feature, the fourth value feature and the fourth key feature to obtain a local decoding intermediate state of the current word.

[0107] In the embodiment of the present invention, the local self-attention processing module 212 includes a local masked self-attention network. The local encoding results of each word in the multiple source sequences and the above-mentioned local target encoding results are input into the local self-attention processing module 212 of the decoder 21. When the local self-attention processing module 212 performs self-attention decoding processing on the current word, the masking matrix M4 of the local masked self-attention network can mask the local encoding results of each word in the context sequence of the current sequence where the current word is located in the multiple source sequences, and mask the local target encoding results corresponding to each target word in the context sequence of the target sequence where the target word corresponding to the current word in the target sequence is located. Therefore, only the local encoding results of each word in the current sequence where the current word is located and the local target encoding results of the target word corresponding to the current word can be processed. In the local masked self-attention network, the local target encoding result of the target word corresponding to the current word is compared with the preset fourth query parameter feature W. Q4 The fourth query feature Q4 is obtained by performing calculations, and the local encoding results of each word in the current sequence where the current word is located are compared with the preset fourth value parameter feature W V4 The fourth value feature V4 is obtained by performing the operation, and the local encoding results of each word in the current sequence where the current word is located are compared with the preset fourth key parameter feature W K4 Perform calculation to obtain the fourth key feature K4.

[0108] In the embodiment of the present invention, three preset parameter features W are used when calculating Q4, V4 and K4. Q4 , W V4 and W K4 It can be in the form of a matrix, for example, a matrix with a dimension of (64,512). The specific operation process of performing the above operation to obtain the fourth query feature Q4, the fourth value feature V4, and the fourth key feature K4 can refer to the specific description of the self-attention encoding process during local encoding in the above implementation, which will not be repeated here.

[0109] In the embodiment of the present invention, the above-mentioned preset fourth parameter feature, such as the preset fourth query parameter feature W Q4 , the preset fourth value parameter characteristic W V4 And the preset fourth key parameter feature W K4 It can also be obtained during the training of the local masked self-attention network of the local self-attention processing module 212.

[0110] <About self-attention encoding processing during global decoding>

[0111] In one embodiment, based on the neural network machine translation device, in step S304, according to the target end sequence corresponding to the translated target sentence and the global encoding results of each word in the multiple source end sequences, the global decoding intermediate state of the current word is obtained, including:

[0112] Determine a target word corresponding to the current word in the target-side sequence corresponding to the translated target sentence, and perform self-attention encoding processing on the target word corresponding to the current word according to the previous context information of the target word corresponding to the current word in the target-side sequence corresponding to the translated target sentence to obtain a global target encoding result;

[0113] According to the global encoding results of each word in the multiple source sequences and the global target encoding result, the current word is subjected to self-attention decoding processing to obtain a global decoding intermediate state of the current word.

[0114] Reference Figure 2 As shown, when encoding the target end sequence corresponding to the translated target sentence, the global target encoding module 213 of the decoder 21 not only focuses on the feature information of the target word corresponding to the current word, but also can obtain the previous context information of the target word corresponding to the current word in the target end sequence corresponding to the translated target sentence, so that the target word corresponding to the current word can be self-attentionally encoded according to the previous context information of the target word corresponding to the current word in the target end sequence corresponding to the translated target sentence to obtain the global target encoding result. When decoding the current word, the global self-attention processing module 214 of the decoder 21 not only focuses on the global encoding result corresponding to the current word, but also can obtain the global encoding results of each word in multiple source end sequences, that is, the context information of the current word in multiple source end sequences, so that the current word can be decoded according to the previous and next context information of the current word in multiple source end sequences and the previous context information of the target word corresponding to the current word in the target end sequence corresponding to the translated target sentence to obtain the global decoding intermediate state of the current word.

[0115] In a specific embodiment, the above-mentioned determination of the target word corresponding to the current word in the target-side sequence corresponding to the translated target sentence, and performing self-attention encoding processing on the target word corresponding to the current word according to the previous context information of the target word in the target-side sequence corresponding to the translated target sentence to obtain a global target encoding result specifically includes the following steps:

[0116] Determine a target end sequence corresponding to the target sentence that has been translated, and search and determine a target word corresponding to the current word in the target end sequence corresponding to the target sentence that has been translated;

[0117] According to the preset fifth parameter feature and the previous text information of the target word corresponding to the current word in the target end sequence corresponding to the translated target sentence, a fifth query feature, a fifth value feature and a fifth key feature are obtained;

[0118] The target word corresponding to the current word is subjected to self-attention encoding processing according to the fifth query feature, the fifth value feature and the fifth key feature to obtain a global target encoding result of the current word.

[0119] In the embodiment of the present invention, the global target encoding module 213 includes a global masked self-attention network. The target end sequence of the translated target sentence is input into the global target encoding module 213 of the decoder 21. When the global target encoding module 213 performs self-attention encoding processing on the target word corresponding to the current word, the masking matrix M5 of the global masked self-attention network can mask all feature information after the target word corresponding to the current word in the target end sequence. Therefore, the target end sequence corresponding to the translated target sentence can be processed. In the global masked self-attention network, the target word encoding feature of each target word in the target end sequence corresponding to the translated target sentence is compared with the preset fifth query parameter feature W. Q5 The fifth query feature Q5 is obtained by performing an operation, and the target word encoding feature of each target word in the target end sequence corresponding to the translated target sentence is compared with the preset fifth value parameter feature W V5 The fifth value feature V5 is obtained by performing an operation, and the target word encoding feature of each target word in the target end sequence corresponding to the translated target sentence is compared with the preset fifth key parameter feature W K5 Perform the operation to obtain the fifth key feature K5.

[0120] In the embodiment of the present invention, three preset parameter features W are used when calculating Q5, V5 and K5. Q5 , W V5 and W K5It can be in the form of a matrix, for example, a matrix with a dimension of (64,512). The specific operation process of performing the above operation to obtain the fifth query feature Q5, the fifth value feature V5, and the fifth key feature K5 can refer to the specific description of the self-attention encoding process during local encoding in the above implementation, which will not be repeated here.

[0121] In the embodiment of the present invention, the above-mentioned preset fifth parameter feature, such as the preset fifth query parameter feature W Q5 , the preset fifth value parameter characteristic W V5 And the preset fifth key parameter feature W K5 It can also be obtained during the training of the global masked self-attention network of the global target encoding module 213.

[0122] In a specific embodiment, the self-attention decoding process is performed on the current word according to the global encoding results of each word in the multiple source sequences and the global target encoding result to obtain the global decoding intermediate state of the current word, including:

[0123] Obtaining a sixth query feature, a sixth value feature, and a sixth key feature according to a preset sixth parameter feature, the global target encoding result, and the global encoding results of each word in the plurality of source end sequences;

[0124] The current word is subjected to self-attention decoding processing according to the sixth query feature, the sixth value feature and the sixth key feature to obtain a global decoding intermediate state of the current word.

[0125] In the embodiment of the present invention, the global self-attention processing module 214 includes a global self-attention network. The global encoding results of each word in the multiple source end sequences and the global target encoding result are input into the global self-attention processing module 214 of the decoder 21. When the global self-attention processing module 214 performs self-attention decoding processing on the current word, the global self-attention network can process the global encoding results of each word in the multiple source end sequences and the global target encoding result of the target word corresponding to the current word. In the global self-attention network, the global target encoding result of the target word corresponding to the current word is compared with the preset sixth query parameter feature W. Q6 The sixth query feature Q6 is obtained by performing an operation, and the global encoding results of each word of the multiple source end sequences are compared with the preset sixth value parameter feature W V6 The sixth value feature V6 is obtained by performing the operation, and the global encoding results of each word of the multiple source end sequences are compared with the preset sixth key parameter feature W K6 Perform calculation to obtain the sixth key feature K6.

[0126] In the embodiment of the present invention, three preset parameter features W are used when calculating Q6, V6 and K5. Q6 , WV6 and W K6 It can be in the form of a matrix, for example, a matrix with a dimension of (64,512). The specific operation process of performing the above operation to obtain the sixth query feature Q6, the sixth value feature V6, and the sixth key feature K6 can refer to the specific description of the self-attention encoding process during local encoding in the above implementation, which will not be repeated here.

[0127] In the embodiment of the present invention, the above-mentioned preset sixth parameter feature, such as the preset sixth query parameter feature W Q6 , the preset sixth value parameter characteristic W V6 And the preset sixth key parameter feature W K6 It can also be obtained during the process of training the global self-attention network of the global self-attention processing module 214.

[0128] <About the fusion processing when obtaining the decoding result of the current word>

[0129] Reference Figure 2 As shown, the decoder 21 of the neural network machine translation device further includes a fusion module 215 and a feedforward neural network 113 connected to the fusion module 215. The specific process of performing the above step S305 may include:

[0130] The fusion module 215 performs fusion processing on the local decoding intermediate state of the current word and the global decoding intermediate state to obtain a fusion decoding result of the current word;

[0131] The feedforward neural network 113 connected to the fusion module 215 performs unidirectional propagation on the fusion decoding result to obtain the decoding result of the current word.

[0132] The fusion module 215 may perform concatenation processing on the local decoding intermediate state of the current word and the global decoding intermediate state. The specific method for implementing the concatenation processing may refer to the description in the prior art and is not specifically limited here.

[0133] Of course, in order to obtain the target word corresponding to the current word, the decoding result of the current word can be processed by the method in the prior art, such as by Softmax operation, to obtain the final output possibility result, that is, the translation result of the current word. The process of obtaining the translation result of the current word according to the decoding result of the current word can refer to the description in the prior art, and is not specifically limited in the embodiments of the present invention.

[0134] In a specific embodiment, the method may further include:

[0135] During the translation process, the translated target sentence is displayed to the user;

[0136] A word modified by a user on at least one target word in the translated target sentence is received to obtain a modified translated target sentence.

[0137] In the embodiment of the present invention, by displaying the translated target sentence to the user, the user can make corrections based on the translation result of the target word of the translated target sentence to improve the translation accuracy.

[0138] In a specific embodiment, after obtaining the decoding result of the current word, the method may further include: displaying the decoding result of the current word to the user for modification by the user. By displaying the translation result of the current word in the source sentence to the user, the user can make corrections in time based on the translation result of the target word to improve the accuracy of subsequent translation results.

[0139] The following is an introduction to the process of implementing translation by the neural network machine translation device in the embodiment of the present invention using a specific example:

[0140] Reference Figure 2 As shown, the encoding end 1 of the neural network machine translation device further includes a word encoder 12, wherein the word encoder 12 includes a word embedding encoding module 121 and a position encoding module 122. For any source language segment including multiple source sentences, it can be encoded by the word embedding encoding module 121 and the position encoding module 122 of the encoding end 1, the word embedding encoding module 121 encodes the input multiple source sentences, and the position encoding module 122 performs position encoding on each word of the input multiple source sentences, and finally merges to obtain multiple source end sequences corresponding to the multiple source sentences.

[0141] In the embodiment of the present invention, the multiple source-end sequences are obtained by performing a word embedding operation (used to map natural language features into digital features, i.e., mapping text features into sequence features) on the input data of the multiple source sentences by the neural network machine translation device. Assuming that the multiple source sentences are "Julia saw a cat. It was hungry.", the corresponding multiple source-end sequences X are obtained through the word embedding encoding module 121 and the position encoding module 122.

[0142] In the embodiment of the present invention, when the obtained multiple source sequences X are input into the local masked self-attention network of the local encoding module 111, when the local masked self-attention network processes each word, the masking matrix M1 of the local masked self-attention network masks the feature information of the context sequence of the current sequence where the current word in the source sequence X is located. Figure 4As shown, the masking matrix M1 of the local masked self-attention network of the local encoding module 111 of the encoder 11 is a diagonal block matrix (Diagonal Block Matrix), each block in the masking matrix M1 represents the local attention of a source sequence, the blank point represents 0, and the grid point represents -∞.

[0143] In the process of the first layer encoder 11 performing self-attention encoding processing on the source sequence corresponding to "Julia saw a cat. It was hungry.", when the masking matrix M1 performs a masking operation on the feature information of the context sequence of the current sequence where the current word is located in the multiple source sequences X, for the feature information corresponding to the current sequence where the current word is located in the multiple source sequences X, adding 0 does not affect its original feature information, and for the feature information corresponding to the source sequence other than the current sequence where the current word is located in the multiple source sequences X, adding -∞ results in -∞, thereby masking the feature information corresponding to the source sequence other than the current sequence where the current word is located in the multiple source sequences, and will not affect the current sequence where the current word is located, thereby achieving self-attention encoding processing on the current sequence, and therefore, only the current sequence can be processed. Assuming that the current word is "was", the local encoding module 111 can mask the feature information of each word in the source sequence corresponding to "Julia saw a cat."

[0144] For example, assuming that the word embedding feature corresponding to "It" in the source sequence is x1, the word embedding feature corresponding to "was" is x2, and the word embedding feature corresponding to "hungury" is x3, when the source sequence X is input into the local encoding module for processing, the local masked self-attention network will calculate three new features, which are used to reflect the weights of the words from three sides, namely the first query feature Q1, the first value feature V1, and the first key feature K1. The three preset parameter features W used in calculating Q1, V1, and K1 are Q1 , W V1 and W K1 It can be in the form of a matrix, for example, a matrix with a dimension of (64,512). The specific calculation process can be referred to Figure 5 As shown, x1 is respectively related to the matrix W Q1 , W V1 and W K1 Multiply the features q1, v1 and k1, x2 and the matrix W Q1 , W V1 and W K1 Multiply the features q2, v2 and k2, x3 and the matrix W Q1 , W V1 and W K1Multiply them to get features q3, v3 and k3. Finally, combine the obtained q1, q2 and q3 to get the first query feature Q1, combine the obtained v1, v2 and v3 to get the first query feature V1, and combine the obtained k1, k2 and k3 to get the first query feature K1. Then, use the following formula (1) to perform softmax operation in the local masked self-attention network to get the weight A1 of each word in the current sequence to the current word:

[0145]

[0146] Among them, Q1 represents the first query feature in the local masked self-attention network, represents the matrix transposition result of the first value feature K1 in the local masked self-attention network, d / h is a constant; M1 represents the masking matrix of the local masked self-attention network;

[0147] The operation obtains the weight A1 of each word in the current sequence “It was hungry.” where “was” is located, and then the first query feature V1 is weighted and summed according to the weight A1 to obtain the local encoding result of “was”.

[0148] In the embodiment of the present invention, the local masked self-attention network of the local encoding module 111 may include a multi-head masked self-attention network, in which the word encoding features of each word in the current sequence “It was hungry.” where “was” is located are respectively compared with the preset query parameter features W Q1 , preset value parameter feature W V1 and the preset key parameter characteristics W K1 Perform calculation processing to obtain a set of first query feature Q1, first value feature V1 and first key feature K1, perform point multiplication operation on the first query feature Q1 and the first key feature K1 to obtain the point multiplication result, perform softmax operation on the obtained point multiplication result to obtain the weight A1 of each word in the current sequence "It was hungry." where "was" is located for "was", perform weighted summation processing on the first value feature V1 according to the weight A1, and obtain the local hidden layer encoding result of "was". Perform concatenation operation on each local hidden layer encoding result obtained in the multi-head masked self-attention network to obtain a final local hidden layer encoding result, and perform unidirectional propagation on the final local hidden layer encoding result through the feedforward neural network 113 connected to the output end of the local masked self-attention network to obtain the local encoding result of "was".

[0149] In the embodiment of the present invention, referring to Figure 6As shown, the matrix in the global self-attention network of the global encoding module 112 of the encoder 11 is not masked, the entire block represents the global attention matrix of multiple source sequences, and the blank point represents 0. When the first layer encoder 11 encodes the source sequence corresponding to "Julia saw a cat. It was hungry.", the global encoding module 112 can obtain the context information that needs to be paid attention to for each word in the multiple source sequences, that is, the global encoding module 112 can obtain the feature information of each word of the two source sequences corresponding to "Julia saw a cat. It was hungry." In the global self-attention network, the word encoding feature of each word in "Julia saw a cat. It was hungry." is compared with the preset query parameter feature W. Q2 The second query feature Q2 and the preset value parameter feature W are obtained by calculation. V2 The operation is performed to obtain the second value feature V2 and the preset key parameter feature W K2 The calculation is performed to obtain the second key feature K2. Specifically, according to the three preset parameter features W Q2 , W K2 and W Q2 The realization process of obtaining the above three features Q2, K2 and V2 can refer to the above Figure 5 Description.

[0150] When determining the global encoding result of "was", softmax calculation is performed according to formula (2) to obtain the weight A2 of each word in multiple source sequences for the current word:

[0151]

[0152] Among them, Q2 represents the second query feature in the global self-attention network, Represents the matrix transposition result of the second key feature in the global self-attention network, d / h is a constant;

[0153] The calculation obtains the weight A2 of each word in “Julia saw a cat. It was hungry.” for “was”, and then the second query feature V2 is weighted and summed according to the weight A2 to obtain the global encoding result of “was”.

[0154] In a specific embodiment, the global self-attention network may include a multi-head self-attention network, each head self-attention network based on the word encoding feature of each word in "Julia saw a cat. It was hungry." and the preset query parameter feature W Q2, according to the local encoding results of each word in "Julia saw a cat. It was hungry." and the preset value parameter feature W V2 and the preset key parameter characteristics W K2 Perform calculation processing to obtain a set of second query features Q2, second value features V2 and second key features K2; perform a dot product operation on the second query feature Q2 and the second key feature K2 to obtain a dot product result; perform a softmax operation on the obtained dot product result to obtain the weight A2 of each word in "Julia saw a cat. It was hungry." for "was"; perform weighted sum processing on the second value feature V2 according to the weight A2 to obtain the global hidden layer encoding result of "was". Perform a splicing operation on the global hidden layer encoding results obtained in the multi-head masked self-attention network to obtain a final global hidden layer encoding result, and perform a one-way propagation on the final global hidden layer encoding result through the feedforward neural network 113 connected to the output end of the global self-attention network to obtain the global encoding result of "was".

[0155] Reference Figure 2 As shown, the decoding end 2 of the neural network machine translation device further includes a target word encoder 22, wherein the target word encoder 22 includes a target word embedding encoding module 221 and a target word position encoding module 222. For target sentences that have been translated from multiple source sentences, the target word embedding encoding module 221 and the target word position encoding module 222 of the decoding end 2 can be used for encoding. The target word embedding encoding module 221 encodes the translated target sentence, and the target word position encoding module 222 performs position encoding on each target word of the input translated target sentence, and finally fuses to obtain the target end sequence of the translated target sentence. Then, in the decoding process of the current word "was" in the source sentence "Julia saw a cat. It was hungry.", the target sentence that has been translated includes the target sentence corresponding to "Julia saw a cat. It", and the target end sequence corresponding to the target sentence "Julia saw a cat. It" is obtained through the target word embedding encoding module 221 and the target word position encoding module 222.

[0156] In the embodiment of the present invention, referring to Figure 7As shown, the masking matrix M3 of the local masked self-attention network of the local target encoding module 211 of the decoder 21 is a block upper triangular matrix (Block Upper Triangular Matrix), and each block in the masking matrix M3 represents the local attention of a target end sequence, the blank point represents 0, and the grid point represents -∞. When the first layer decoder 21 encodes the target words in the target end sequence of the target sentence corresponding to "Julia saw a cat.It", the masking matrix M3 can mask the feature information of each target word in the target end sequence of the target sentence corresponding to "Julia saw a cat". In the local masked self-attention network, the word encoding features of each target word in the target end sequence ".It" corresponding to the current sequence "It was hungry." where "was" is located are compared with the preset query parameter features W. Q3 The third query feature Q3 and the preset value parameter feature W are obtained by calculation. V3 The calculation is performed to obtain the third value feature V3 and the preset key parameter feature W K3 The calculation is performed to obtain the third key feature K3. Specifically, according to the three preset parameter features W Q3 , W K3 and W Q3 The realization process of obtaining the above three features Q3, K3 and V3 can refer to the above Figure 5 Description.

[0157] When determining the local target encoding result, the masking matrix M3 is substituted into formula (3) and a softmax operation is performed to obtain the weight A3 of each target word for the target word corresponding to the current word in the target end sequence where the target word corresponding to the current word is located:

[0158]

[0159] Among them, Q3 represents the third query feature in the local masked self-attention network, Represents the matrix transposition result of the third value feature K3 in the local masked self-attention network, d / h is a constant;

[0160] The weight A3 of each target word in the target end sequence of "It" is obtained by calculation, and then the third query feature V3 is weighted and summed according to the weight A3 to obtain the local target encoding result of "was".

[0161] In a specific embodiment, the local masked self-attention network may include a multi-head masked self-attention network, each head masked self-attention network is respectively combined with the preset query parameter feature W according to the target word encoding features of each target word in the target end sequence where “It” is located. Q3, preset value parameter feature W V3 and the preset key parameter characteristics W K3 Perform calculation processing to obtain a set of third query features Q3, third value features V3 and third key features K3. Perform a dot product operation on the third query feature Q3 and the third key feature K3 to obtain a dot product result. Perform a softmax operation on the obtained dot product result to obtain the weight A3 of each target word in the target end sequence where "It" is located for "It". Perform weighted sum processing on the third value feature V3 according to the weight A3 to obtain the local intermediate hidden layer decoding result of "was". Perform a concatenation operation on each local intermediate hidden layer decoding result obtained in the multi-head masked self-attention network to obtain the local target encoding result of "was".

[0162] In the embodiment of the present invention, referring to Figure 8 As shown, the masking matrix M4 of the local masked self-attention network of the local self-attention processing module 212 of the decoder 21 is also a diagonal block matrix. The specific structure of the masking matrix M4 can refer to the above-mentioned masking matrix M1 and the description in the prior art, which will not be repeated here. When decoding "was", the masking matrix M4 can mask the local encoding results of each word in the source sequence corresponding to "Julia saw a cat." and the local target encoding results of each target word in the target sequence corresponding to "Julia saw a cat.". In the local masked self-attention network, the local target encoding result of "was" is compared with the preset query parameter feature W. Q4 The fourth query feature Q4 is obtained by performing the operation, and the local encoding results of each word of "It was hungry." are compared with the preset value parameter feature W V4 The fourth value feature V4 is obtained by performing the operation, and the local encoding results of each word of "It was hungry." are compared with the preset key parameter feature W K4 The fourth key feature K4 is obtained by performing calculations. Specifically, according to the three preset parameter features W Q4 , W K4 and W Q4 The realization process of obtaining the above three features Q4, K4 and V4 can refer to the above Figure 5 Description.

[0163] When determining the intermediate state of local decoding, substitute the masking matrix M4 into formula (4) and perform softmax operation to obtain the weight A4 of the local encoding result of each word in the current sequence for the current word:

[0164]

[0165] Among them, Q4 represents the fourth query feature in the local masked self-attention network, Represents the matrix transposition result of the fourth value feature K4 in the local masked self-attention network, d / h is a constant;

[0166] The operation obtains the weight A4 of the local encoding result of each word in “It was hungry.” for “was”, and then the fourth query feature V4 is weighted summed according to the weight A4 to obtain the local decoding intermediate state corresponding to “was”.

[0167] In a specific embodiment, the local masked self-attention network may include a multi-head masked self-attention network, each head masked self-attention network based on the local encoding results of each word in "It was hungry." and the local target encoding result of "was" and the preset query parameter feature W Q4 , preset value parameter feature W V4 and the preset key parameter characteristics W K4 Perform calculation processing to obtain a set of fourth query features Q4, fourth value features V4 and fourth key features K4, perform point multiplication operation on the fourth query feature Q4 and the fourth key feature K4 to obtain the point multiplication result, perform softmax operation on the obtained point multiplication result to obtain the weight A4 of the local encoding result of each word in "It was hungry." for "was", perform weighted summation processing on the fourth value feature V4 according to the weight A4, and obtain the local hidden layer decoding result of "was". Perform splicing operation on each local hidden layer decoding result obtained in the multi-head masked self-attention network to obtain the final local hidden layer decoding result of "was", and perform unidirectional propagation on the final local hidden layer decoding result through the feedforward neural network 113 connected to the output end of the local masked self-attention network to obtain the local decoding intermediate state of "was".

[0168] In the embodiment of the present invention, in the embodiment of the present invention, reference is made to Fig. 9 As shown, the masking matrix M5 of the global masked self-attention network of the global intermediate module 213 of the decoder 21 is an upper triangular matrix (Upper Triangular Matrix). Each block in the masking matrix M5 represents the local attention of a target end sequence, the blank point represents 0, and the grid point represents -∞. When the first layer decoder 21 encodes the target word in the target end sequence of the target sentence corresponding to "Julia saw a cat. It", the masking matrix M5 can mask the feature information other than "Julia saw a cat. It". In the global masked self-attention network, the target word encoding features of each target word in the target end sequence of the target sentence corresponding to "Julia saw a cat. It" are compared with the preset query parameter features W. Q5Perform calculations to obtain the fifth query feature Q5 and the preset value parameter feature W V5 Perform calculations to obtain the fifth value feature V5 and the preset key parameter feature W K5 The fifth key feature K5 is obtained by performing calculations. Specifically, according to the three preset parameter features W Q5 , W K5 and W Q5 The realization process of obtaining the above three features Q5, K5 and V5 can refer to the above Figure 5 Description.

[0169] When determining the global target encoding result, the masking matrix M5 is substituted into formula (5) and a softmax operation is performed to obtain the weight A5 of each target word for the target word corresponding to the current word in the target end sequence corresponding to the translated target sentence:

[0170]

[0171] Among them, Q5 represents the fifth query feature in the local masked self-attention network, Represents the matrix transposition result of the fifth value feature K5 in the local masked self-attention network, d / h is a constant;

[0172] The weight A5 of each target word in the target end sequence of the target sentence corresponding to "Julia saw a cat. It" for "It" is obtained by calculation, and then the fifth query feature V5 is weighted and summed according to the weight A5 to obtain the global target encoding result of "was".

[0173] In a specific embodiment, the global masked self-attention network may include a multi-head masked self-attention network, each head masked self-attention network compares the target word encoding features of each target word in the target end sequence of the target sentence corresponding to "Julia saw a cat. It" with the preset query parameter feature W Q5 , preset value parameter feature W V5 and the preset key parameter characteristics W K5 Perform calculation processing to obtain a set of fifth query features Q5, fifth value features V5 and fifth key features K5. Perform a dot product operation on the fifth query feature Q5 and the fifth key feature K5 to obtain a dot product result. Perform a softmax operation on the obtained dot product result to obtain the weight A5 of each target word in the target end sequence of the target sentence corresponding to "Julia saw a cat. It" for "It". Perform weighted sum processing on the fifth value feature V5 according to the weight A5 to obtain the global intermediate hidden layer decoding result of "was". Perform a splicing operation on each global intermediate hidden layer decoding result obtained in the multi-head masked self-attention network to obtain the global target encoding result of "was".

[0174] In the embodiment of the present invention, referring to Fig.10 As shown, the matrix in the global self-attention network of the global self-attention processing module 214 of the decoder 21 is not masked, the entire block represents the global attention of multiple source sequences, and the blank point represents 0. When the first-layer encoder 11 decodes the target sequence corresponding to "Julia saw a cat. It was hungry.", the global encoding module 112 can obtain the global encoding results of each word in the multiple source sequences, that is, the global encoding module 112 can obtain the global encoding results of each word corresponding to "Julia saw a cat. It was hungry." and the global target encoding results of each target word in the target sequence. In the global self-attention network, the global target encoding result of "was" is compared with the preset query parameter feature W. Q6 The sixth query feature Q6 is obtained by performing an operation, and the global encoding results of each word corresponding to "Julia saw a cat. It washungry." are compared with the preset value parameter feature W V6 The sixth value feature V6 and the preset key parameter feature W are obtained by calculation. K6 The sixth key feature K6 is obtained by calculation. Specifically, according to the three preset parameter features W Q6 , W K6 and W Q6 The realization process of obtaining the above three features Q6, K6 and V6 can refer to the above Figure 5 Description.

[0175] When determining the intermediate state of global decoding, a softmax operation is performed according to formula (6) to obtain the weight A6 of the global encoding result of each word in multiple source sequences for the current word:

[0176]

[0177] Among them, Q6 represents the sixth query feature in the global self-attention network, Represents the matrix transposition result of the sixth feature K6 in the global self-attention network, d / h is a constant;

[0178] The operation obtains the weight A6 of the global encoding results of each word corresponding to "Julia saw a cat. It was hungry." for "was", and then the sixth query feature V6 is weighted summed according to the weight A6 to obtain the global decoding intermediate state corresponding to "was".

[0179] In a specific embodiment, the global self-attention network may include a multi-head self-attention network, each head self-attention network according to the global encoding results of each word corresponding to "Julia saw a cat. It was hungry." and the global target encoding result of the current word and the preset query parameter feature W Q6 , preset value parameter feature W V6 and the preset key parameter characteristics W K6 Perform calculation processing to obtain a set of sixth query feature Q6, sixth value feature V6 and sixth key feature K6, perform point multiplication operation on the sixth query feature Q6 and the sixth key feature K6 to obtain the point multiplication result, perform softmax operation on the obtained point multiplication result to obtain the weight A6 of each word corresponding to "Julia saw a cat. It was hungry." For "was", perform weighted summation processing on the sixth value feature V6 according to the weight A6 to obtain the global hidden layer decoding result of "was". Perform splicing operation on each global hidden layer decoding result obtained in the multi-head self-attention network to obtain the final global hidden layer decoding result of "was", and perform unidirectional propagation on the final global hidden layer decoding result through the feedforward neural network 113 connected to the output end of the global self-attention network to obtain the global decoding intermediate state of "was".

[0180] It should be noted that, in the embodiment of the present invention, starting from the second layer encoder 11, the input of its local encoding module 111 is the output of the local encoding module 111 from the previous layer encoder 11, and the input of its global encoding module 112 is also the output of the local encoding module 111 from the previous layer encoder 11. Fig.11 As shown, when the encoder 11 is not the first layer encoder 11, during the encoding process, the first key feature K1 and the first value feature V1 of the local encoding module 111 are the local encoding results corresponding to each word in the current sequence "It was hungry." where "was" is located, obtained by the local encoding module 111 of the previous layer encoder 11, respectively, and the preset query parameter feature W K1 and preset value parameter characteristic W V1 The local encoding result corresponding to the first query feature Q1 of the local encoding module 111 is “was” and the preset value parameter feature W Q1 The second key feature K2 and the second value feature V2 of the global encoding module 112 are the local encoding results corresponding to each word in the multiple source sequences "Julia saw a cat. It was hungry." obtained in the local encoding module 111 of the previous layer encoder 11 and the preset query parameter feature W K2 and preset value parameter characteristic W V1The global encoding result corresponding to the second query feature Q2 of the global encoding module 112 is the value parameter feature W corresponding to the global encoding result corresponding to the second query feature Q2 of the global encoding module 112. Q2 Obtained by calculation.

[0181] Correspondingly, starting from the second layer decoder 21, the input of its local target encoding module 211 is the output of the local self-attention processing module 212 of the previous layer decoder 21, the input of its local self-attention processing module 212 is the output of the local encoding module 111 of the corresponding layer encoder 11 and the output of the local target encoding module 211 of the current layer decoder 21, and the input of its global target encoding module 213 also comes from the output of the global self-attention processing module 214 of the previous layer decoder 21. The input of its local target encoding module 211 is the output of the local self-attention processing module 212 of the previous layer decoder 21, and the input of its global self-attention processing module 214 is the output of the global encoding module 112 of the corresponding layer encoder 11 and the output of the global target encoding module 213 of the current layer decoder 21.

[0182] Reference Fig.12 As shown, when the decoder 21 is not the first layer decoder 21, in the decoding process, the third key feature K3 and the third value feature V3 of the local target encoding module 211 are the local decoding intermediate states corresponding to each target word in the target end sequence where the target word "was" is located obtained in the local self-attention processing module 212 of the previous layer decoder 21, respectively, and the preset query parameter feature W K3 and preset value parameter characteristic W V3 The local target encoding module 211 obtains the third query feature Q3 by performing the calculation, and the local decoding intermediate state corresponding to “It” and the preset value parameter feature W Q3 Obtained by calculation;

[0183] Reference Fig.12 As shown, the fourth key feature K4 and the fourth value feature V4 of the local self-attention processing module 212 are the local encoding results of each word in the current sequence where "was" is located obtained in the local encoding module 111 of the corresponding layer encoder 11, respectively, and the preset query parameter feature W K4 and preset value parameter characteristic W V4 The fourth query feature Q4 of the local self-attention processing module 212 is the target word corresponding to "was", that is, "the local target encoding result of It" and the preset value parameter feature W Q4 Obtained by calculation;

[0184] Reference Fig.11As shown, the fifth key feature K5 and the fifth value feature V5 of the global target encoding module 213 are the local decoding intermediate states corresponding to the target words before the target word "was" in the target end sequence obtained in the global self-attention processing module 214 of the previous layer decoder 21, respectively, and the preset query parameter feature W K5 and preset value parameter characteristic W V5 The fifth query feature Q5 of the global target encoding module 213 is the global decoding intermediate state corresponding to “It” and the preset value parameter feature W Q5 Obtained by calculation;

[0185] Reference Fig.13 As shown, the sixth key feature K6 and the sixth value feature V6 of the local self-attention processing module 212 are the global encoding results of each word in the multiple source end sequences obtained in the global encoding module 112 of the corresponding layer encoder 11, respectively, and the preset query parameter feature W K6 and preset value parameter characteristic W V6 The sixth query feature Q6 of the global self-attention processing module 214 is the target word corresponding to “was”, that is, “the global target encoding result of It” and the preset value parameter feature W Q6 Obtained by calculation.

[0186] Finally, the obtained “local decoding intermediate state of was” and “global decoding intermediate state of was” are fused to finally obtain the decoding result of “was”. Then, the decoding result of “was” is subjected to Softmax operation to obtain the final output possibility result, that is, the translation result of “was”.

[0187] Based on the same inventive concept, an embodiment of the present invention further provides a method for translating a passage, comprising:

[0188] For each paragraph of the article, a plurality of source sentence segments are obtained according to the reading order of the sentences; the source sentence segments include one or more source sentences;

[0189] For each source sentence segment, the neural network machine translation method described in Example 1 or Example 2 is executed to obtain a corresponding target sentence.

[0190] In the embodiment of the present invention, the step of obtaining multiple source sentence fragments according to the reading order of sentences for each paragraph of the chapter can refer to the specific implementation process in the prior art and is not specifically limited here.

[0191] Based on the same inventive concept, an embodiment of the present invention further provides a method for translating pictures and texts, comprising:

[0192] Recognize the text information in the acquired image to obtain the source sentence corresponding to the text information in the image;

[0193] For the source sentence corresponding to the text information in the picture, the neural network machine translation method described in Example 1 or Example 2 is executed to obtain the target sentence corresponding to the text information in the picture.

[0194] In the embodiment of the present invention, the step of performing recognition on the text information in the acquired image to obtain the source sentence corresponding to the text information in the image can refer to the specific implementation process in the prior art and is not specifically limited here.

[0195] Based on the same inventive concept, an embodiment of the present invention further provides a speech translation method, comprising:

[0196] Acquire voice information, and perform recognition to obtain source sentences corresponding to the voice information;

[0197] For the source sentence corresponding to the voice information, the neural network machine translation method described in Example 1 or Example 2 is executed to obtain the target sentence corresponding to the voice information.

[0198] In the embodiment of the present invention, the steps of acquiring voice information and identifying the source sentence corresponding to the voice information can refer to the specific implementation process in the prior art and are not specifically limited here.

[0199] Based on the same inventive concept, the embodiment of the present invention also provides a neural network translation device, related storage media and translation equipment. Since the principles of the problems solved by these systems, related storage media and equipment are similar to those of the aforementioned text recognition method, the implementation of the systems, related storage media and equipment can refer to the implementation of the aforementioned method, and the repeated parts will not be repeated.

[0200] Reference Figure 2 As shown, an embodiment of the present invention provides a neural network machine translation device, including: an encoding end 1 and a decoding end 2;

[0201] The encoding end 1 includes at least one encoder 11;

[0202] The decoding end 2 has at least one decoder 21;

[0203] The encoder 11 is used to encode the multiple source end sequences obtained to obtain a global encoding result of the current word, and encode the current sequence where the current word is located in the multiple source end sequences to obtain a local encoding result of the current word;

[0204] The decoder 12 is used to decode according to the target end sequence corresponding to the translated target sentence, the local encoding result and the global encoding result corresponding to the current word, to obtain the local decoding intermediate state and the global decoding intermediate state corresponding to the current word; and to fuse the local decoding intermediate state and the global decoding intermediate state of the current word to obtain the decoding result of the current word.

[0205] In one embodiment, the encoder 11 includes: a local encoding module 111 and a global encoding module 112;

[0206] The local encoding module 111 is used to obtain a local encoding result of the current word according to the context information of the current word in the current sequence in the multiple source end sequences;

[0207] The global encoding module 112 obtains a global encoding result of the current word according to the context information of the current word in the multiple source sequences and the local encoding results of each word in the multiple source sequences.

[0208] In one embodiment, the decoder 21 includes: a local self-attention processing module 212, a global self-attention processing module 214;

[0209] The local self-attention processing module 212 is used to obtain the local decoding intermediate state of the current word according to the target end sequence corresponding to the current sequence in the target end sequence corresponding to the translated target sentence and the local encoding results of each word in the current sequence;

[0210] The global self-attention processing module 214 is used to obtain the global decoding intermediate state of the current word according to the target end sequence corresponding to the translated target sentence and the global encoding results of each word in the multiple source end sequences.

[0211] In one embodiment, the local encoding module 111 is specifically configured to obtain a first query feature, a first value feature, and a first key feature according to a preset first parameter feature and context information in the current sequence;

[0212] The current word is subjected to self-attention encoding processing according to the first query feature, the first value feature and the first key feature to obtain a local encoding result of the current word.

[0213] In one embodiment, the global encoding module 112 is specifically configured to obtain a second query feature, a second value feature, and a second key feature according to a preset second parameter feature and a local encoding result of each word in the plurality of source end sequences;

[0214] The current word is subjected to self-attention encoding processing according to the second query feature, the second value feature and the second key feature to obtain a global encoding result of the current word.

[0215] In one embodiment, the neural network machine translation device further includes a local target encoding module 211, which is used to determine a target word corresponding to the current word in the target end sequence corresponding to the current sequence, and perform self-attention encoding processing on the target word corresponding to the current word according to the previous context information of the target word in the target end sequence corresponding to the current sequence to obtain a local target encoding result;

[0216] The local self-attention processing module 212 is specifically used to perform self-attention decoding processing on the current word according to the local encoding results of each word in the current sequence and the local target encoding results to obtain the local decoding intermediate state of the current word.

[0217] In one embodiment, the local intermediate module 211 is specifically used to determine a target end sequence corresponding to the current sequence, and search and determine a target word corresponding to the current word in the target end sequence corresponding to the current sequence;

[0218] Obtaining a third query feature, a third value feature, and a third key feature according to a preset third parameter feature and previous text information of a target word corresponding to the current word in a target end sequence corresponding to the current sequence;

[0219] The target word corresponding to the current word is subjected to self-attention encoding processing according to the third query feature, the third value feature and the third key feature to obtain the local target encoding result.

[0220] In one embodiment, the local self-attention processing module 212 is specifically used to obtain a fourth query feature, a fourth value feature and a fourth key feature according to a preset fourth parameter feature, the local target encoding result and the local encoding result of each word in the current sequence;

[0221] The current word is subjected to self-attention decoding processing according to the fourth query feature, the fourth value feature and the fourth key feature to obtain a local decoding intermediate state of the current word.

[0222] In one embodiment, the neural network machine translation device further includes a global target encoding module 213, which is used to determine a target word corresponding to the current word in the target end sequence corresponding to the translated target sentence, and perform self-attention encoding processing on the target word corresponding to the current word according to the previous context information of the target word in the target end sequence corresponding to the translated target sentence to obtain a global target encoding result;

[0223] The global self-attention processing module 214 is specifically used to perform self-attention decoding processing on the current word according to the global encoding results of each word in the multiple source sequences and the global target encoding results to obtain the global decoding intermediate state of the current word.

[0224] In one embodiment, the global intermediate module 213 is specifically used to determine the target end sequence corresponding to the translated target sentence, and search and determine the target word corresponding to the current word in the target end sequence corresponding to the translated target sentence;

[0225] According to the preset fifth parameter feature and the previous text information of the target word corresponding to the current word in the target end sequence corresponding to the translated target sentence, a fifth query feature, a fifth value feature and a fifth key feature are obtained;

[0226] The target word corresponding to the current word is subjected to self-attention encoding processing according to the fifth query feature, the fifth value feature and the fifth key feature to obtain a global target encoding result of the current word.

[0227] In one embodiment, the global self-attention processing module 214 is specifically used to obtain a sixth query feature, a sixth value feature and a sixth key feature according to a preset sixth parameter feature, the global target encoding result and the global encoding result of each word in the plurality of source end sequences;

[0228] The current word is subjected to self-attention decoding processing according to the sixth query feature, the sixth value feature and the sixth key feature to obtain a global decoding intermediate state of the current word.

[0229] In one embodiment, the neural network machine translation device further includes a word encoder 12, which is used to obtain a word feature sequence and position information of each word in the received multiple source sentences;

[0230] Encoding the word feature sequence and position information of each word in the plurality of source sentences to obtain the plurality of source end sequences;

[0231] The target word encoder 22 is used to obtain the word feature sequence and position information of each word in the translated target sentence; encode the word feature sequence and position information of each word in the translated target sentence to obtain the target end sequence.

[0232] In one embodiment, the neural network machine translation device further includes a user interaction module, which is used to display the translated target sentence to the user during the translation process;

[0233] A word modified by a user on at least one target word in the translated target sentence is received to obtain a modified translated target sentence.

[0234] In one embodiment, the user interaction module is further used to display the decoding result to the user for modification by the user.

[0235] The embodiment of the present invention further provides a paragraph translation system, comprising: a source sentence acquisition device and the above-mentioned neural network machine translation device;

[0236] A source sentence acquisition device is used to obtain a plurality of source sentence segments according to the reading order of sentences for each paragraph of the article; the source sentence segments include one or more source sentences;

[0237] The neural network machine translation device is used to execute the neural network machine translation method described in the first or second embodiment for each source sentence segment to obtain a corresponding target sentence.

[0238] The embodiment of the present invention also provides a picture-text translation system, comprising: a picture-text recognition device and the above-mentioned neural network machine translation device;

[0239] A picture text recognition device is used to recognize the text information in the acquired picture and obtain the source sentence corresponding to the text information in the picture;

[0240] A neural network machine translation device is used to execute the neural network machine translation method described in Example 1 or Example 2 for a source sentence corresponding to the text information in the picture, so as to obtain a target sentence corresponding to the text information in the picture.

[0241] The embodiment of the present invention also provides a speech translation method, comprising: a speech recognition device and the above-mentioned neural network machine translation device;

[0242] A speech recognition device, used to obtain speech information and perform recognition to obtain a source sentence corresponding to the speech information;

[0243] A neural network machine translation device is used to execute the neural network machine translation method described in Example 1 or Example 2 for a source sentence corresponding to the voice information to obtain a target sentence corresponding to the voice information.

[0244] In the embodiment of the present invention, the steps of acquiring voice information and identifying the source sentence corresponding to the voice information can refer to the specific implementation process in the prior art and are not specifically limited here.

[0245] An embodiment of the present invention also provides a computer-readable storage medium having computer instructions stored thereon, which, when executed by a processor, implement the above-mentioned neural network machine translation method, passage translation method, picture-text translation method or speech translation method.

[0246] An embodiment of the present invention also provides a translation device, comprising: a processor and a memory for storing processor executable commands; wherein the processor is configured to execute the above-mentioned neural network machine translation method, paragraph translation method, picture-text translation method or speech translation method.

[0247] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage and optical storage, etc.) containing computer-usable program code.

[0248] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0249] These computer program instructions may also be stored in a computer readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0250] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process in the computer or other programmable device. Figure 1 A process or multiple processes and / or boxes Figure 1The steps for the functions specified in one or more boxes.

[0251] Obviously, those skilled in the art can make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these modifications and variations.

Claims

1. A neural network machine translation method, include: Performing encoding processing on the obtained multiple source end sequences to obtain a global encoding result of the current word, and performing encoding processing on the current sequence where the current word is located in the multiple source end sequences to obtain a local encoding result of the current word; Decoding is performed according to the target end sequence corresponding to the translated target sentence, the local encoding result and the global encoding result of the current word to obtain the local decoding intermediate state and the global decoding intermediate state of the current word; The local decoding intermediate state and the global decoding intermediate state are fused to obtain a decoding result of the current word.

2. According to the method of claim 1, encoding the obtained multiple source end sequences to obtain a global encoding result of the current word, encoding the current sequence where the current word is located in the multiple source end sequences to obtain a local encoding result of the current word, include: Obtaining a local encoding result of the current word according to the context information of the current word in the current sequence of the multiple source end sequences; A global encoding result of the current word is obtained according to the context information of the current word in the multiple source sequences and the local encoding results of each word in the multiple source sequences.

3. The method according to claim 2, wherein decoding is performed according to the target end sequence corresponding to the translated target sentence, the local encoding result and the global encoding result of the current word to obtain the local decoding intermediate state and the global decoding intermediate state of the current word, include: Obtaining a local decoding intermediate state of the current word according to a target-side sequence corresponding to the current sequence in a target-side sequence corresponding to the translated target sentence and a local encoding result of each word in the current sequence; According to the global encoding results of the target-end sequence corresponding to the translated target sentence and each word in the multiple source-end sequences, a global decoding intermediate state of the current word is obtained.

4. The method according to claim 2 or 3, wherein the local encoding result of the current word is obtained according to the context information of the current word in the current sequence in the multiple source end sequences, include: Obtaining a first query feature, a first value feature, and a first key feature according to a preset first parameter feature and context information in the current sequence; The current word is subjected to self-attention encoding processing according to the first query feature, the first value feature and the first key feature to obtain a local encoding result of the current word.

5. The method according to claim 4, wherein the global encoding result of the current word is obtained according to the context information of the current word in the multiple source sequences and the local encoding results of each word in the multiple source sequences, include: Obtaining a second query feature, a second value feature, and a second key feature according to a preset second parameter feature and a local encoding result of each word in the plurality of source end sequences; The current word is subjected to self-attention encoding processing according to the second query feature, the second value feature and the second key feature to obtain a global encoding result of the current word.

6. The method according to claim 5, wherein the local decoding intermediate state of the current word is obtained according to the target end sequence corresponding to the current sequence in the target end sequence corresponding to the translated target sentence and the local encoding results of each word in the current sequence, include: Determine a target word corresponding to the current word in a target end sequence corresponding to the current sequence, and perform self-attention encoding processing on the target word corresponding to the current word according to previous context information of the target word in the target end sequence corresponding to the current sequence to obtain a local target encoding result; According to the local encoding results of each word in the current sequence and the local target encoding result, the current word is subjected to self-attention decoding processing to obtain the local decoding intermediate state of the current word.

7. The method according to claim 6, wherein the target word corresponding to the current word in the target end sequence corresponding to the current sequence is determined, and the target word corresponding to the current word is self-attentionally encoded according to the previous information of the target word in the target end sequence corresponding to the current sequence to obtain a local target encoding result, include: Determine a target end sequence corresponding to the current sequence, and search and determine a target word corresponding to the current word in the target end sequence corresponding to the current sequence; Obtaining a third query feature, a third value feature, and a third key feature according to a preset third parameter feature and previous text information of a target word corresponding to the current word in a target end sequence corresponding to the current sequence; The target word corresponding to the current word is subjected to self-attention encoding processing according to the third query feature, the third value feature and the third key feature to obtain the local target encoding result.

8. The method according to claim 7, wherein the current word is subjected to self-attention decoding processing according to the local encoding results of each word in the current sequence and the local target encoding result to obtain the local decoding intermediate state of the current word, include: Obtaining a fourth query feature, a fourth value feature and a fourth key feature according to a preset fourth parameter feature, the local target encoding result and the local encoding results of each word in the current sequence; The current word is subjected to self-attention decoding processing according to the fourth query feature, the fourth value feature and the fourth key feature to obtain a local decoding intermediate state of the current word.

9. The method according to claim 5, obtaining the global decoding intermediate state of the current word according to the target end sequence corresponding to the translated target sentence and the global encoding results of each word in the multiple source end sequences, include: Determine a target word corresponding to the current word in the target-side sequence corresponding to the translated target sentence, and perform self-attention encoding processing on the target word corresponding to the current word according to the previous context information of the target word in the target-side sequence corresponding to the translated target sentence to obtain a global target encoding result; According to the global encoding results of each word in the multiple source sequences and the global target encoding result, the current word is subjected to self-attention decoding processing to obtain a global decoding intermediate state of the current word.

10. The method according to claim 9, wherein the target word corresponding to the current word in the target end sequence corresponding to the translated target sentence is determined, and according to the previous context information of the target word in the target end sequence corresponding to the translated target sentence, the target word corresponding to the current word is self-attentionally encoded to obtain a global target encoding result, include: Determine a target end sequence corresponding to the target sentence that has been translated, and search and determine a target word corresponding to the current word in the target end sequence corresponding to the target sentence that has been translated; According to the preset fifth parameter feature and the previous text information of the target word corresponding to the current word in the target end sequence corresponding to the translated target sentence, a fifth query feature, a fifth value feature and a fifth key feature are obtained; The target word corresponding to the current word is subjected to self-attention encoding processing according to the fifth query feature, the fifth value feature and the fifth key feature to obtain a global target encoding result of the current word.

11. The method according to claim 10, wherein the self-attention decoding process is performed on the current word according to the global encoding results of each word in the multiple source sequences and the global target encoding result to obtain the global decoding intermediate state of the current word, include: Obtaining a sixth query feature, a sixth value feature, and a sixth key feature according to a preset sixth parameter feature, the global target encoding result, and the global encoding results of each word in the plurality of source end sequences; The current word is subjected to self-attention decoding processing according to the sixth query feature, the sixth value feature and the sixth key feature to obtain a global decoding intermediate state of the current word.

12. The method according to claim 1, further comprising: include: The multiple source end sequences are obtained by: Obtaining word feature sequences and position information of each word in the received multiple source sentences; Encoding the word feature sequence and position information of each word in the multiple source sentences to obtain the multiple source end sequences; and obtaining the target end sequence corresponding to the translated target sentence in the following manner: Obtaining word feature sequences and position information of each word in the translated target sentence; The word feature sequence and position information of each word in the translated target sentence are encoded to obtain the target end sequence.

13. The method according to claim 12, before executing the step of obtaining the target end sequence corresponding to the target sentence that has been translated, further comprising: include: During the translation process, the translated target sentence is displayed to the user; A word modified by a user on at least one target word in the translated target sentence is received to obtain a modified translated target sentence.

14. The method according to claim 1, further comprising: include: The decoding result is displayed to the user for modification by the user.

15. A method for text translation, include: For each paragraph of the article, multiple source sentence fragments are obtained according to the reading order of the sentences; The source sentence fragment includes one or more source sentences; For each source sentence fragment, the neural network machine translation method described in any one of claims 1 to 14 is executed to obtain a corresponding target sentence.

16. A method for translating pictures and texts. include: Recognize the text information in the acquired image to obtain the source sentence corresponding to the text information in the image; For the source sentence corresponding to the text information in the picture, execute the neural network machine translation method described in any one of claims 1 to 14 to obtain the target sentence corresponding to the text information in the picture.

17. A speech translation method, include: Acquire voice information, and perform recognition to obtain source sentences corresponding to the voice information; For the source sentence corresponding to the speech information, execute the neural network machine translation method described in any one of claims 1 to 14 to obtain the target sentence corresponding to the speech information.

18. A neural network machine translation device, include: Encoding end and decoding end; The encoding end includes at least one encoder; The decoding end has at least one decoder; The encoder is used to encode the multiple source end sequences obtained to obtain a global encoding result of the current word, and encode the current sequence where the current word is located in the multiple source end sequences to obtain a local encoding result of the current word; The decoder is used to decode according to the target end sequence corresponding to the translated target sentence, the local encoding result and the global encoding result corresponding to the current word, to obtain the local decoding intermediate state and the global decoding intermediate state corresponding to the current word; and to fuse the local decoding intermediate state and the global decoding intermediate state of the current word to obtain the decoding result of the current word.

19. A computer-readable storage medium having computer instructions stored thereon, wherein when the instructions are executed by a processor, the method according to any one of claims 1 to 17 is implemented.

20. A translation device, include: A processor and a memory for storing processor executable instructions; wherein the processor is configured to execute the method as described in any one of claims 1-17.