Neural machine translation system, method, electronic device and readable storage medium

Abstract

The present disclosure provides a neural machine translation system, including: a standard Transformer multi-layer encoder and a multi-layer decoder; a chapter structure analysis module, which is arranged at the top layer of the encoder, for modeling the dependency structure information of the chapter, and obtaining an EDU representation vector with enhanced structure information; an encoding-side EDU fusion module, which fuses the EDU representation vector with the encoder output state vector; and a decoding-side EDU fusion module, which fuses the EDU representation vector with the decoder output state vector. The present disclosure also provides a neural machine translation method, an electronic device, and a readable storage medium.

Description

Technical Field

[0001] The present disclosure belongs to the field of language processing technology, and in particular, relates to a neural machine translation system, method, electronic device, and readable storage medium. Background Art

[0002] In paragraph-level neural machine translation, contextual information in the paragraph is crucial for document-level translation tasks. In many real application scenarios, the input of the translation system is often a paragraph or a document that is related to each other, such as legal documents, scientific literature, chat records, etc. Cross-sentence contextual information can provide appropriate paragraph context for the translation of the current sentence, enhance the semantic encoding and decoding representation of the current sentence, and assist the model in correcting some translation errors, thereby generating more accurate and coherent paragraph translations.

[0003] In addition to the cohesion phenomenon such as word repetition, as an organized and complete language unit, the text also has the characteristics of structure, which can be formally represented as a topological structure composed of semantically related basic text units. However, most of the current text-level neural machine translation methods only consider the contextual lexical sequence information and ignore the use of text structure information. Therefore, in neural machine translation, modeling the structural information represented by text theory and effectively integrating it with the neural machine translation model can provide richer linguistic context information for translation, so as to achieve the purpose of improving translation quality. Summary of the invention

[0004] In order to solve at least one of the above technical problems, the present disclosure provides a neural machine translation system, method, electronic device and readable storage medium.

[0005] According to one aspect of the present disclosure, a neural machine translation system is provided, including: a standard Transformer multi-layer encoder and a multi-layer decoder; a text structure parsing module, which is arranged at the top layer of the encoder and is used to model the dependency structure information of the text to obtain an EDU representation vector with enhanced structural information; an encoding-side EDU fusion module, which fuses the EDU representation vector with the encoder output state vector; and a decoding-side EDU fusion module, which fuses the EDU representation vector with the decoder output state vector.

[0006] According to the neural machine translation system of at least one embodiment of the present disclosure, the paragraph structure parsing module generates a set of initial EDU representation vectors for the input word state vector after multi-layer Transformer encoding through EDU segmentation and maximum pooling operations, and then uses the paragraph structure-aware multi-head self-attention mechanism to model the dependency structure information, and outputs the EDU representation vector with enhanced structural information.

[0007] According to the neural machine translation system of at least one embodiment of the present disclosure, the EDU segmentation is performed in word x i，j The probability of splitting afterwards is: S i，j =σ(W s h i，j +b s ),in and b s is the model parameter, σ(·) represents the sigmoid activation function; when S i，i When x is greater than a fixed threshold, i，j Then perform EDU segmentation.

[0008] According to the neural machine translation system of at least one embodiment of the present disclosure, the maximum pooling operation generates an initial EDU vector: m =MaxPooling(H m ), where H m is the word encoding state vector contained in the mth EDU; MaxPooling(·) is the maximum pooling function.

[0009] According to the neural machine translation system of at least one embodiment of the present disclosure, the paragraph structure-aware multi-head attention mechanism adds an EDU dependency head to model the EDU dependency structure and relationship on the basis of the standard multi-head attention mechanism, wherein the output DSAMHA of the multi-head attention mechanism is calculated as follows:

[0010] DSAMHA(Q, K, V) = [head DEP ;head 1 ;…;head N ]W o ,

[0011] Among them, input Q, K, V are the initial EDU vectors, W O is the model parameter; ordinary head 1 , ..., head N The output of} is: Where W O , W Q n , W K n , W V n are model parameters;

[0012] ATTN is the attention calculation formula,

[0013] Where d represents the dimension of the hidden state vector of K. The ATTN function calculates the dot product of the query vector Q and the key vector K and normalizes it through softmax as the attention weight A. Then, the attention is used to perform weighted summation on the value vector V to obtain the final output.

[0014] For head DEP The calculation uses a double affine operation:

[0015] head DEP =softmax(Q DEP UK DEPT )V DEP =A DEP V DEP ,in and U are model parameters; attention weight matrix ADEP The dimension is M×M, and the mth row vector in the matrix represents e m The relationship with each EDU in the passage, including itself.

[0016] According to the neural machine translation system of at least one embodiment of the present disclosure, the EDU fusion module at the encoding end performs a i，j , source word encoding state vector h i，j Its EDU vector The fusion process is as follows: Where FNN(·) represents a feedforward neural network, σ(·) represents a sigmoid activation function, [·;·] represents a vector concatenation operation, and the gate weight r i，j It is used to balance the encoding state information of the word and its EDU information. The word encoding state vector after integrating the EDU information is

[0017] According to the neural machine translation system of at least one embodiment of the present disclosure, the decoding end EDU fusion module is for a decoding state vector g output by the Transformer multi-layer decoder t , in g t The process of integrating EDU information in the .

[0018] S1, respectively calculate the EDU-level attention weight and word-level attention weight of the decoding state vector gt to the source end;

[0019] S2, use the EDU level attention weight to modify the word level attention weight, and then generate g t The vector representation g of the global context t ';

[0020] S3, using gated weighting to fuse the original decoding vector gt and g t ', generate the final decoding state vector of the fused EDU information Used to predict the probability distribution of the target vocabulary.

[0021] According to the neural machine translation system of at least one embodiment of the present disclosure, in step S1, the calculation formula of the EDU level attention weight is:

[0022]

[0023] in, and About GT and The linear change of EDU representation vector enhanced with structural information, are model parameters, The dimension is M;

[0024] The word-level attention weight calculation formula is:

[0025]

[0026] in is the word encoding state vector matrix integrating EDU information, and its dimension is IJ×d, and are model parameters, The dimension is IJ.

[0027] According to the neural machine translation system of at least one embodiment of the present disclosure, in step S2, g t The calculation process of ' is as follows:

[0028]

[0029] in

[0030] According to the neural machine translation system of at least one embodiment of the present disclosure, in step S3, the decoding state vector fused with EDU information The fusion process is as follows:

[0031] r′ t =σ(FFN(g r ; g′ t ]))

[0032] FNN(·) represents a feed-forward neural network, σ(·) represents the sigmoid activation function, and [·;·] represents a vector concatenation operation.

[0033] According to another aspect of the present disclosure, a neural machine translation method is provided, which uses any of the neural machine translation systems described above to perform neural machine translation.

[0034] According to another aspect of the present disclosure, there is provided an electronic device, including:

[0035] a memory storing execution instructions; and

[0036] A processor, wherein the processor executes the execution instructions stored in the memory, so that the processor executes the above-mentioned neural machine translation method.

[0037] According to another aspect of the present disclosure, a readable storage medium is provided, characterized in that the readable storage medium stores execution instructions, and the execution instructions are used to implement the above-mentioned neural machine translation method when executed by a processor. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] The accompanying drawings illustrate exemplary embodiments of the present disclosure and together with the description serve to explain the principles of the present disclosure. These drawings are included to provide a further understanding of the present disclosure and are incorporated in and constitute a part of this specification.

[0039] Figure 1 It is a structural diagram of a neural machine translation system according to an embodiment of the present disclosure.

[0040] Figure 2 It is a structural diagram of a chapter structure analysis module according to an embodiment of the present disclosure.

[0041] Figure 3 It is a flowchart of a decoding-end fusion module fusing an EDU representation vector with a decoder output state vector according to an embodiment of the present disclosure.

[0042] Figure 4 One embodiment of the present disclosure is a neural machine translation system performing a neural machine translation method.

[0043] Figure 5 It is a schematic diagram of the structure of a neural machine translation device according to one embodiment of the present disclosure.

[0044] Reference numerals

[0045] 1001 Transformer Model

[0046] 1002 Chapter structure analysis module

[0047] 1003 Encoding end EDU fusion module

[0048] 1004 Decoding end EDU fusion module

[0049] 1000 Neural Machine Translation Device

[0050] 1100 Bus

[0051] 1200 processor

[0052] 1300 Memory

[0053] 1400 Other circuits. DETAILED DESCRIPTION

[0054] The present disclosure is further described in detail below in conjunction with the accompanying drawings and implementations. It is understood that the specific implementations described herein are only used to explain the relevant content, rather than to limit the present disclosure. It should also be noted that, for ease of description, only the parts related to the present disclosure are shown in the accompanying drawings.

[0055] It should be noted that, in the absence of conflict, the embodiments and features of the embodiments in the present disclosure can be combined with each other. The technical solution of the present disclosure will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0056] Unless otherwise specified, the exemplary embodiments / embodiments shown will be understood as providing exemplary features of various details of some ways in which the technical concept of the present disclosure can be implemented in practice. Therefore, unless otherwise specified, the features of the various embodiments / embodiments can be combined, separated, interchanged and / or rearranged without departing from the technical concept of the present disclosure.

[0057] The use of cross-hatching and / or shading in the accompanying drawings is generally used to make the boundaries between adjacent components clear. As such, unless otherwise specified, the presence or absence of cross-hatching or shading does not convey or indicate any preference or requirement for the specific materials, material properties, dimensions, proportions, commonalities between the components shown, and / or any other characteristics, attributes, properties, etc. of the components. In addition, in the accompanying drawings, the sizes and relative sizes of the components may be exaggerated for clarity and / or descriptive purposes. When the exemplary embodiments can be implemented differently, the specific process sequence can be performed in a different order than described. For example, two successively described processes can be performed substantially simultaneously or in an order opposite to the described order. In addition, the same figure numbers represent the same components.

[0058] When a component is referred to as being "on" or "over," "connected to," or "coupled to" another component, the component may be directly on, directly connected to, or directly coupled to the other component, or intervening components may be present. However, when a component is referred to as being "directly on," "directly connected to," or "directly coupled to" another component, there are no intervening components. For this purpose, the term "connected" may refer to a physical connection, an electrical connection, etc., with or without intervening components.

[0059] For descriptive purposes, the present disclosure may use spatially relative terms such as "under," "beneath," "under," "down," "over," "upper," "above," "higher," and "side (e.g., as in "sidewall")," to describe the relationship of one component to another (other) component as shown in the accompanying drawings. The spatially relative terms are intended to encompass different orientations of the device in use, operation, and / or manufacture in addition to the orientation depicted in the accompanying drawings. For example, if the device in the accompanying drawings is turned over, components described as "under" or "beneath" other components or features would subsequently be positioned "over" the other components or features. Thus, the exemplary term "under" can encompass both the "above" and "below" orientations. Furthermore, the device may be otherwise oriented (e.g., rotated 90 degrees or at other orientations), and the spatially relative descriptors used herein should be interpreted accordingly.

[0060] The terms used here are for the purpose of describing specific embodiments, and are not intended to be restrictive. As used here, unless the context clearly indicates otherwise, the singular forms "one (kind, person)" and "said (the)" are also intended to include plural forms. In addition, when the terms "comprise" and / or "include" and their variations are used in this specification, it is explained that there are stated features, integral bodies, steps, operations, parts, assemblies and / or their groups, but it is not excluded that there are or add one or more other features, integral bodies, steps, operations, parts, assemblies and / or their groups. It should also be noted that, as used here, the terms "substantially", "approximately" and other similar terms are used as approximate terms and not as degree terms, so that they are used to explain the inherent deviations of the measured values, calculated values ​​and / or the values ​​provided that will be recognized by those of ordinary skill in the art.

[0061] Figure 1It is a schematic diagram of the structure of a neural machine translation system of an embodiment of the present disclosure, in which the residual connection and layer normalization operation layer are not shown. The neural machine translation system of the embodiment of the present disclosure includes: a standard Transformer multi-layer encoder and a multi-layer decoder; a chapter structure parsing module, which is arranged at the top layer of the encoder and is used to model the dependency structure information of the chapter and obtain an EDU representation vector with enhanced structure information; an EDU fusion module at the encoding end, which fuses the EDU representation vector with the encoder output state vector; and an EDU fusion module at the decoding end, which fuses the EDU representation vector with the decoder output state vector.

[0062] The standard Transformer model is an end-to-end encoder-decoder framework. The multi-layer encoder encodes the source language sentence into a continuous semantic vector representation, and the multi-layer decoder decodes the target language sentence word by word from the semantic vector representation.

[0063] The chapter structure parsing module is set at the top level of the encoder to model the dependency structure information of the chapter and obtain the EDU representation vector with enhanced structural information.

[0064] The chapter structure analysis module of the embodiment of the present disclosure can be Figure 2 Specifically, the chapter structure parsing module takes the input word state vector after multi-layer encoding of Transformer, generates a set of initial EDU representation vectors through EDU segmentation and maximum pooling operations, and then uses the multi-head self-attention mechanism of chapter structure perception to model the dependency structure information, and outputs the EDU representation vector with enhanced structure information.

[0065] Preferably, in this embodiment, the EDU segmentation of the chapter structure analysis module is performed in the following manner: i,j The probability of splitting afterwards is: s i，j =σ(W s h i，j +b s ),in and b s is the model parameter, σ(·) represents the sigmoid activation function; when S i,j When x is greater than a fixed threshold, i,j Then perform EDU segmentation.

[0066] Preferably, in this embodiment, the maximum pooling operation generates the initial EDU vector:

[0067] e m =MaxPooling(H m ), where H mis the word encoding state vector contained in the mth EDU; MaxPooling(·) is the maximum pooling function.

[0068] Preferably, in this embodiment, the multi-head attention mechanism of the chapter structure perception used by the chapter structure parsing module adds an EDU dependency head to model the EDU dependency structure and relationship on the basis of the standard multi-head attention mechanism, wherein the output DSAMHA of the multi-head attention mechanism is calculated as follows:

[0069] DSAMHA(Q, K, V) = [head DEP ;head 1 ;…;head N ]W o ,

[0070] Among them, input Q, K, V are the initial EDU vectors, W O is the model parameter; ordinary head 1 , ..., head N The output of} is: Where W o , W Q n , W K n , W V n are model parameters;

[0071] ATTN is the attention calculation formula,

[0072] Where d represents the dimension of the hidden state vector of K. The ATTN function calculates the dot product of the query vector Q and the key vector K and normalizes it through softmax as the attention weight A. Then, the attention is used to perform weighted summation on the value vector V to obtain the final output.

[0073] For head DEP The calculation uses a double affine operation:

[0074] in and U are model parameters; attention weight matrix ADEP The dimension is M×M, and the mth row vector in the matrix represents e m The relationship with each EDU in the passage, including itself.

[0075] like Figure 1As shown, the EDU fusion module at the encoding end receives the EDU representation vector after the structure information is enhanced output by the text structure analysis module, and fuses the EDU representation vector with the encoder output state vector. i，j , source word encoding state vector h i，j With its EDU vector The fusion process is as follows:

[0076] Where FNN(·) represents a feedforward neural network, σ(·) represents a sigmoid activation function, [·;·] represents a vector concatenation operation, and the gate weight r i，j It is used to balance the encoding state information of the word and its EDU information. The word encoding state vector after integrating the EDU information is

[0077] like Figure 1 As shown, the EDU fusion module at the decoding end receives the EDU representation vector after the structural information is enhanced output by the chapter structure analysis module, and fuses the EDU representation vector with the decoder output state vector. Figure 3 FIG. 4 shows a schematic diagram of a process in which a decoding end fusion module fuses an EDU representation vector with a decoder output state vector. Figure 3 As shown, the decoding end EDU fusion module is a decoding state vector g output by the Transformer multi-layer decoder t , in g t The process of integrating EDU information in the .

[0078] S1, respectively calculate the EDU-level attention weight and word-level attention weight of the decoding state vector gt to the source end;

[0079] S2, use the EDU level attention weight to modify the word level attention weight, and then generate g t The vector representation g of the global context t ';

[0080] S3, using gated weighting to fuse the original decoding vector gt and g t ', generate the final decoding state vector of the fused EDU information Used to predict the probability distribution of the target vocabulary.

[0081] Preferably, in step S1 of this embodiment, the calculation formula of the EDU level attention weight is:

[0082]

[0083] in, and About GT and The linear change of EDU representation vector enhanced with structural information, are model parameters, The dimension is M;

[0084] The word-level attention weight calculation formula is:

[0085] in is the word encoding state vector matrix integrating EDU information, and its dimension is IJ×d, and are model parameters, The dimension is IJ.

[0086] Preferably, in step S2 of this embodiment, g t The calculation process of ' is as follows:

[0087] in

[0088] Preferably, in step S3 of this embodiment, the decoding state vector g fused with EDU information t The fusion process is as follows:

[0089] FNN(·) represents a feed-forward neural network, σ(·) represents the sigmoid activation function, and [·;·] represents a vector concatenation operation.

[0090] In the embodiments of the present disclosure, the auxiliary task of paragraph dependency structure analysis is introduced into the neural machine translation framework. In order to make the model have the ability of paragraph translation and paragraph analysis at the same time, and avoid dependence on the paragraph analyzer in the testing phase, preferably, the translation task and the paragraph dependency structure analysis related tasks can be jointly optimized by multi-task learning. The final loss function of model training using multi-task learning is:

[0091]

[0092] Among them, the hyperparameter λ 1 and λ 2 Used to adjust the proportion between various losses.

[0093] L mle The maximum likelihood loss function for the standard neural machine translation model during training

[0094]

[0095] L seg Binary cross entropy loss for the EDu segmentation of the document

[0096]

[0097] S i，j For word x i，j The probability of segmentation; i，j is the segmentation label in the correct answer, Π i，j =1 means segmentation, Π i，j =0 means no splitting.

[0098] Ldep is the cross entropy loss predicted by the dependency head node, Attention weights to model dependency structures; A DEP The elements in the matrix represent the probability that the dependency head node of the mth EDU in the passage is the nth EDU; dep(m) is the dependency head node of the mth EDU in the answer provided by the passage analyzer.

[0099] L rel is the cross entropy loss for classification,

[0100]

[0101] z m，dep (m) is the dependency label between the mth EDU and its dependency head node dep(m) in the answer provided by the text analyzer during the training phase;

[0102] and EDU vector generated by the chapter structure parsing module, and is the model parameter.

[0103] In the training phase, the neural machine translation system of the embodiment of the present disclosure requires an RST chapter analyzer to parse the source language chapter to obtain a chapter dependency structure converted from the chapter tree structure, which is used to supervise the parsing of the chapter dependency structure.

[0104] Learning of the same subtask. In the testing phase, the method in this section directly translates the input document, and the parsing process of the text dependency structure can be implemented within the model, and no longer requires preprocessing by the text analyzer.

[0105] Figure 4A neural machine translation method is performed by a neural machine translation system according to an embodiment of the present disclosure, comprising the following steps: S10, generating a standard Transformer model through training; S20, performing a text dependency structure analysis on an input document at the top level of the encoder; S30, fusing EDU vectors at the encoding end and the decoding end of the translation model respectively.

[0106] Preferably, in this embodiment, in step S20, an EDU vector with enhanced structural information is generated through EDU segmentation and paragraph structure-aware multi-head self-attention encoding.

[0107] According to another embodiment of the present disclosure, an electronic device is provided, including: a memory storing execution instructions; and a processor executing the execution instructions stored in the memory, so that the processor executes the above-mentioned neural machine translation method.

[0108] According to another embodiment of the present disclosure, a readable storage medium is provided, wherein the readable storage medium stores execution instructions, and the execution instructions are used to implement the above-mentioned neural machine translation method when executed by a processor.

[0109] Figure 5 is a schematic structural diagram of a neural machine translation device 1000 according to an embodiment of the present disclosure.

[0110] The device may include a corresponding module for performing each or several steps in the above flowchart. Therefore, each step or several steps in the above flowchart may be performed by a corresponding module, and the device may include one or more modules in these modules. The module may be one or more hardware modules specially configured to perform the corresponding steps, or implemented by a processor configured to perform the corresponding steps, or stored in a computer-readable medium for implementation by a processor, or implemented by some combination.

[0111] The hardware structure can be implemented using a bus architecture. The bus architecture can include any number of interconnecting buses and bridges, depending on the specific application and overall design constraints of the hardware. The bus 1100 connects various circuits including one or more processors 1200, memory 1300, and / or hardware modules together. The bus 1100 can also connect various other circuits 1400 such as peripherals, voltage regulators, power management circuits, external antennas, etc.

[0112] The bus 1100 may be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Component (EISA) bus, etc. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, the figure only uses one connecting line, but does not mean that there is only one bus or one type of bus.

[0113] Figure 5 In the neural machine translation device 1000, the Transformer model 1001 is a Transformer translation model obtained by training with a training set and a development set; the text structure analysis module 1002 is used to model the dependency structure information of the text and obtain an EDU representation vector with enhanced structure information; the encoding-side EDU fusion module 1003 fuses the EDU representation vector with the encoder output state vector of the Transformer translation model; the decoding-side EDU fusion module 1004 fuses the EDU representation vector with the decoder output state vector of the Transformer translation model.

[0114] Any process or method description in the flowchart or otherwise described herein can be understood as a module, fragment or portion of a code representing one or more executable instructions for implementing the steps of a specific logical function or process, and the scope of the preferred embodiments of the present disclosure includes other implementations, in which the functions may not be performed in the order shown or discussed, including performing the functions in a substantially simultaneous manner or in a reverse order according to the functions involved, which should be understood by a person skilled in the art of the present disclosure. The processor performs the various methods and processes described above. For example, the method implementation in the present disclosure can be implemented as a software program, which is tangibly contained in a machine-readable medium, such as a memory. In some embodiments, part or all of the software program can be loaded and / or installed via a memory and / or a communication interface. When the software program is loaded into the memory and executed by the processor, one or more steps in the method described above can be performed. Alternatively, in other embodiments, the processor can be configured to perform one of the above methods in any other appropriate manner (e.g., by means of firmware).

[0115] The logic and / or steps represented in the flowchart or otherwise described herein may be embodied in any readable storage medium for use by an instruction execution system, apparatus, or device (such as a computer-based system, a system including a processor, or other system that can fetch instructions from and execute instructions on an instruction execution system, apparatus, or device), or in combination with such instruction execution systems, apparatuses, or devices.

[0116] For the purposes of this specification, a "readable storage medium" may be any device that can contain, store, communicate, propagate or transmit a program for use with or in conjunction with an instruction execution system, device or apparatus. More specific examples of readable storage media (a non-exhaustive list) include the following: an electrical connection with one or more wirings (electronic device), a portable computer disk case (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and editable read-only memory (EPROM or flash memory), a fiber optic device, and a portable read-only memory (CDROM). In addition, the readable storage medium may even be paper or other suitable medium on which the program can be printed, since the program can be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, deciphering or, if necessary, processing in another suitable manner, and then stored in a memory.

[0117] It should be understood that the various parts of the present disclosure can be implemented in hardware, software, or a combination thereof. In the above-mentioned embodiments, multiple steps or methods can be implemented in software stored in a memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0118] A person skilled in the art may understand that all or part of the steps of implementing the above-mentioned implementation method may be completed by instructing related hardware through a program, and the program may be stored in a readable storage medium, which, when executed, includes one or a combination of the steps of the implementation method.

[0119] In addition, each functional unit in each embodiment of the present disclosure may be integrated into a processing module, or each unit may exist physically separately, or two or more units may be integrated into one module. The above-mentioned integrated module may be implemented in the form of hardware or in the form of a software functional module. If the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it may also be stored in a readable storage medium. The storage medium may be a read-only memory, a disk or an optical disk, etc.

[0120] The present disclosure also provides an electronic device, including: a memory, the memory storing execution instructions; and a processor or other hardware module, the processor or other hardware module executes the execution instructions stored in the memory, so that the processor or other hardware module executes the above method.

[0121] The present disclosure also provides a readable storage medium, in which execution instructions are stored. When the execution instructions are executed by a processor, they are used to implement the above method.

[0122] In the description of this specification, the description with reference to the terms "one embodiment / method", "some embodiments / methods", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment / method or example are included in at least one embodiment / method or example of the present application. In this specification, the schematic representations of the above terms are not necessarily the same embodiment / method or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments / methods or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments / methods or examples described in this specification and the features of the different embodiments / methods or examples, unless they are contradictory.

[0123] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In the description of this application, the meaning of "plurality" is at least two, such as two, three, etc., unless otherwise clearly and specifically defined.

[0124] Those skilled in the art should understand that the above embodiments are only for the purpose of clearly illustrating the present disclosure, and are not intended to limit the scope of the present disclosure. For those skilled in the art, other changes or modifications may be made based on the above disclosure, and these changes or modifications are still within the scope of the present disclosure.

Claims

1. A neural machine translation system, characterized in that: include: Standard Transformer multi-layer encoder and multi-layer decoder; the chapter structure analysis module, which is set at the top layer of the encoder to model the dependency structure information of the chapter and obtain the EDU representation vector with enhanced structure information; the EDU fusion module on the encoding side, which fuses the EDU representation vector with the encoder output state vector; the EDU fusion module on the decoding side, which fuses the EDU representation vector with the decoder output state vector; The text structure parsing module generates a set of initial EDU representation vectors by EDU segmentation and maximum pooling operations on the input word state vector after multi-layer encoding by Transformer, and then uses the multi-head self-attention mechanism of text structure perception to model the dependency structure information, and outputs the EDU representation vector with enhanced structure information; The article structure-aware multi-head attention mechanism adds an EDU dependency head to model the EDU dependency structure and relationship based on the standard multi-head attention mechanism. The output DSAMHA of the multi-head attention mechanism is calculated as follows: DSAMHA(Q,K,V)=[head DEP ;head1;…;head N ]W O , Among them, input Q, K, V are the initial EDU vectors, W O are model parameters; common heads {head1,…,head N The output of} is: Where W O , W Q n , W K n , W V n is the model parameter; ATTN is the attention calculation formula, Where d represents the dimension of the hidden state vector of K. The ATTN function calculates the dot product of the query vector Q and the key vector K and normalizes it through softmax as the attention weight A. Then, the attention is used to perform weighted summation on the value vector V to obtain the final output. For head DEP The calculation uses a double affine operation: in and U are model parameters; attention weight matrix ADEP The dimension is M×M, and the mth row vector in the matrix represents e m The relationship with each EDU in the text, including itself; The EDU fusion module on the encoding side is used for any word X i，j , source word encoding state vector h i，j Its EDU vector The fusion process is as follows: Where FNN(·) represents a feedforward neural network, σ(·) represents a sigmoid activation function, [·;·] represents a vector concatenation operation, and the gate weight r i，j It is used to balance the encoding state information of the word and its EDU information. The word encoding state vector after integrating the EDU information is The decoding end EDU fusion module is a decoding state vector g output by the Transformer multi-layer decoder t , in g t The process of integrating EDU information in the . S1, respectively calculate the EDU-level attention weight and word-level attention weight of the decoding state vector gt to the source end; S2, use the EDU level attention weight to modify the word level attention weight, and then generate g t The vector representation g of the global context t '; S3, using gated weighting to fuse the original decoding vector gt and g t ', generate the final decoding state vector of the fused EDU information Used to predict the probability distribution of the target vocabulary; In step S1, the calculation formula of EDU level attention weight is: in, and About GT and The linear change of EDU representation vector enhanced with structural information, are model parameters, The dimension is M; The word-level attention weight calculation formula is: in is the word encoding state vector matrix integrating EDU information, whose dimension is IJ×d, and are model parameters, The dimension is IJ; In step S2, g t The calculation process of ' is as follows: in In step S3, the decoding state vector of the EDU information is integrated The fusion process is as follows: FNN(·) represents a feed-forward neural network, σ(·) represents the sigmoid activation function, and [·;·] represents a vector concatenation operation.

2. The neural machine translation system according to claim 1, characterized in that: The EDU segmentation is in word x i，j The probability of splitting afterwards is: s i，j =σ(W s h i，j +b s ),in and b s is the model parameter, σ(·) represents the sigmoid activation function; when S i，j When x is greater than a fixed threshold, i，j Then perform EDU segmentation.

3. The neural machine translation system according to claim 1, characterized in that: The maximum pooling operation generates the initial EDU vector: m =MaxPooling(H m ), where H m is the word encoding state vector contained in the mth EDU; MaxPooling(·) is the maximum pooling function.

4. A neural machine translation method, which uses the neural machine translation system according to any one of claims 1 to 3 to perform neural machine translation.

5. An electronic device, characterized in that: include: A memory storing execution instructions; and a processor, wherein the processor executes the execution instructions stored in the memory so that the processor executes the neural machine translation method according to claim 4.

6. A readable storage medium, characterized in that: The readable storage medium stores execution instructions, which are used to implement the neural machine translation method according to claim 4 when executed by the processor.

Images