Machine translation method, machine translation model training method, device and apparatus

Abstract

The application discloses a machine translation method, a training method, device and equipment of a machine translation model, and relates to the field of machine translation. The method comprises the following steps: obtaining an encoded word vector of a source sentence after embedding coding; performing mapping processing on the encoded word vector to obtain a mapped intermediate vector; performing fitting processing on the mapped intermediate vector according to a dependency relationship between the mapped intermediate vectors, to obtain a fitted vector, wherein the dependency relationship is used to represent an association relationship between words in the source sentence; and performing decoding processing on the fitted vector to output a target sentence after translation of the source sentence. The mapped intermediate vector is fitted by using the dependency relationship, a large number of vectors conforming to the target sentence are output, and the target sentence output by the machine translation model is more accurate.

Description

Technical Field

[0001] This application relates to the field of machine translation, and in particular to a machine translation method, a training method for a machine translation model, an apparatus, and equipment. Background Technology

[0002] Machine translation is the process of using computers to convert one natural language into another. It is widely used in all aspects of life, such as translation assistants, simultaneous voice interpretation, and cross-language retrieval, enabling barrier-free communication between different languages.

[0003] Machine translation models typically employ an encoder-decoder architecture, where both the encoder and decoder include attention modules and feedforward networks (FFN) modules. Taking a weighted machine translation model as an example, the encoder in a weighted machine translation model comprises multiple parallel attention modules and FFN modules. During the translation of the source sentence, the source sentence is input into multiple parallel attention modules, and the outputs of these attention modules are input into multiple parallel FFN modules. The FFN modules output an intermediate representation of the source sentence, which is then used as the input to the decoder. Finally, the target sentence is obtained through a weighted summation.

[0004] In the weighted machine translation model described above, the parallel Attention modules and FFN modules are independent of each other. They output the target sentence by simply stacking words, which can easily lead to inaccurate translation results. Summary of the Invention

[0005] This application provides a machine translation method, a machine translation model training method, apparatus, and device. By acquiring the dependencies between vectors, the errors between word vectors in the target sentence are compensated for, thereby improving the accuracy of the translation results. The technical solution is as follows:

[0006] According to one aspect of this application, a machine translation method is provided, the method comprising:

[0007] Obtain the encoded word vectors of the source sentence after embedding encoding;

[0008] The encoded word vectors are mapped to obtain the mapped intermediate vectors;

[0009] Based on the dependencies between the mapped intermediate vectors, the mapped intermediate vectors are fitted to obtain fitted vectors, where the dependencies are used to characterize the association between words in the source sentence.

[0010] The fitted vector is decoded to output the target sentence after translation of the source sentence.

[0011] According to another aspect of this application, a method for training a machine translation model is provided, the method comprising:

[0012] Obtain the sample encoded word vector after embedding and encoding the source sentence, and the source sentence corresponds to the translation result;

[0013] The sample encoded word vector is biased to obtain the first sample vector after the biasing process;

[0014] The first sample vector after bias processing is mapped to obtain the second sample vector after mapping.

[0015] The mapped second sample vector is decoded to output the translated target sentence of the source sentence.

[0016] The machine translation model is trained based on the translation results and the target sentence of the sample to obtain the trained machine translation model.

[0017] According to another aspect of this application, a machine translation apparatus is provided, the apparatus comprising:

[0018] The first acquisition module is used to acquire the encoded word vectors of the source sentence after embedding and encoding.

[0019] The first mapping module is used to map the encoded word vectors to obtain the mapped intermediate vectors;

[0020] The fitting module is used to fit the mapped intermediate vectors according to the dependency relationship between them to obtain the fitted vectors. The dependency relationship is used to characterize the association relationship between words in the source sentence.

[0021] The first decoding module is used to decode the fitted vector and output the target sentence after the source sentence has been translated.

[0022] According to another aspect of this application, a training apparatus for a machine translation model is provided, the apparatus comprising:

[0023] The second acquisition module is used to acquire the sample encoded word vector after the source sentence of the sample has been embedded and encoded, and the source sentence corresponds to the translation result;

[0024] The bias module is used to bias the sample encoded word vector to obtain the first sample vector after the bias processing.

[0025] The second mapping module is used to perform mapping processing on the biased first sample vector to obtain the mapped second sample vector.

[0026] The second decoding module is used to decode the mapped second sample vector and output the translated sample target sentence from the sample source sentence.

[0027] The machine translation model is trained based on the translation results and the target sentence of the sample to obtain the trained machine translation model.

[0028] According to another aspect of this application, a computer device is provided, the computer device including a processor and a memory, the memory storing at least one instruction, at least one program, code set or instruction set, the at least one instruction, the at least one program, the code set or instruction set being loaded and executed by the processor to implement the machine translation method and the machine translation model training method as described above.

[0029] According to another aspect of this application, a computer-readable storage medium is provided, wherein at least one instruction, at least one program, code set, or instruction set is stored therein, wherein the at least one instruction, the at least one program, the code set, or the instruction set is loaded and executed by a processor to implement the machine translation method and the machine translation model training method as described above.

[0030] According to another aspect of this application, a computer program product or computer program is provided, the computer program product or computer program including computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, causing the computer device to perform the machine translation method and the machine translation model training method as described above.

[0031] The beneficial effects of the technical solutions provided in this application include at least the following:

[0032] By determining the dependencies between the mapped intermediate vectors and using these dependencies to fit the mapped intermediate vectors, a large number of vectors that match the target sentence are output. This enables the machine translation model to translate the source sentence based on the dependencies between the vectors, rather than simply stacking the output decoded word vectors, making the target sentence output by the machine translation model more accurate. Attached Figure Description

[0033] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0034] Figure 1 This is a framework diagram of a computer system provided in an exemplary embodiment of this application;

[0035] Figure 2 This is a flowchart of a machine translation method provided in an exemplary embodiment of this application;

[0036] Figure 3 This is a structural framework diagram of a machine translation model provided in an exemplary embodiment of this application;

[0037] Figure 4 This is a flowchart of a machine translation method provided in another exemplary embodiment of this application;

[0038] Figure 5 This is a structural framework diagram of a machine translation model provided in another exemplary embodiment of this application;

[0039] Figure 6 This is a flowchart of a training method for a machine translation model provided in an exemplary embodiment of this application;

[0040] Figure 7 This is a structural framework diagram of a machine translation model provided in another exemplary embodiment of this application;

[0041] Figure 8 This is a comparison table of training results of a machine translation model provided in an exemplary embodiment of this application;

[0042] Figure 9 This is a structural block diagram of a machine translation apparatus provided in an exemplary embodiment of this application;

[0043] Figure 10 This is a structural block diagram of a training apparatus for a machine translation model provided in an exemplary embodiment of this application;

[0044] Figure 11 This is a schematic diagram of the device structure of a server provided in an exemplary embodiment of this application. Detailed Implementation

[0045] To make the objectives, technical solutions, and advantages of this application clearer, the embodiments of this application will be described in further detail below with reference to the accompanying drawings.

[0046] First, the terms used in the embodiments of this application will be introduced.

[0047] Transformer model: A model including encoder and decoder. The encoder and decoder layers have similar or identical structures. The encoder typically includes multiple attention mechanism units and feedforward neural network units, such as self-attention and multi-head attention mechanisms. The decoder typically includes self-attention and cross-attention mechanisms. The Transformer model usually divides the input of the attention mechanism units into N "heads" (referring to the number of multi-head attention mechanism units) according to the model's dimensions, and then performs attention mechanism calculations on each "head". The machine translation model in this application embodiment is built based on the Transformer model.

[0048] Artificial Intelligence (AI) is the theory, methods, technology, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. In other words, AI is a comprehensive technology within computer science that attempts to understand the essence of intelligence and produce a new kind of intelligent machine that can react in a way similar to human intelligence. AI studies the design principles and implementation methods of various intelligent machines, enabling them to possess the functions of perception, reasoning, and decision-making.

[0049] Artificial intelligence (AI) is a comprehensive discipline encompassing a wide range of fields, including both hardware and software technologies. Fundamental AI technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interactive systems, and mechatronics. AI software technologies primarily include computer vision, speech processing, natural language processing, and machine learning / deep learning.

[0050] With the research and advancement of artificial intelligence (AI) technology, AI is being studied and applied in various fields, such as smart homes, smart wearable devices, virtual assistants, smart speakers, smart marketing, autonomous driving, drones, robots, smart healthcare, and smart customer service. It is believed that with the development of technology, AI will be applied in more fields and play an increasingly important role.

[0051] The machine translation method provided in this application can be applied to the following scenarios:

[0052] I. Machine Translation

[0053] In this application scenario, the machine translation model trained using the method provided in this application embodiment can be applied to applications that support translation functions, such as electronic dictionary applications, e-book applications, web browsing applications, social applications, and image recognition applications. When the aforementioned applications receive content to be translated, the trained machine translation model outputs a translation result based on the input content. Indicatively, the content to be translated includes at least one of text, image, audio, and video types. Image content includes photos taken by the terminal's camera or images containing the content to be translated; this application embodiment does not limit this. By recognizing the text content in the image, the recognized text content is input into the trained machine translation model, and the translation result is output and displayed to the user. As an illustration, by recognizing text in audio (such as converting human voices in audio into text) and inputting the recognized text into a trained machine translation model, a translation result can also be output; similarly, by recognizing text in video (such as recognizing subtitles in video or converting human voices in video into text) and inputting the recognized text into a trained machine translation model, a translation result can also be output.

[0054] II. Dialogue and Q&A

[0055] In this application scenario, the machine translation model trained using the method provided in this application embodiment can be applied to smart devices such as smart terminals or smart homes. Taking a virtual assistant set up in a smart terminal as an example, the automatic answering function of the virtual assistant is implemented through the machine translation model trained as described above. The user asks the virtual assistant a translation-related question. When the virtual assistant receives the user's input question (which can be in the form of voice or text input), the machine translation model outputs a translation result based on the input question. The translation result is then converted into voice or text form and fed back to the user through the virtual assistant.

[0056] The above description uses only two application scenarios as examples. The method provided in this application embodiment can also be applied to application scenarios of extracting text summaries. This application embodiment does not limit the specific application scenarios.

[0057] The machine translation method and machine translation model training method provided in this application can be applied to computer devices with strong data processing capabilities. In one possible implementation, the machine translation method and machine translation model training method provided in this application can be applied to a personal computer, workstation, or server, that is, machine translation and machine translation model training can be performed using a personal computer, workstation, or server.

[0058] The trained machine translation model can be integrated into an application and installed on the terminal, allowing the terminal to output the translation result when it receives content to be translated; alternatively, the trained machine translation model can be set up in the application's backend server, so that the terminal with the application installed can use the backend server to perform the translation function.

[0059] Figure 1 A schematic diagram of a computer system provided in one embodiment of this application is shown. The computer system includes a terminal 110 and a server 120. The terminal 110 and the server 120 communicate with each other via a communication network. Optionally, the communication network can be a wired network or a wireless network, and the communication network can be at least one of a local area network (LAN), a metropolitan area network (MAN), and a wide area network (WAN).

[0060] The terminal 110 is equipped with an application that supports translation functions. This application may be an e-book reading application, an electronic dictionary application, a web browsing application, a game application, a social application, etc. This application embodiment does not limit this.

[0061] Optionally, the terminal 110 can be a mobile terminal such as a smartphone, smartwatch, tablet computer, laptop computer, or smart robot, or a terminal such as a desktop computer or projector computer. This application embodiment does not limit the type of terminal.

[0062] Server 120 can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms. In one possible implementation, server 120 is the backend server for the application in terminal 110.

[0063] In some embodiments, the terminal 110 includes a microphone; the terminal 110 collects voice content through the microphone and sends the collected voice content to the server 120.

[0064] Server 120 includes a speech translation module; the speech translation module includes a trained machine translation model; server 120 receives speech content sent by terminal 110, translates the speech content through the machine translation model to obtain the translation result, and sends the translation result to terminal 110 for display.

[0065] Alternatively, terminal 110 may include a speech translation module; this speech translation module may include a trained machine translation model; after terminal 110 acquires speech content, it may translate the speech content using the machine translation model to obtain the translation result and then display the translation result.

[0066] In some embodiments, terminal 110 includes a text input control; terminal 110 obtains text content through the text input control and sends the obtained text content to server 120.

[0067] The server 120 includes a text translation module, which includes a trained machine translation model. The server 120 receives text content sent by the terminal 110, translates the text content using the machine translation model, obtains the translation result, and sends the translation result to the terminal 110 for display.

[0068] Alternatively, terminal 110 may include a text translation module; this text translation module may include a trained machine translation model; after terminal 110 obtains the text content, it may translate the text content using the machine translation model to obtain the translation result and then display the translation result.

[0069] In some embodiments, terminal 110 includes a camera component; terminal 110 acquires image content through the camera component, the image content including content to be translated, and terminal 110 sends the image to server 120.

[0070] The server 120 includes an image translation module, which includes a trained machine translation model. The server 120 receives images sent by the terminal 110, identifies and translates the content in the images using the machine translation model, and sends the translation results to the terminal 110 for display.

[0071] Alternatively, terminal 110 may include an image translation module; this image translation module may include a trained machine translation model; after acquiring an image, terminal 110 may translate the image content using the machine translation model to obtain the translation result and then display the translation result.

[0072] In some embodiments, the terminal displays information in either voice or text format.

[0073] As an illustration, the machine translation model provided in this application is applied to scenarios such as document translation, article reading, foreign language website translation, foreign language learning queries, oral dialogue assistance, and overseas tourism services. Examples are provided below:

[0074] Machine translation models, applied in text translation software, can take documents such as contracts, files, materials, papers, and emails as input data. The input data undergoes word embedding processing to obtain encoded word vectors. These encoded word vectors are then mapped to intermediate vectors. By fitting these intermediate vectors based on the dependencies between them, a fitted vector is obtained. Finally, the decoder in the machine translation model decodes the fitted vector to obtain the decoded word vector, thus outputting the translation result.

[0075] Machine translation models are applied to various websites to achieve translation assistance functions. For example, when reading an article on a website, the website's text translation function is triggered to obtain the text content of the article. This text content is used as input data, and word embedding processing is performed on the input data to obtain encoded word vectors. By mapping the encoded word vectors, intermediate mapping vectors are obtained. By fitting the intermediate mapping vectors according to the dependencies between them, a fitted vector is obtained. Finally, the decoder in the machine translation model decodes the fitted vector to obtain the decoded word vector, thus outputting the translation result.

[0076] Alternatively, when browsing a foreign language website, the website's translation function can be triggered to obtain the text content from the webpage. This text content is then used as input data, and word embedding processing is performed on the input data to obtain encoded word vectors. By mapping the encoded word vectors, intermediate mapping vectors are obtained. By fitting the intermediate mapping vectors according to the dependencies between them, a fitted vector is obtained. Finally, the decoder in the machine translation model decodes the fitted vector to obtain the decoded word vector, thus outputting the translation result, which enables the translation of the website.

[0077] Machine translation models are applied in electronic dictionaries. When a sentence is input into the electronic dictionary, the dictionary first performs word embedding processing on the sentence to obtain encoded word vectors. By mapping the encoded word vectors, intermediate mapping vectors are obtained. By fitting the intermediate mapping vectors according to the dependencies between them, a fitted vector is obtained. Finally, the decoder in the machine translation model decodes the fitted vector to obtain the decoded word vector, thus outputting the translation result.

[0078] Machine translation models are applied to real-time translation tools, such as machine translators. These tools acquire audio via a microphone, process the audio to obtain the source sentence, and perform word embedding on the source sentence to obtain encoded word vectors. These encoded word vectors are then mapped to intermediate vectors. By fitting these intermediate vectors based on the dependencies between them, a fitted vector is obtained. Finally, the decoder in the machine translation model decodes the fitted vector to obtain the decoded word vector, thus outputting the translation result. The translation result can be displayed or converted into speech and played through a speaker. This translation tool can provide rapid responses in multinational conferences and during overseas travel.

[0079] Machine translation models are applied in instant messaging applications. When a user receives voice or video through an instant messaging application, the terminal acquires the audio, processes the audio to obtain the source sentence, and performs word embedding on the source sentence to obtain encoded word vectors. By mapping the encoded word vectors, intermediate mapping vectors are obtained. By fitting the intermediate mapping vectors according to the dependencies between them, fitted vectors are obtained. Finally, the decoder in the machine translation model decodes the fitted vectors to obtain decoded word vectors, thereby outputting the translation result and displaying the translation result, or converting the translation result into speech and playing it through a speaker.

[0080] Machine translation models are applied in video playback programs. When a video is played on a terminal, the terminal captures audio, processes the audio to obtain the source sentence, and performs word embedding on the source sentence to obtain encoded word vectors. By mapping the encoded word vectors, intermediate mapping vectors are obtained. By fitting the intermediate mapping vectors according to the dependencies between them, fitted vectors are obtained. Finally, the decoder in the machine translation model decodes the fitted vectors to obtain decoded word vectors, thereby outputting the translation result and displaying the translation result, or converting the translation result into speech and playing it through a speaker.

[0081] Alternatively, the terminal extracts subtitles, uses the text content of the subtitles as input data, performs word embedding processing on the input data to obtain intermediate mapping vectors, fits the intermediate mapping vectors according to the dependencies between them to obtain fitted vectors, and finally decodes the fitted vectors through the decoder in the machine translation model to obtain decoded word vectors, thereby outputting the translation result.

[0082] For ease of explanation, the following embodiments are illustrated using the example of a machine translation model training method and a machine translation method executed by a server.

[0083] Figure 2 The flowchart of the machine translation method provided by an exemplary embodiment of the present application is shown. In this embodiment, it is described by taking the server 120 in the computer system as shown in Figure 1 as an example. The method includes the following steps:

[0084] Step 201: Obtain the encoded word vectors after the source-side sentence is embedded and encoded.

[0085] In the process of natural language translation, a sentence in the first language category is translated into a sentence in the second language category. The sentence in the first language category is the source-side sentence, and the sentence in the second language category is the target-side sentence. Schematically, when translating a Chinese sentence into an English sentence, the Chinese sentence is the source-side sentence, and the English sentence is the target-side sentence; for example, when a machine translation model translates the sentence "我是一名学生 (I am a student)" into the sentence "I am a student.", the sentence "我是一名学生 (I am a student)" is the source-side sentence, and the sentence "I am a student." is the target-side sentence.

[0086] Input the source-side sentence 10 into the machine translation model as shown in Figure 3 The machine translation model performs source-side embedding and encoding 11 processing on the source-side sentence 10 to obtain encoded word vectors, and uses the encoded word vectors as the input of the encoder 12. Source-side embedding and encoding processing refers to performing word embedding encoding on the words in the source-side sentence. Word embedding encoding (WordEmbedding) processing refers to the technology of representing words as real number vectors in a vector space. In the implementation of this application, word embedding encoding processing refers to mapping words to word vectors; for example, mapping the word "我 (I)" to obtain the word vector (0.1, 0.5, 5), and the word vector (0.1, 0.5, 5) is the encoded word vector after word embedding processing.

[0087] Schematically, the source-side sentence is a sentence manually input by the user in the terminal, or a sentence in the text content uploaded by the user to the terminal, or a sentence included in the voice collected by the terminal, or a sentence included in the picture collected by the terminal, or a sentence included in the video collected by the terminal. The implementation of this application does not limit the acquisition method of the source-side sentence.

[0088] Step 202: Perform mapping processing on the encoded word vectors to obtain the mapped intermediate vectors.

[0089] Mapping processing refers to using an attention mechanism unit to map the input vector to other vector spaces, so as to obtain the representation of the input vector from the corresponding vector spaces.

[0090] The encoder 12 includes multiple coding layers, each containing a mapping unit. Encoded word vectors are input into the coding layer, specifically into the mapping unit, which then maps the encoded word vectors. The mapping unit performs the mapping process using the following formula:

[0091] Formula 1:

[0092] Among them, X k SelfAttn represents the input data of the k-th layer mapping unit. i (X k ) indicates that the k-th layer mapping unit performs mapping processing on the input data. This represents the mapping result of the k-th layer mapping unit, where k is a positive integer and its initial value is 2.

[0093] It should be noted that the input to the first coding layer is the encoded word vector, and the input vectors of the other coding layers are the output vectors of the previous coding layer.

[0094] The mapping unit in this embodiment includes a multi-head attention mechanism unit, which performs mapping processing using the above formula 1.

[0095] Indicatively, the mapping unit in this embodiment further includes a feedforward neural network unit, which transmits the output mapping result to the previous encoding layer. The feedforward neural network unit transmits the mapping result forward using the following formula:

[0096] Formula 2:

[0097] in, This represents the mapping result output by the k-th layer mapping unit. This represents the processing of the mapping result by the k-th layer feedforward neural network unit. This represents the output of the i-th mapping unit in the k-th layer.

[0098] In some embodiments, the encoder includes multiple mapping units connected in parallel. The outputs of the multiple feedforward neural network units are fused using a weighted fusion method to output the final output of the k-th layer mapping unit. The multiple feedforward neural network units are fused using the following formula:

[0099] Formula 3:

[0100] in, α represents the vector output by the i-th feedforward neural network unit in the k-th layer. idenotes the weight corresponding to the i-th mapping unit, I represents the number of mapping units in the k-th encoding layer, I is a positive integer, F e (X k ) represents the final output of the k-th layer mapping unit.

[0101] Step 203: According to the dependency relationship between the mapped intermediate vectors, perform fitting processing on the mapped intermediate vectors to obtain the fitted vectors, where the dependency relationship is used to characterize the association relationship between words in the source sentence.

[0102] In the embodiment of the present application, the dependency relationship is used to characterize the association relationship between words in the source sentence. Usually, between two word vectors, a change in one word vector will affect the result of the other word vector. For example, there is a dependency relationship between the word vectors of two words. The first word is "go", and the second word is "work", forming the word "go to work". There is a dependency relationship between the first word and the second word. If the second word changes, for example, "work" is changed to "school", then the word "go to school" is formed. "Go to work" and "go to school" are two words with different meanings. It can be understood that after the word changes, the corresponding word vectors also change.

[0103] The fitting processing refers to the data operation performed on the vectors output by the mapping unit through the sequential dependency unit.

[0104] The encoder 12 includes multiple encoding layers, and each encoding layer further includes a sequential dependency unit to determine the dependency relationship between the mapped intermediate vectors. The output result of the k-th layer mapping unit is input to the k-th layer sequential dependency unit. The k-th layer sequential dependency unit sorts the mapped intermediate vectors to obtain the dependency relationship between the sorted intermediate vectors, and performs fitting on the sorted intermediate vectors according to the dependency relationship. The k-th layer sequential dependency unit outputs the fitted vectors.

[0105] Step 204: Perform decoding processing on the fitted vectors to output the target sentence after translation of the source sentence.

[0106] After the source sentence 10 is processed by the source embedding encoding 11, the encoded word vectors are obtained. The encoded word vectors are input into the encoder 12, and the encoded word vectors are processed by multiple encoding layers to output intermediate vectors. The intermediate vectors are input into the decoder 13 for decoding to output decoded word vectors, thereby outputting the translation result.

[0107] In some embodiments, the decoder 13 processes the intermediate vector output by the encoder and also combines it with the word sequence 17 already output in the target sentence to output the translation result. Similarly, the word sequence 17 already output in the target sentence is processed by target-end embedding encoding 18, which refers to word embedding encoding of words in the target sentence. The encoded word vector corresponding to the target end is input into the decoder 13, and the intermediate vector output by the encoder 12 is combined with it to decode the two, outputting the decoded vector. The decoded vector is further calculated by the linear computation unit 14 (Linear) and the logistic regression unit 15 (Softmax) to output the translation result.

[0108] The machine translation model provided in this application is a multi-unit machine translation model (MUTE) built on the Transformer model, also known as the multi-module Transformer model.

[0109] In summary, the method provided in this embodiment determines the dependencies between the mapped intermediate vectors and uses these dependencies to fit the mapped intermediate vectors, outputting a large number of vectors that match the target sentence. This enables the machine translation model to translate the source sentence based on the dependencies between the vectors, rather than simply stacking the output decoded word vectors, making the target sentence output by the machine translation model more accurate.

[0110] Figure 4 A flowchart illustrating a translation method provided in another exemplary embodiment of this application is shown. This embodiment describes the method using a server 120 in a computer system as shown in Figure 1 as an example. The method includes the following steps:

[0111] Step 401: Obtain the encoded word vector of the source sentence after embedding and encoding.

[0112] For illustrative purposes, the source sentence is a sentence in the first language, specifically a Chinese sentence. In one example, the source sentence is "I am a student". The source sentence is input into the machine translation model, which performs source embedding encoding on it, resulting in the encoded word vector.

[0113] Step 402: Call the (k+1)th layer mapping unit to map the fitted vector output by the kth layer sequential dependency unit, and output the first intermediate vector after mapping.

[0114] Machine translation models consist of N cascaded coding layers, i.e., in... Figure 3The encoder 12 shown is formed by concatenating N coding layers. Taking the (k+1)th coding layer as an example, the output vector of the kth coding layer is the input vector of the (k+1)th coding layer, where k is a positive integer and its initial value is 2.

[0115] like Figure 5 As shown, the (k+1)th coding layer 121 includes a (k+1)th layer mapping unit 21 and a (k+1)th layer sequential dependency unit 22. The output vector of the (k+1)th layer mapping unit 21 is the input vector of the (k+1)th layer sequential dependency unit 22.

[0116] With X k+1 This represents the input data of the (k+1)th layer mapping unit. The (k+1)th layer mapping unit 21 includes a (k+1)th layer multi-head attention mechanism unit 211 and a (k+1)th layer feedforward neural network unit 212, which processes the input data X. k+1 The input is fed into the (k+1)th layer multi-head attention mechanism unit 211, which uses the formula in the above embodiment to pair the input data X. k+1 Perform mapping processing and output the first intermediate vector after mapping.

[0117] It should be noted that the input to the first encoding layer is the encoded word vector corresponding to the source sentence. That is, the first mapping unit 21 in the first encoding layer performs mapping processing on the encoded word vector. At this time, the first mapping unit inputs data X according to Formula 1 above. k+1 Given the encoded word vector after embedding, the first intermediate vector S1 is output by the first mapping unit.

[0118] Schematic, the mapping unit 21 includes a multi-head attention mechanism unit 211 and a feedforward neural network unit 212. The multi-head attention mechanism unit 211 outputs the mapped first intermediate vector using the above formula 1, and the feedforward neural network unit 212 is used to forward the mapped first intermediate vector to the sequential dependency unit 22. The sequential dependency unit 22 outputs the mapped intermediate vector corresponding to the encoder in the (k+1)th layer according to the above formula 2.

[0119] Step 403: Call the (k+1)th layer sequential dependency unit to obtain the dependency relationship between the mapped first intermediate vectors, and perform fitting processing on the mapped first intermediate vectors according to the dependency relationship to output the first fitted vector corresponding to the (k+1)th layer coding layer.

[0120] like Figure 5 As shown, the (k+1)th layer sequential dependency unit 22 includes the (k+1)th layer sorting unit 221 and the (k+1)th layer dependency unit 222.

[0121] Step 403 can be replaced with the following steps:

[0122] Step 4031: Call the (k+1)th layer sorting unit to sort the mapped first intermediate vector and output the sorted first intermediate vector.

[0123] The first intermediate vector after mapping, output by the (k+1)th layer mapping unit 21, is input into the (k+1)th layer sorting unit 221. The (k+1)th layer sorting unit 221 sorts the first intermediate vector after mapping using Formula 4:

[0124] Formula 4:

[0125] in, This represents the vector X output by the (k+1)th layer sorting unit based on the (k+1)th layer mapping unit. k+1 Perform sorting processing. Let I represent the first intermediate vector after sorting, and let I represent the number of sorting units included in the (k+1)th level sorting unit.

[0126] Step 4032: Call the (k+1)th layer dependency unit to build a dependency relationship for the sorted first intermediate vector.

[0127] The sorted first intermediate vector output by the (k+1)th layer sorting unit 221 is input to the (k+1)th layer dependency unit 222. The (k+1)th layer dependency unit 222 constructs the dependency relationship between the sorted first intermediate vectors according to Formula 5:

[0128] Formula 5:

[0129] in, This represents the first intermediate vector after sorting. This represents a vector that, in the order of the word sequence, precedes the first intermediate vector after sorting. Let R represent the intermediate vector after constructing the dependencies, where R represents the real number field, n represents the length of the word sequence, and d represents the dimension of the vector.

[0130] It should be noted that in this embodiment, complementary relationships are used as the dependency relationships between adjacent sorted first intermediate vectors. The (k+1)th layer dependency unit package 222 includes multiple parallel dependency units 222. The (i+1)th dependency unit 222 fits the vector output by the ith sorting unit 221 and the vector output by the (i+1)th sorting unit 221, thereby enabling the (i+1)th dependency unit 222 to compensate for the error output by the ith dependency unit 222. That is... This represents the output of the (i+1)th dependent unit.

[0131] Step 4033: Call the (k+1)th layer dependency unit to fit the sorted first intermediate vector according to the dependency relationship and output the first fitted vector.

[0132] Formula 6 is used to fuse the vectors output by each dependent unit in the (k+1)th layer:

[0133] Formula Six:

[0134] Where, α i Let I represent the weight corresponding to the i-th mapping unit, and let I represent the number of mapping units in the k-th coding layer, where I is a positive integer. F represents the first intermediate vector after sorting. e (X k+1 ) represents the final output of the (k+1)th coding layer.

[0135] Step 404: Repeat the steps of outputting the first intermediate vector after mapping and outputting the first fitted vector until the Nth coding layer outputs the fitted vector; where N is a positive integer and k≤N.

[0136] The above embodiment only uses the (k+1)th coding layer as an example. The (k)th and (k+2)th coding layers also process the input vector in the same way. For example, the first fitted vector output by the (k+1)th coding layer is input to the (k+2)th coding layer. The mapping unit in the (k+2)th coding layer maps the first fitted vector to obtain a mapped second intermediate vector. The mapped second intermediate vector is input to the (k+2)th layer sequential dependency unit. The (k+2)th layer sequential dependency unit sorts the mapped second intermediate vectors and constructs dependencies on the sorted second intermediate vectors. Based on the dependencies, it fits the sorted second intermediate vectors and outputs the fitted second vector. This fitted second vector is input to the (k+3)th coding layer, and so on. This continues until the Nth coding layer outputs a vector, which is then used as the encoder output vector.

[0137] Step 405: Decode the fitted vector to output the target sentence after translation of the source sentence.

[0138] The machine translation model includes a decoder. Following the steps described above, encoder output 12 outputs the fitted vector corresponding to the entire encoder. This fitted vector is then input into decoder 13, which outputs decoded word vectors. Based on these decoded word vectors, the model performs probability predictions on the target words in the target sentence and outputs the prediction results. For example, if the machine translation model intends to output the second word in the target sentence, the decoder outputs the predicted probability of the second word, determines the second word based on the predicted probability, and thus outputs the second word. The target sentence is output in this manner.

[0139] The machine translation model in this application is based on a multi-module Transformer model, which constructs sequentially dependent units to form a sequentially biased multi-module Transformer model (MUTE).

[0140] In summary, the method provided in this embodiment determines the dependencies between the mapped intermediate vectors and uses these dependencies to fit the mapped intermediate vectors, outputting a large number of vectors that match the target sentence. This enables the machine translation model to translate the source sentence based on the dependencies between the vectors, rather than simply stacking the output decoded word vectors, making the target sentence output by the machine translation model more accurate.

[0141] The encoder employs a layered structure, with each coding layer acting as a unit. The input vector of the previous layer becomes the output vector of the next. Each coding layer's units perform mapping, sorting, and encoding processes on the input vector, repeating this process until the Nth coding layer outputs the fitted vector. This ensures the accuracy of the vector during transmission, resulting in a more accurate target sentence output by the machine translation model.

[0142] The first intermediate vector after mapping is sorted and fitted by the sorting and dependency units in the sequential dependency unit. The dependency unit is used to build dependency relationships on the sorted first intermediate vector, so that there are dependency relationships between the output vectors of each layer of dependency unit. The source sentence is translated according to the dependency relationship, instead of simply stacking the output decoded word vectors, so that the target short sentence output by the machine translation model is more accurate.

[0143] The decoder is used to predict the probability of target words in the target sentence and output the prediction results. This makes the prediction results output based on the prediction probability more accurate, thereby ensuring that the target sentence output by the machine translation model is more accurate.

[0144] based on Figure 4In an optional embodiment, each coding layer 121 includes multiple mapping units 21 and multiple sequential dependency units 22. Taking the (k+1)th layer as an example, the (k+1)th layer mapping unit 21 includes I parallelly connected mapping units 21, each mapping unit 21 including a multi-head attention unit 211 and a feedforward neural network unit 212, the output vector of the multi-head attention unit 211 being the input vector of the feedforward neural network unit 212; the (k+1)th layer sequential dependency unit 22 includes a (k+1)th layer sorting unit 221 and a (k+1)th layer dependency unit 222, wherein the (k+1)th layer sorting unit includes I parallelly connected sorting units 221, and the (k+1)th layer dependency unit 222 includes I parallelly connected dependency units 222, the output vector of the sorting unit 221 being the input vector of the dependency unit 222.

[0145] Step 4031 above can be replaced with step:

[0146] Step 4031a: Obtain the first intermediate vectors after mapping from the outputs of the I feedforward neural network units.

[0147] according to Figure 5 In the structure shown, I feedforward neural network units 212 output the first intermediate vector after mapping according to the output vectors of I multi-head attention mechanism units 211.

[0148] Step 4031b: Concatenate the I mapped intermediate vectors to obtain the concatenated intermediate vector.

[0149] Step 4031c: Obtain the permutation matrix. Fit the permutation matrix with the fitting matrix to obtain the fitted permutation matrix. The permutation matrix is ​​used to sort the I mapped intermediate vectors, and the fitting matrix is ​​used to fit the permutation matrix.

[0150] The permutation matrix is ​​a matrix set up for machine translation models to reorder I mapped intermediate vectors. Each row and column of the permutation matrix contains only one 1, with the rest being 0s. That is, the permutation matrix is ​​a discrete matrix, which is not convenient for sorting the mapped intermediate vectors. Therefore, a fitting matrix is ​​needed to fit the discrete permutation matrix. Illustratively, the fitting matrix is ​​treated as a learnable parameter of the machine translation model, denoted by M, which is a continuous and non-negative matrix M∈R. I×I M i,j ∈(0,1),M ij This represents a vector in the fitted matrix.

[0151] Step 4031d: Input the concatenated intermediate vector and the fitted permutation matrix into the (k+1)th layer sorting unit, and output the sorted first vector.

[0152] The (k+1)th layer sorting unit sorts the concatenated intermediate vector and the fitted permutation matrix using Formula 7:

[0153] Formula 7:

[0154] Where [·;·] represents a splicing operation, M represents the vector output by the i-th feedforward neural network unit in the (k+1)-th layer. T Let M ∈ R be the transpose of the fitted matrix. I×I M i，j ∈(0,1), This indicates that the (k+1)th level sorting unit is based on a certain order p(i) for vector X. k+1 Sorting process.

[0155] To ensure that the fitted matrix yields a reasonable permutation matrix after fitting the permutation matrix, the above-mentioned fitted matrix is ​​obtained in the following way:

[0156] 1. Normalize the fitted matrix to obtain the normalized fitted matrix.

[0157] The fitted matrix is ​​normalized using formulas 8 and 9:

[0158] Formula 8: M i，j =max(M i，j ,0)

[0159] Formula Nine:

[0160] Among them, M ij Let M represent a value in the i-th row and j-th column of the fitting matrix, M represent an I×I matrix, max(Mij, 0) represents taking the maximum value Mij from the fitting matrix, and Formula 9 represents normalizing the vector Mij in the fitting matrix.

[0161] Each coding layer includes a sorting unit. The k-th layer sorting unit sorts the vectors output by the k-th layer mapping unit and then sorts the vectors output by the k-th layer mapping unit using a permutation matrix. The fitting matrix needs to be updated accordingly so that the updated fitting matrix can fit the sorting matrix into a matrix suitable for sorting the vectors output by the k-th layer mapping unit.

[0162] 2. Calculate the first objective function corresponding to the normalized fitting matrix based on the Lipshitz continuity condition.

[0163] The Prishtz continuity condition is used as a non-convex penalty term to ensure that the fitted matrix converges to the permutation matrix. The first objective function is calculated using Equation 10:

[0164] Formula 10:

[0165]

[0166] Among them, L p The first objective function is M. ij It is a vector in the fitted matrix.

[0167] 3. The convergent fitting matrix is ​​obtained after the first objective convergence function converges.

[0168] When the first objective function converges, the converged fitting matrix is ​​obtained using the above formula 10.

[0169] It should be noted that, in order to indicate that the sorting unit 221 sorts the vectors output by the mapping unit 21, the vector output by the i-th mapping unit 21 is input into the (i+1)-th sorting unit 221, so that the sorting unit 221 sorts the first intermediate vector after mapping, as shown below. Figure 5 As shown. Figure 5 The order shown is for illustrative purposes only. In the actual use of the machine translation model, the first intermediate vector after mapping is sorted based on the preset order p(i). For example, 1, 2, 3, 4 are sorted into 4, 2, 3, 1 (the numbers represent the order of the vectors in the word sequence, such as 4 representing the 4th vector).

[0170] Furthermore, step 4033 above can be replaced by the following steps:

[0171] Step 4033a: Call the i-th dependency unit and the (i+1)-th dependency unit to obtain the dependency relationship between the first sub-vector and the second sub-vector.

[0172] The first sub-vector is the vector output by the ith sorting unit in the (k+1)th sorting unit, the second sub-vector is the vector output by the (i+1)th sorting unit in the (k+1)th layer of sorting units, and the output vector of the ith sorting unit is the output vector of the ith dependent unit.

[0173] There is a dependency relationship between the first sub-vector and the second sub-vector. In this embodiment, the dependency relationship is illustrated by the example of the complementary relationship between the vectors. That is, the vector output by the second sub-vector after processing by the dependency unit can make up for the vector output by the first sub-vector after processing by the dependency unit.

[0174] Step 4033b: Call the (i+1)th dependency unit to fit the first and second sub-vectors and output the third sub-vector.

[0175] The first and second sub-vectors are input into the (i+1)th dependent unit, and the (i+1)th dependent unit outputs the third sub-vector using the above formula five.

[0176] It should be noted that there are no dependent units before the first dependent unit. Therefore, the output vector of the first dependent unit is obtained by fitting the output vector of the first sorting unit separately.

[0177] Step 4033c: Repeat the steps of obtaining the dependency relationship between the first sub-vector and the second sub-vector, and the step of outputting the third sub-vector, until I dependency units output the first fitting vector corresponding to the (k+1)th layer dependency unit. The first fitting vector includes the third sub-vector, where I and i are both positive integers and i≤I.

[0178] Each dependency unit outputs its corresponding sub-vector according to steps 4033a and 4033b. Then, the sub-vectors corresponding to each of the I dependency units are fused to obtain the first fitting vector corresponding to the (k+1)th layer dependency unit. That is, the first fitting vector output by the (k+1)th layer includes the sub-vectors output by each dependency unit in the I dependency units (the vectors output by each dependency unit belong to the same level). This embodiment only uses the first sub-vector and the second sub-vector as examples for illustration.

[0179] This embodiment only describes the fitting process between the i-th dependency unit and the (i+1)-th dependency unit in the (k+1)-th layer. It can be understood that, using the method in this embodiment, all dependency units in the (k+1)-th layer fit the vector output by their respective sorting units, and each dependency unit outputs the first fitted vector. Formula 6 is used to fuse the fitted first vector output by each dependent unit to obtain the fitted vector output by the (k+1)th dependent unit, which is also the fitted vector output by the (k+1)th coding layer. This process is repeated until the fitted vectors corresponding to the Nth coding layer are obtained, which are the fitted vectors output by the encoder.

[0180] In summary, the method provided in this embodiment decomposes a coding layer into smaller units, fuses the vectors output by multiple units, and uses the decomposed units to compensate for the residuals in the outputs between units, thereby accurately obtaining the output vector corresponding to a coding layer, making the source sentence output based on the vector output by the encoder more accurate.

[0181] By sorting the first intermediate vector after mapping using a permutation matrix, the machine translation model can select the translation that is closer to the source sentence based on more vector representations, thereby improving the accuracy of the target sentence output.

[0182] By fitting a fitting matrix to fit a permutation matrix, the permutation matrix can be adapted to the mapped vectors of each layer. At the same time, the fitting matrix is ​​used to fit the permutation matrix into a continuous matrix, so that the machine translation model can be optimized and updated based on the continuous permutation matrix, thereby improving the accuracy of the target sentence output.

[0183] The training method for machine translation models will be explained below.

[0184] Figure 6 A flowchart illustrating a training method for a machine translation model provided in an exemplary embodiment of this application is shown. This embodiment describes the method using a server 120 in a computer system 100 as shown in Figure 1 as an example. The method includes the following steps:

[0185] Step 601: Obtain the sample encoded word vector after embedding and encoding the source sentence. The source sentence has a corresponding translation result.

[0186] In illustrative terms, the sample source sentences include sentences from historical translation records or are obtained through publicly available datasets. This application does not limit the method of obtaining sample source sentences. Each source sentence corresponds to a correct translation result. The sample source sentences are input into a machine translation model, which includes a word embedding processing layer. This layer includes a source word embedding processing layer, which performs word embedding processing on the sample source sentences to obtain encoded sample word vectors.

[0187] Step 602: The sample encoded word vector is biased to obtain the first sample vector after biasing.

[0188] The structure of the machine translation model during training is as follows: Figure 7 As shown, the (k+1)th coding layer includes a (k+1)th bias unit 23 and a (k+1)th mapping unit 21.

[0189] In training machine translation models, to increase the diversity among the learning units and enable multiple units to learn different data characteristics, a bias module is added before each mapping unit. The bias module biases the input encoded word vectors, thereby increasing the diversity of the input mapping unit vectors and allowing the machine translation model to be trained based on diverse sample vectors.

[0190] Bias processing refers to the processing operation that changes the position or content represented by a vector through bias units to increase the diversity of input sample vectors. Bias processing includes at least one of the following: position swapping, order shuffling, and occlusion.

[0191] Step 603: The first sample vector after bias processing is mapped to obtain the second sample vector after mapping.

[0192] The biased first sample vector output by the (k+1)th layer bias unit is input into the (k+1)th layer mapping unit. After the mapping processing by the mapping unit, the (k+1)th layer mapping unit outputs the mapped sample sub-vector.

[0193] It should be noted that the mapped sample subvector output by the (k+1)th layer mapping unit is input into the (k+2)th layer coding layer. The bias unit in the (k+2)th layer coding layer performs bias processing on the mapped sample subvector and inputs it into the mapping unit in the (k+2)th layer coding layer, thereby outputting the mapped sample subvector corresponding to the (k+2)th layer coding layer. This process continues until the encoder outputs the final mapped second sample vector.

[0194] The input vector of the first-layer bias unit is the sample encoded word vector after embedding encoding. In addition to the first-layer bias unit, the output of the k-th layer mapping unit is the input of the k+1-th layer bias unit.

[0195] The bias unit performs a bias operation on the input vector using Equation 11:

[0196] Formula 11:

[0197] Among them, Bias i (X k This indicates that the vector output by the i-th mapping unit in the k-th layer is biased. This indicates the processing result of the mapping unit.

[0198] It is understandable that the i-th mapping unit of the k-th layer will As the input vector, use Equations 1 and 2 to... The mapping process is performed to obtain the first mapped sample vector. Formula 3 is used to fuse the mapped sample vectors output by each mapping unit to output the vector corresponding to the (k+1)th layer mapping unit.

[0199] Step 604: Decode the mapped second sample vector to output the translated target sentence of the source sentence.

[0200] The mapped second sample vector is input into the decoder, which outputs the target sentence. Illustratively, the decoder operates in the same way during the training and testing phases (usage phases) of the machine translation model: it decodes the input vector to obtain decoded word vectors, predicts the probability of target words in the target sentence based on these word vectors, and outputs the prediction results.

[0201] Step 605: Train the machine translation model based on the translation results and the target sentence of the sample to obtain the trained machine translation model.

[0202] The machine translation model is trained by comparing the translation results of the source sentences with the errors between the target sentences, allowing the machine translation model to gradually converge and thus obtain the trained machine translation model.

[0203] Biased MUTEs are formed by constructing biased units based on the multi-module Transformer model.

[0204] In summary, the method in this embodiment biases the input sample encoded word vectors through the machine translation model, enabling each unit in each layer of the machine translation model to learn more features. This ensures that the machine translation model obtains diverse sample vectors, which is beneficial for training the machine translation model based on diverse sample vectors, resulting in more accurate target sentences output by the trained machine translation model.

[0205] based on Figure 6 In an optional embodiment, the bias unit included in the machine translation model can process the sample encoded word vectors as follows:

[0206] 1. Position swapping (replacement) processing

[0207] The bias unit is invoked to randomly swap the positions of any p sample encoded word vectors to obtain the swapped sample encoded word vectors; the swapped sample encoded word vectors are used as the first sample vector after bias processing, where p is a positive integer.

[0208] As an illustration, p is set to 2, and the bias unit randomly swaps the positions of any two sample encoded word vectors. For example, in the sentence "I am a student," the positions of the sample encoded word vectors corresponding to "I" and "student" are swapped. This is illustrated through... Figure 7 The swapping unit 231 shown implements position swapping processing.

[0209] 2. Disrupt the order processing

[0210] The bias unit is invoked to shuffle a portion of the sample encoded word vectors to obtain shuffled sample encoded word vectors; the shuffled sample encoded word vectors are then used as the first sample vector after bias processing.

[0211] As an illustration, the bias unit shuffles the order of some vectors corresponding to "a student" in the sentence "I am a student." This order can be randomly generated by the machine translation model or pre-set by the model. For example, it can change the order of the vectors corresponding to "a student" to the order of the vectors corresponding to "student name 'life'". This is illustrated through... Figure 7 The shuffled unit (Autoshuffle) 232 shown in

[0212] 3. Masking processing

[0213] The bias unit is called to randomly select some vectors from the sample encoded word vectors for masking processing to obtain the masked sample encoded word vectors; the masked sample encoded word vectors are used as the first sample vectors after bias processing.

[0214] Schematically, the bias unit randomly masks the partial vectors corresponding to "学" and "是" in "我是一名学生。" (Mask), or replaces the partial vectors corresponding to "学" and "是" with masking processing. In this way, the masked sample encoded word vectors are obtained. Schematically, the masking processing can be implemented by Figure 7 the exchange unit 231 or the shuffled unit 232 shown in Figure 7 (not shown in

[0215] In summary, through the above three methods of bias processing, each unit between layers in the machine translation model learns more features, and the machine translation model is trained based on the diversity of sample vectors, so that the trained machine translation model can translate various types of source sentences, improving the accuracy of the target sentence output.

[0216] Based on Figure 6 In the optional embodiment of

[0217] The bias unit includes a first type of bias unit (exchange unit) and a second type of bias unit (shuffled unit).

[0218] In response to the selection probability corresponding to the i-th mapping unit being greater than the probability threshold, the i-th bias unit corresponding to the i-th mapping unit performs target bias processing, that is, the i-th bias unit performs bias processing corresponding to the i-th bias unit on the sample encoded word vectors.

[0219] In response to the selection probability corresponding to the i-th mapping unit being less than the probability threshold, the i-th bias unit corresponding to the i-th mapping unit does not perform target bias processing, that is, the i-th bias unit does not perform bias processing corresponding to the i-th bias unit on the sample encoded word vectors.

[0220] The selection probability is used to represent the probability of whether the bias unit performs the target bias operation. A probability threshold pB is set for the bias unit. When the selection probability corresponding to the mapping unit is greater than the probability threshold, the bias unit performs the target bias operation.

[0221] Schematic illustration: A probability threshold of 0.6 is set for the bias unit. The bias unit corresponding to the i-th mapping unit is a swapping unit. When the selection probability corresponding to the i-th mapping unit is 0.85, since the selection probability of 0.85 is greater than the probability threshold of 0.6, the swapping unit will perform a swapping operation (target bias operation) on the input encoded sample word vector. Conversely, when the selection probability corresponding to the i-th mapping unit is 0.47, since the selection probability of 0.47 is less than the probability threshold of 0.6, the swapping unit will not perform a swapping operation (target bias operation) on the input encoded sample word vector.

[0222] It should be noted that the input vector of the bias unit in the first layer is the sample encoded word vector, and the input vector of the bias unit in the (k+1)th layer is the output vector of the mapping unit in the kth layer. k is a positive integer, and the initial value of k is 2.

[0223] No bias processing is performed during the testing phase, thus introducing inconsistencies between the training and testing phases. During the training phase, the choice of whether to perform bias processing and what type of bias processing to perform can be made by selecting the probability pB.

[0224] Because of the introduction of selection probability, the bias unit may perform bias processing or not. Therefore, the machine translation model includes: an N-layer cascaded encoder, which includes N layers of bias units, N layers of mapping units, and N layers of sequential dependency units. The k-th layer bias unit comprises I parallel bias units, and its output vector is the input vector of the k-th layer mapping unit. When the first layer bias unit does not perform bias operation, the encoded word vector is directly input into the first layer mapping unit. The structures of the mapping units and sequential dependency units in the machine translation model have been described in detail in the above embodiments and will not be repeated here.

[0225] based on Figure 6 In an optional embodiment, step 605 above can be replaced by the following steps:

[0226] Step 6051: Calculate the error between the translation result and the target sentence in the sample.

[0227] The error between the translation result and the target sentence is calculated by calculating the first objective function and the second objective function.

[0228] Step S1: Obtain the first objective function and the learning parameters of the machine translation model. The first objective function is used to calculate the normalized fitting matrix according to the Lipshitz continuity condition. The fitting matrix is ​​used to fit the permutation matrix. The permutation matrix is ​​used to sort the mapped second sample vectors.

[0229] As can be seen from the above embodiments, the first objective function is L in Formula 10. p To illustrate, the parameters to be learned for the machine translation model are λ.

[0230] Step S2: Obtain the second objective function, which includes the cross-entropy loss function.

[0231] The second objective function characterizes how close the actual output of the machine translation model is to the expected result. Illustratively, the second objective function is defined by L... CE The cross-entropy loss function is as follows:

[0232]

[0233] Where M represents the number of categories, y ic The variable y represents (0 or 1) the class of sample i if it is the same as the class of sample i. ic The value of y is 1. If the class is different from the class of sample i, then y ic The value of p is 0. ic This represents the probability that sample i belongs to category c.

[0234] Step S3: Calculate the error between the translation result and the sample target sentence based on the first objective function, the parameters to be learned, and the second objective function.

[0235] Based on steps S1 and S2, the final objective function of the machine translation model is the sum of the first objective function and the second objective function, expressed by Formula Twelve:

[0236] Formula 12: L = L CE +λL p

[0237] Among them, L p Let L represent the first objective function, λ represent the parameters to be learned in the machine translation model, and L represent the second objective function. CE This represents the second objective function.

[0238] Step 6052: Train the machine translation model based on the error results to obtain the trained machine translation model.

[0239] Formula 12 is used to calculate the translation result of the source sentence and the error result of the target sentence. The error result is then used to backpropagate and train the machine translation model to obtain the trained machine translation model.

[0240] In summary, the method of this embodiment utilizes the objective function to calculate the result error, enabling the machine translation model to be trained based on the accurately calculated result error, thus making the target sentence output by the trained machine translation model more accurate.

[0241] It should be noted that the structure of the machine translation model during training is different from that of the machine translation model during use.

[0242] During the training phase of the machine translation model, the model comprises N concatenated coding layers, each consisting of N bias units and N mapping units. The k-th bias unit comprises I parallel-connected bias units, and the k-th mapping unit comprises I parallel-connected mapping units. The output vector of the ith bias unit is the input vector of the ith mapping unit. The I parallel-connected bias units include swapping units and out-of-order units, with the sum of the number of swapping units and out-of-order units being I. The ith mapping unit comprises the ith multi-head attention mechanism unit and the ith feedforward neural network unit, with the output vector of the ith multi-head attention mechanism unit being the input vector of the ith feedforward neural network unit. Here, i, k, N, and I are all positive integers, i ≤ I, k ≤ N, and the initial value of k is 2.

[0243] In the application phase of the machine translation model, the machine translation model includes N layers of cascaded coding layers. Each N-layer cascaded coding layer comprises N layers of mapping units and N layers of sequential dependency units. The k-th mapping unit includes a k-th multi-head attention mechanism unit and a k-th feedforward neural network unit. The k-th multi-head attention mechanism unit comprises I parallel-connected multi-head attention mechanism units, and the k-th feedforward neural network unit comprises I parallel-connected feedforward neural network units. The output vector of the ith multi-head attention mechanism unit is the input vector of the ith feedforward neural network unit. The k-th sequential dependency unit includes a k-th sorting unit and a k-th dependency unit. The k-th sorting unit comprises I parallel-connected sorting units, and the k-th dependency unit comprises I parallel-connected dependency units. The output vector of the ith sorting unit is the input vector of the ith dependency unit. The output vector of the k-th feedforward neural network unit is the input vector of the k-th sorting unit. Where i, k, N, and I are all positive integers, i ≤ I, k ≤ N, and the initial value of k is 2.

[0244] In the embodiments of this application, the encoder and decoder have the same structure. The layered structure inside the decoder is similar to the implementation of the encoder, and will not be described again here.

[0245] In some embodiments, an additional selection probability is set for the bias unit, which represents the probability that the structure of the sample vector can achieve maximum sample diversity when biased.

[0246] If the selection probability of the bias unit corresponding to the i-th mapping unit is greater than the probability threshold, then the bias unit of the first type is used as the bias unit corresponding to the i-th mapping unit.

[0247] To illustrate, a probability threshold of 60% is set for the bias unit. When the sample vector structure undergoes position swapping, if the probability of obtaining maximum sample diversity is 75%, then the unit corresponding to the i-th mapping unit is selected as the swapping unit. That is, the i-th bias unit is the swapping unit, and the output vector of the i-th swapping unit is the input vector of the i-th mapping unit.

[0248] If the selection probability of the bias unit corresponding to the i-th mapping unit is less than or equal to the probability threshold, then the bias unit of the second type is used as the bias unit corresponding to the i-th mapping unit.

[0249] To illustrate, a probability threshold of 60% is set for the bias unit. When the sample vector structure undergoes position swapping, the probability of obtaining maximum sample diversity is 30%. In this case, the unit corresponding to the i-th mapping unit is selected as a scrambled unit. That is, the i-th bias unit is a scrambled unit, and the output vector of the i-th scrambled unit is the input vector of the i-th mapping unit.

[0250] The sample encoded word vector is biased according to the bias processing corresponding to the i-th bias unit.

[0251] like Figure 8 The training performance of the machine translation model provided in this embodiment is shown in comparison with a benchmark machine translation model. The parameters in Table 1 are explained below:

[0252] #Param. indicates the number of learnable parameters in the machine translation model; △ indicates the change in the Bilingual Evaluation Understudy (BLEU) average of the bilingual translation quality aids compared to the Transformer model (512-dimensional) on both the validation and test sets; bolded numbers represent the best-performing machine translation model; "+" indicates significantly better performance than the Transformer model (512-dimensional, P < 0.05); "++" indicates significantly better performance than the Transformer model based on relative position encoding (512-dimensional, P < 0.01); MT02 to MT08 represent different test sets. Neural Machine Translation (NMT) systems can be trained to map from one sequence to another, and the output can be a variable-length sequence, exhibiting superior performance in translation, dialogue, and text summarization.

[0253] Depend on Figure 8The training performance comparison table shown demonstrates that the machine translation model provided in this embodiment outperforms other machine translation models. The machine translation model provided in this embodiment improves training performance by reasonably increasing the number of parameters to be learned. Under various evaluation metrics, the machine translation model provided in this embodiment significantly outperforms other machine translation models.

[0254] Figure 9 This is a structural block diagram of a machine translation apparatus provided in an exemplary embodiment of this application. The apparatus includes:

[0255] The first acquisition module 910 is used to acquire the encoded word vector of the source sentence after embedding encoding;

[0256] The first mapping module 920 is used to map the encoded word vectors to obtain the mapped intermediate vectors;

[0257] The fitting module 930 is used to fit the mapped intermediate vectors according to the dependency relationship between them to obtain the fitted vector. The dependency relationship is used to characterize the association relationship between words in the source sentence.

[0258] The first decoding module 940 is used to decode the fitted vector and output the target sentence after the source sentence has been translated.

[0259] In an optional embodiment, the machine translation model includes N cascaded coding layers, where the input of the first coding layer is the encoded word vector, the output vector of the k-th coding layer is the input vector of the (k+1)-th coding layer, and the k-th coding layer includes a k-th mapping unit and a k-th sequential dependency unit, where k is a positive integer and the initial value of k is 2.

[0260] The fitting module 930 is used to call the (k+1)th layer mapping unit to perform mapping processing on the fitting vector output by the kth layer sequential dependency unit, and output the first intermediate vector after mapping.

[0261] The fitting module 930 is used to call the k+1th layer sequential dependency unit to obtain the dependency relationship between the mapped first intermediate vectors, and to perform fitting processing on the mapped first intermediate vectors according to the dependency relationship, and output the first fitting vector corresponding to the k+1th layer coding layer.

[0262] The fitting module 930 is used to repeatedly execute the steps of outputting the first intermediate vector after mapping and outputting the first fitted vector until the Nth coding layer outputs the fitted vector; where N is a positive integer and k≤N.

[0263] In an optional embodiment, the device includes a sorting module 950, and the (k+1)th layer sequential dependency unit includes a (k+1)th layer sorting unit and a (k+1)th layer dependency unit.

[0264] The sorting module 950 is used to call the (k+1)th layer sorting unit to sort the mapped first intermediate vector and output the sorted first intermediate vector; and to call the (k+1)th layer dependency unit to build dependency relationships for the sorted first intermediate vector.

[0265] The fitting module 930 is used to call the (k+1)th layer dependency unit to perform fitting processing on the sorted first intermediate vector according to the dependency relationship, and output the first fitting vector.

[0266] In an optional embodiment, the (k+1)th layer sorting unit includes I parallel sorting units, the (k+1)th layer dependency unit includes I parallel dependency units, and the sorted first intermediate vector includes a first sub-vector and a second sub-vector. The first sub-vector is the vector output by the ith sorting unit in the (k+1)th layer sorting unit, and the second sub-vector is the vector output by the ith sorting unit in the (k+1)th layer sorting unit. The output vector of the ith sorting unit is the input vector of the ith dependency unit; the output vector of the ith dependency unit is obtained by fitting the output vector of the ith sorting unit.

[0267] The fitting module 930 is used to call the i-th dependency unit and the (i+1)-th dependency unit to obtain the dependency relationship between the first sub-vector and the second sub-vector; call the (i+1)-th dependency unit to perform fitting processing on the first sub-vector and the second sub-vector, and output the third sub-vector; repeat the above steps of obtaining the dependency relationship between the first sub-vector and the second sub-vector, and the steps of outputting the third sub-vector, until the I dependency unit outputs the first fitting vector corresponding to the (k+1)-th layer dependency unit, the first fitting vector including the third sub-vector, where I and i are both positive integers, and i≤I.

[0268] In an optional embodiment, the (k+1)th layer mapping unit includes a (k+1)th layer feedforward neural network unit, and the (k+1)th layer feedforward neural network unit includes I feedforward neural network units connected in parallel.

[0269] The sorting module 950 is used to obtain I mapped first intermediate vectors output by I feedforward neural network units; concatenate the I mapped intermediate vectors to obtain concatenated intermediate vectors; obtain a permutation matrix, and fit the permutation matrix with a fitting matrix to obtain a fitted permutation matrix. The permutation matrix is ​​used to sort the I mapped intermediate vectors, and the fitting matrix is ​​used to fit the permutation matrix; input the concatenated intermediate vectors and the fitted permutation matrix into the (k+1)th layer sorting unit, and output the sorted first vector.

[0270] In an optional embodiment, the fitting matrix is ​​obtained as follows:

[0271] The sorting module 950 is used to normalize the fitting matrix to obtain a normalized fitting matrix; calculate the first objective function corresponding to the normalized fitting matrix according to the Lipshitz continuity condition; and obtain the converged fitting matrix in response to the convergence of the first objective convergence function.

[0272] In an optional embodiment, the machine translation model includes a decoder;

[0273] The first decoding module 940 is used to input the fitted vector into the decoder and output the decoded word vector; and to perform probability prediction on the target words in the target sentence based on the decoded word vector and output the prediction result.

[0274] In summary, the apparatus provided in this embodiment determines the dependencies between the mapped intermediate vectors and uses these dependencies to fit the mapped intermediate vectors, outputting a large number of vectors that match the target sentence. This enables the machine translation model to translate the source sentence based on the dependencies between the vectors, rather than simply stacking the output decoded word vectors, making the target sentence output by the machine translation model more accurate.

[0275] The encoder employs a layered structure, with each coding layer acting as a unit. The input vector of the previous layer becomes the output vector of the next. Each coding layer's units perform mapping, sorting, and encoding processes on the input vector, repeating this process until the Nth coding layer outputs the fitted vector. This ensures the accuracy of the vector during transmission, resulting in a more accurate target sentence output by the machine translation model.

[0276] The first intermediate vector after mapping is sorted and fitted by the sorting and dependency units in the sequential dependency unit. The dependency unit is used to build dependency relationships on the sorted first intermediate vector, so that there are dependency relationships between the output vectors of each layer of dependency unit. The source sentence is translated according to the dependency relationship, instead of simply stacking the output decoded word vectors, so that the target short sentence output by the machine translation model is more accurate.

[0277] The decoder is used to predict the probability of target words in the target sentence and output the prediction results. This makes the prediction results output based on the prediction probability more accurate, thereby ensuring that the target sentence output by the machine translation model is more accurate.

[0278] By breaking down a single coding layer into smaller units and fusing the vectors output by multiple units, the residuals between the outputs of the decomposed units are compensated for, thereby accurately obtaining the output vector corresponding to a single coding layer. This makes the source sentence output from the encoder vector more accurate.

[0279] By sorting the first intermediate vector after mapping using a permutation matrix, the machine translation model can select the translation that is closer to the source sentence based on more vector representations, thereby improving the accuracy of the target sentence output.

[0280] By fitting a fitting matrix to fit a permutation matrix, the permutation matrix can be adapted to the mapped vectors of each layer. At the same time, the fitting matrix is ​​used to fit the permutation matrix into a continuous matrix, so that the machine translation model can be optimized and updated based on the continuous permutation matrix, thereby improving the accuracy of the target sentence output.

[0281] It should be noted that the machine translation device provided in the above embodiments is only an example of the division of the above functional modules. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. In addition, the machine translation device and the machine translation method embodiments provided in the above embodiments belong to the same concept, and the specific implementation process can be found in the method embodiments, which will not be repeated here.

[0282] Figure 10 This is a structural block diagram of a machine translation model training apparatus provided in an exemplary embodiment of this application. The apparatus includes:

[0283] The second acquisition module 1010 is used to acquire the sample encoded word vector after the source sentence of the sample is embedded and encoded, and the source sentence has a corresponding translation result;

[0284] The bias module 1020 is used to bias the sample encoded word vector to obtain the first sample vector after bias processing.

[0285] The second mapping module 1030 is used to perform mapping processing on the biased first sample vector to obtain the mapped second sample vector.

[0286] The decoding module 1040 is used to decode the mapped second sample vector and output the translated target sentence of the sample source sentence.

[0287] Training module 1050 is used to train the machine translation model based on the translation results and the target sentences of the samples, so as to obtain the trained machine translation model.

[0288] In an optional embodiment, the machine translation model includes bias units;

[0289] The bias module 1020 is used to call the bias unit to randomly swap the positions of any p sample encoded word vectors to obtain the swapped sample encoded word vectors; and use the swapped sample encoded word vectors as the first sample vector after bias processing, where p is a positive integer; or, call the bias unit to shuffle the order of some vectors of the sample encoded word vectors to obtain shuffled sample encoded word vectors; and use the shuffled sample encoded word vectors as the first sample vector after bias processing; or, call the bias unit to randomly select some vectors from the sample encoded word vectors for masking processing to obtain masked sample encoded word vectors; and use the masked sample encoded word vectors as the first sample vector after bias processing.

[0290] In an optional embodiment, the machine translation model includes N cascaded coding layers, each of which includes N mapping units. Each mapping unit includes I parallel mapping units, each mapping unit corresponds to I bias units, and each mapping unit corresponds to a selection probability, where I is a positive integer. The bias units are set with probability thresholds, and the selection probability is used to characterize the probability that the structure of the sample vector will obtain the maximum sample diversity when bias processing is performed, where i is a positive integer and i≤I.

[0291] The bias module 1020 is configured to, in response to the selection probability of the bias unit corresponding to the i-th mapping unit being greater than a probability threshold, use a first type of bias unit as the bias unit corresponding to the i-th mapping unit; in response to the selection probability of the bias unit corresponding to the i-th mapping unit being less than or equal to a probability threshold, use a second type of bias unit as the bias unit corresponding to the i-th mapping unit; and perform bias processing on the sample encoded word vector according to the bias processing corresponding to the i-th bias unit.

[0292] In an optional embodiment, the training module 1050 is used to calculate the error between the translation result and the target sentence of the sample; and to train the machine translation model based on the error result to obtain the trained machine translation model.

[0293] In an optional embodiment, the second acquisition module 1010 is used to acquire a first objective function and the learning parameters of the machine translation model. The first objective function is used to calculate a normalized fitting matrix based on the Lipshitz continuity condition. The fitting matrix is ​​used to fit a permutation matrix, and the permutation matrix is ​​used to sort the mapped second sample vectors. The second objective function is acquired, and the second objective function includes a cross-entropy function. The training module 1050 is used to calculate the error result between the translation result and the sample target sentence based on the first objective function, the learning parameters, and the second objective function.

[0294] In an optional embodiment, the machine translation model includes N layers of cascaded coding layers, each layer comprising N layers of bias units and N layers of mapping units. The k-th bias unit comprises I parallel-connected bias units, and the k-th mapping unit comprises I parallel-connected mapping units. The output vector of the ith bias unit is the input vector of the ith mapping unit, where k is a positive integer. The I parallel-connected bias units include swapping units and out-of-order units, with the sum of the number of swapping units and out-of-order units being I. The ith mapping unit comprises an ith multi-head attention mechanism unit and an ith feedforward neural network unit, where the output vector of the ith multi-head attention mechanism unit is the input vector of the ith feedforward neural network unit. Here, i, N, and I are all positive integers, and i ≤ I.

[0295] In an optional embodiment, the machine translation model includes N cascaded coding layers, each of which includes N mapping units. Each mapping unit includes I parallel mapping units, each mapping unit corresponds to I bias units, and each mapping unit has a selection probability, where I is a positive integer. The bias units are set with a probability threshold, and the selection probability is used to characterize the probability that the bias unit performs the bias processing, where i is a positive integer and i≤I.

[0296] The bias module 1020 is configured to bias the sample encoded word vector according to the i-th bias unit corresponding to the i-th mapping unit in response to the selection probability corresponding to the i-th mapping unit being greater than the probability threshold.

[0297] In summary, the apparatus provided in this embodiment biases the input sample encoded word vectors through the machine translation model, enabling each unit in each layer of the machine translation model to learn more features. This ensures that the machine translation model obtains diverse sample vectors, which is beneficial for training the machine translation model based on diverse sample vectors, resulting in more accurate target sentences output by the trained machine translation model.

[0298] By employing the three bias processing methods described above, each unit in each layer of the machine translation model learns more features, enabling the machine translation model to be trained based on the diversity of sample vectors, thereby making the target sentence output by the trained machine translation model more accurate.

[0299] It should be noted that the machine translation model training device provided in the above embodiments is only an example of the division of the above functional modules. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. In addition, the machine translation model training device and the machine translation model training method embodiments provided in the above embodiments belong to the same concept, and their specific implementation process can be found in the method embodiments, which will not be repeated here.

[0300] Figure 11 This illustration shows a schematic diagram of a server provided in an exemplary embodiment of this application. The server can be as follows: Figure 1 Server 120 in the computer system 100 shown.

[0301] Server 1100 includes a central processing unit (CPU) 1101, a system memory 1104 including random access memory (RAM) 1102 and read-only memory (ROM) 1103, and a system bus 1105 connecting the system memory 1104 and the CPU 1101. Server 1100 also includes a basic input / output system (I / O system) 1106 that facilitates the transfer of information between various devices within the computer, and a mass storage device 1107 for storing the operating system 1113, application programs 1114, and other program modules 1115.

[0302] The basic input / output system 1106 includes a display 1108 for displaying information and an input device 1109 for user input, such as a mouse or keyboard. Both the display 1108 and the input device 1109 are connected to the central processing unit 1101 via an input / output controller 1110 connected to the system bus 1105. The basic input / output system 1106 may also include the input / output controller 1110 for receiving and processing input from multiple other devices such as a keyboard, mouse, or electronic stylus. Similarly, the input / output controller 1110 also provides output to a display screen, printer, or other types of output devices.

[0303] Mass storage device 1107 is connected to central processing unit 1101 via a mass storage controller (not shown) connected to system bus 1105. Mass storage device 1107 and its associated computer-readable media provide non-volatile storage for server 1100. That is, mass storage device 1107 may include computer-readable media (not shown) such as hard disk or compact disc read-only memory (CD-ROM) drives.

[0304] Computer-readable media can include computer storage media and communication media. Computer storage media include volatile and non-volatile, removable and non-removable media implemented using any method or technology for storing information such as computer-readable instructions, data structures, program modules, or other data. Computer storage media include RAM, ROM, erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other solid-state storage technologies, CD-ROM, digital versatile optical disc (DVD), or solid-state drives (SSD), other optical storage, magnetic tape cassettes, magnetic tape, disk storage, or other magnetic storage devices. Random access memory can include resistive random access memory (ReRAM) and dynamic random access memory (DRAM). Of course, those skilled in the art will recognize that computer storage media are not limited to the above-mentioned types. The system memory 1104 and the mass storage device 1107 mentioned above can be collectively referred to as memory.

[0305] According to various embodiments of this application, server 1100 can also be connected to a remote computer on a network, such as the Internet. That is, server 1100 can be connected to network 1112 via network interface unit 1111 connected to system bus 1105, or it can also use network interface unit 1111 to connect to other types of networks or remote computer systems (not shown).

[0306] The aforementioned memory also includes one or more programs, which are stored in the memory and configured to be executed by the CPU.

[0307] In an optional embodiment, a computer device is provided, the computer device including a processor and a memory, the memory storing at least one instruction, at least one program, code set or instruction set, the at least one instruction, at least one program, code set or instruction set being loaded and executed by the processor to implement the machine translation method and the machine translation model training method as described above.

[0308] In an optional embodiment, a computer-readable storage medium is provided, which stores at least one instruction, at least one program, code set, or instruction set, wherein the at least one instruction, at least one program, code set, or instruction set is loaded and executed by a processor to implement the machine translation method and the machine translation model training method as described above.

[0309] Optionally, the computer-readable storage medium may include: read-only memory (ROM), random access memory (RAM), solid-state drives (SSDs), or optical discs, etc. The random access memory may include resistive random access memory (ReRAM) and dynamic random access memory (DRAM). The sequence numbers of the embodiments described above are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0310] This application also provides a computer program product or computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the machine translation method and the machine translation model training method described above.

[0311] Those skilled in the art will understand that all or part of the steps of the above embodiments can be implemented by hardware or by a program instructing related hardware. The program can be stored in a computer-readable storage medium, such as a read-only memory, a disk, or an optical disk.

[0312] The above description is merely an optional embodiment of this application and is not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. A machine translation method, characterized in that, The method includes: Obtain the encoded word vectors of the source sentence after embedding encoding; The encoded word vectors are mapped to obtain the mapped intermediate vectors; The method is executed by a machine translation model, which includes N cascaded coding layers. The input of the first coding layer is the encoded word vector, and the output vector of the k-th coding layer is the input vector of the (k+1)-th coding layer. The k-th coding layer includes a k-th mapping unit and a k-th sequential dependency unit, where k is a positive integer and the initial value of k is 2. The mapping process is performed on the fitted vector output by the k+1 layer mapping unit, and the first intermediate vector after mapping is output. The k+1 layer sequential dependency unit is called to obtain the dependency relationship between the mapped first intermediate vectors, and the first intermediate vectors after mapping are fitted according to the dependency relationship to output the first fitted vector corresponding to the k+1 layer coding layer. Repeat the steps of outputting the first intermediate vector after mapping and outputting the first fitted vector as described above until the Nth encoding layer outputs the fitted vector; where N is a positive integer and k≤N, and the dependency relationship is used to characterize the association relationship between words in the source sentence; The fitted vector is decoded to output the target sentence after translation of the source sentence.

2. The method according to claim 1, characterized in that, The (k+1)th layer sequential dependency unit includes the (k+1)th layer sorting unit and the (k+1)th layer dependency unit; The step of calling the (k+1)th layer sequential dependency unit to obtain the dependency relationship between the mapped first intermediate vectors, and performing the fitting process on the mapped first intermediate vectors according to the dependency relationship to output the first fitted vector includes: The k+1th layer sorting unit is invoked to sort the mapped first intermediate vector, and the sorted first intermediate vector is output. The (k+1)th layer dependency unit is invoked to construct the dependency relationship for the sorted first intermediate vector; The (k+1)th layer dependency unit is invoked to perform the fitting process on the sorted first intermediate vector according to the dependency relationship, and the first fitted vector is output.

3. The method according to claim 2, characterized in that, The (k+1)th layer sorting unit includes I parallel connected sorting units, the (k+1)th layer dependency unit includes I parallel connected dependency units, and the sorted first intermediate vector includes a first sub-vector and a second sub-vector. The first sub-vector is the vector output by the ith sorting unit in the (k+1)th layer sorting unit, and the second sub-vector is the vector output by the ith sorting unit in the (k+1)th layer sorting unit. The output vector of the ith sorting unit is the input vector of the ith dependency unit; the output vector of the ith dependency unit is obtained by fitting the output vector of the ith sorting unit. The step of calling the (k+1)th layer dependency unit to perform the fitting process on the sorted first intermediate vector according to the dependency relationship, and outputting the first fitted vector, includes: The dependency relationship between the first sub-vector and the second sub-vector is obtained by calling the i-th dependency unit and the (i+1)-th dependency unit; Call the (i+1)th dependency unit to perform the fitting process on the first sub-vector and the second sub-vector, and output the third sub-vector; Repeat the steps of obtaining the dependency relationship between the first sub-vector and the second sub-vector, and the step of outputting the third sub-vector, until the I dependency units output the first fitted vector corresponding to the (k+1)th layer dependency unit, the first fitted vector including the third sub-vector, where I and i are both positive integers and i≤I.

4. The method according to claim 2 or 3, characterized in that, The (k+1)th layer mapping unit includes a (k+1)th layer feedforward neural network unit, and the (k+1)th layer feedforward neural network unit includes I parallel feedforward neural network units; The step of calling the (k+1)th level sorting unit to sort the mapped first intermediate vector and outputting the sorted first intermediate vector includes: Obtain the first intermediate vectors after mapping from the outputs of the I feedforward neural network units; The I mapped intermediate vectors are concatenated to obtain the concatenated intermediate vector; Obtain the permutation matrix, and fit the permutation matrix with a fitting matrix to obtain the fitted permutation matrix. The permutation matrix is ​​used to sort the I mapped intermediate vectors, and the fitting matrix is ​​used to fit the permutation matrix. The concatenated intermediate vector and the fitted permutation matrix are input into the (k+1)th layer sorting unit, and the sorted first vector is output.

5. The method according to claim 4, characterized in that, The fitting matrix was obtained in the following way: The fitting matrix is ​​normalized to obtain the normalized fitting matrix; The first objective function corresponding to the normalized fitting matrix is ​​calculated based on the Lipschitz continuity condition; In response to the convergence of the first objective convergence function, the converged fitting matrix is ​​obtained.

6. The method according to any one of claims 1 to 3, characterized in that, The machine translation model includes a decoder; The process of decoding the fitted vector to output the translated target sentence from the source sentence includes: The fitted vector is input into the decoder, and the decoded word vector is output. Based on the decoded word vectors, the probability of predicting the target words in the target sentence is performed, and the prediction results are output.

7. A method for training a machine translation model, characterized in that, The method includes: Obtain the sample encoded word vector after embedding and encoding the source sentence, and the source sentence corresponds to the translation result; The machine translation model includes bias units; The machine translation model further includes N cascaded encoding layers. The input of the first encoding layer is the sample encoded word vector, and the output vector of the k-th encoding layer is the input vector of the (k+1)-th encoding layer. The k-th encoding layer includes a k-th mapping unit and a k-th sequential dependency unit. The k-th sequential dependency unit is used to obtain the dependency relationship between the mapped vectors and perform fitting processing according to the dependency relationship. k is a positive integer and the initial value of k is 2, N is a positive integer, and k≤N; The bias unit is invoked to randomly swap the positions of any p sample encoded word vectors to obtain the swapped sample encoded word vectors; the swapped sample encoded word vectors are used as the first sample vector after the bias processing, where p is a positive integer; The first sample vector after bias processing is mapped to obtain the second sample vector after mapping. The mapped second sample vector is decoded to output the translated target sentence of the source sentence. The machine translation model is trained based on the translation results and the target sentence of the sample to obtain the trained machine translation model.

8. The method according to claim 7, characterized in that, The method further includes: The bias unit is invoked to shuffle a portion of the sample encoded word vectors to obtain shuffled sample encoded word vectors; the shuffled sample encoded word vectors are used as the first sample vector after the bias processing. or, The bias unit is invoked to randomly select a portion of the vectors from the sample encoded word vectors for masking processing, resulting in masked sample encoded word vectors; the masked sample encoded word vectors are then used as the first sample vector after the bias processing.

9. The method according to claim 7 or 8, characterized in that, The step of training the machine translation model based on the translation result and the target sentence of the sample to obtain the trained machine translation model includes: Calculate the error between the translation result and the target sentence in the sample; The machine translation model is trained based on the error results to obtain the trained machine translation model.

10. The method according to claim 9, characterized in that, The calculation of the error between the translation result and the target sentence in the sample includes: Obtain a first objective function and the learning parameters of the machine translation model. The first objective function is used to calculate a normalized fitting matrix based on the Lipshitz continuity condition. The fitting matrix is ​​used to fit a permutation matrix, and the permutation matrix is ​​used to sort the mapped second sample vector. Obtain the second objective function; The error between the translation result and the sample target sentence is calculated based on the first objective function, the parameters to be learned, and the second objective function.

11. A machine translation device, characterized in that, The device includes: The first acquisition module is used to acquire the encoded word vectors of the source sentence after embedding and encoding. The first mapping module is used to map the encoded word vectors to obtain the mapped intermediate vectors; The machine translation model includes N cascaded encoding layers. The input of the first encoding layer is the encoded word vector, and the output vector of the k-th encoding layer is the input vector of the (k+1)-th encoding layer. The k-th encoding layer includes a k-th mapping unit and a k-th sequential dependency unit, where k is a positive integer and the initial value of k is 2. The fitting module is used to call the (k+1)th layer mapping unit to map the fitting vector output by the kth layer sequential dependency unit, and output the first intermediate vector after mapping. The fitting module is used to call the (k+1)th layer sequential dependency unit to obtain the dependency relationship between the mapped first intermediate vectors, and perform the fitting process on the mapped first intermediate vectors according to the dependency relationship, and output the first fitting vector corresponding to the (k+1)th layer coding layer. The fitting module is used to repeatedly execute the steps of outputting the first intermediate vector after mapping and outputting the first fitted vector until the Nth coding layer outputs the fitted vector; where N is a positive integer and k≤N, and the dependency relationship is used to characterize the association relationship between words in the source sentence; The first decoding module is used to decode the fitted vector and output the target sentence after the source sentence has been translated.

12. The apparatus according to claim 11, characterized in that, The device includes a sorting module, and the (k+1)th layer sequential dependency unit includes a (k+1)th layer sorting unit and a (k+1)th layer dependency unit. The sorting module is used to call the k+1th layer sorting unit to sort the mapped first intermediate vector and output the sorted first intermediate vector. The (k+1)th layer dependency unit is invoked to construct the dependency relationship for the sorted first intermediate vector; The fitting module is used to call the (k+1)th layer dependency unit to perform the fitting process on the sorted first intermediate vector according to the dependency relationship, and output the first fitted vector.

13. The apparatus according to claim 12, characterized in that, The (k+1)th layer sorting unit includes I parallel connected sorting units, the (k+1)th layer dependency unit includes I parallel connected dependency units, and the sorted first intermediate vector includes a first sub-vector and a second sub-vector. The first sub-vector is the vector output by the ith sorting unit in the (k+1)th layer sorting unit, and the second sub-vector is the vector output by the ith sorting unit in the (k+1)th layer sorting unit. The output vector of the ith sorting unit is the input vector of the ith dependency unit; the output vector of the ith dependency unit is obtained by fitting the output vector of the ith sorting unit. The fitting module is used to call the i-th dependency unit and the (i+1)-th dependency unit to obtain the dependency relationship between the first sub-vector and the second sub-vector; Call the (i+1)th dependency unit to perform the fitting process on the first sub-vector and the second sub-vector, and output the third sub-vector; Repeat the steps of obtaining the dependency relationship between the first sub-vector and the second sub-vector, and the step of outputting the third sub-vector, until the I dependency units output the first fitted vector corresponding to the (k+1)th layer dependency unit, the first fitted vector including the third sub-vector, where I and i are both positive integers and i≤I.

14. The apparatus according to claim 12 or 13, characterized in that, The (k+1)th layer mapping unit includes a (k+1)th layer feedforward neural network unit, and the (k+1)th layer feedforward neural network unit includes I parallel feedforward neural network units; The sorting module is used to obtain the first intermediate vectors after mapping from the outputs of the I feedforward neural network units; The I mapped intermediate vectors are concatenated to obtain the concatenated intermediate vector; Obtain the permutation matrix, and fit the permutation matrix with the fitting matrix to obtain the fitted permutation matrix. The permutation matrix is ​​used to sort the I mapped intermediate vectors, and the fitting matrix is ​​used to fit the permutation matrix. The concatenated intermediate vector and the fitted permutation matrix are input into the (k+1)th layer sorting unit, and the sorted first vector is output.

15. The apparatus according to claim 14, characterized in that, The fitting matrix was obtained in the following way: The sorting module is used to normalize the fitting matrix to obtain a normalized fitting matrix. The first objective function corresponding to the normalized fitting matrix is ​​calculated based on the Lipschitz continuity condition; In response to the convergence of the first objective convergence function, the converged fitting matrix is ​​obtained.

16. The apparatus according to any one of claims 11 to 13, characterized in that, The machine translation model includes a decoder; The first decoding module is used to input the fitted vector into the decoder and output the decoded word vector; Based on the decoded word vectors, the probability of predicting the target words in the target sentence is performed, and the prediction results are output.

17. A training device for a machine translation model, characterized in that, The device includes: The second acquisition module is used to acquire the sample encoded word vector after the source sentence of the sample has been embedded and encoded, and the source sentence corresponds to the translation result; The machine translation model includes bias units; The machine translation model further includes N cascaded encoding layers. The input of the first encoding layer is the sample encoded word vector, and the output vector of the k-th encoding layer is the input vector of the (k+1)-th encoding layer. The k-th encoding layer includes a k-th mapping unit and a k-th sequential dependency unit. The k-th sequential dependency unit is used to obtain the dependency relationship between the mapped vectors and perform fitting processing according to the dependency relationship. k is a positive integer and the initial value of k is 2, N is a positive integer, and k≤N; The bias module is used to call the bias unit to randomly swap the positions of any p sample encoded word vectors to obtain the swapped sample encoded word vectors; the swapped sample encoded word vectors are used as the first sample vector after the bias processing, where p is a positive integer; The second mapping module is used to perform mapping processing on the biased first sample vector to obtain the mapped second sample vector. The second decoding module is used to decode the mapped second sample vector and output the translated sample target sentence from the sample source sentence. The machine translation model is trained based on the translation results and the target sentence of the sample to obtain the trained machine translation model.

18. The apparatus according to claim 17, characterized in that, The bias module is further configured to call the bias unit to shuffle the order of a portion of the sample encoded word vectors to obtain shuffled sample encoded word vectors; and to use the shuffled sample encoded word vectors as the first sample vector after the bias processing. or, The bias unit is invoked to randomly select a portion of the vectors from the sample encoded word vectors for masking processing, resulting in masked sample encoded word vectors; the masked sample encoded word vectors are then used as the first sample vector after the bias processing.

19. The apparatus according to claim 17 or 18, characterized in that, The training module is used to calculate the error between the translation result and the target sentence in the sample. The machine translation model is trained based on the error results to obtain the trained machine translation model.

20. The apparatus according to claim 19, characterized in that, The second acquisition module is used to acquire a first objective function and the learning parameters of the machine translation model. The first objective function is used to calculate a normalized fitting matrix based on the Lipshitz continuity condition. The fitting matrix is ​​used to fit a permutation matrix. The permutation matrix is ​​used to sort the mapped second sample vector. Obtain the second objective function; The training module is used to calculate the error between the translation result and the sample target sentence based on the first objective function, the parameters to be learned, and the second objective function.

21. A computer device, characterized in that, The computer device includes a processor and a memory, the memory storing at least one instruction, at least one program, a code set, or an instruction set, the at least one instruction, the at least one program, the code set, or the instruction set being loaded and executed by the processor to implement the machine translation method as described in any one of claims 1 to 6 and the training method for the machine translation model as described in any one of claims 7 to 10.

22. A computer-readable storage medium, characterized in that, The readable storage medium stores at least one instruction, at least one program, code set, or instruction set, wherein the at least one instruction, the at least one program, the code set, or the instruction set is loaded and executed by a processor to implement the machine translation method as described in any one of claims 1 to 6 and the training method for the machine translation model as described in any one of claims 7 to 10.

Images