Translation method and device based on multi-modal machine learning, equipment and storage medium

Abstract

The application discloses a translation method and device based on multi-modal machine learning, equipment and storage medium, relates to the technical field of artificial intelligence. The method performs semantic association on n modal source languages through a multi-modal graph representation layer, constructs a semantic association graph, connects semantic nodes of the same mode by using a first connection edge and connects semantic nodes of different modes by using a second connection edge in the semantic association graph, fully represents the semantic association between the source languages of multiple modes by using the semantic association graph, then fully fuses feature vectors in the semantic association graph by using a multi-modal fusion encoder to obtain encoded feature vectors, and further obtains a more accurate target sentence after decoding processing of the encoded feature vectors. The target sentence is closer to the content, emotion and language environment and the like expressed by the multi-modal source languages.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a translation method, apparatus, device, and storage medium based on multimodal machine learning. Background Technology

[0002] Machine translation is the process of using computers to convert one natural language into another.

[0003] In some application scenarios, machine translation models can be used to translate source languages ​​with multiple different expressions into target languages, that is, to translate multimodal source languages ​​into target languages. For example, an image and its corresponding English annotation are obtained, and a machine translation model is used to extract features from the image and the English annotation respectively. Then, the extracted features are fused, and the French annotation corresponding to the image and the English annotation is translated based on the fused features.

[0004] The above translation methods are not sufficient to establish a semantic connection between the images and the English annotations. Summary of the Invention

[0005] This application provides a translation method, apparatus, device, and storage medium based on multimodal machine learning. It can perform sufficient semantic fusion of multiple modalities of source language during feature encoding, making the target sentence decoded from the encoded vector more closely resemble the content and sentiment expressed in the source language. The technical solution is as follows:

[0006] According to one aspect of this application, a translation method based on multimodal machine learning is provided, applied in a computer device equipped with a multimodal fusion encoder and decoder, the method comprising:

[0007] A semantic association graph is constructed by semantically associating source languages ​​of n modalities through a multimodal graph representation layer. The semantic association graph includes semantic nodes of n different modalities, first connecting edges for connecting semantic nodes of the same modality, and second connecting edges for connecting semantic nodes of different modalities, where n is a positive integer greater than 1.

[0008] The first word vector is extracted from the semantic association graph through the first word vector layer;

[0009] The first word vector is encoded using a multimodal fusion encoder to obtain the encoded feature vector;

[0010] The decoder is invoked to decode the encoded feature vector, resulting in the translated target sentence.

[0011] According to another aspect of this application, a translation apparatus based on multimodal machine learning is provided, the apparatus comprising:

[0012] The semantic association module is used to perform semantic association on source languages ​​of n modalities through the multimodal graph representation layer, and construct a semantic association graph. The semantic association graph includes semantic nodes of n different modalities, first connecting edges for connecting semantic nodes of the same modality, and second connecting edges for connecting semantic nodes of different modalities, where n is a positive integer greater than 1.

[0013] The feature extraction module is used to extract the first word vector from the semantic association graph through the first word vector layer;

[0014] The vector encoding module is used to encode the first word vector through a multimodal fusion encoder to obtain the encoded feature vector;

[0015] The vector decoding module is used to call the decoder to decode the encoded feature vectors and obtain the translated target sentence.

[0016] According to another aspect of this application, a computer device is provided, the computer device comprising:

[0017] Memory;

[0018] A processor connected to memory;

[0019] The processor is configured to load and execute executable instructions to implement the multimodal machine learning-based translation method as described in the preceding aspect and its alternative embodiments.

[0020] According to another aspect of this application, a computer-readable storage medium is provided, wherein at least one instruction, at least one program, code set, or instruction set is stored therein, wherein the at least one instruction, at least one program, code set, or instruction set is loaded and executed by a processor to implement the translation method based on multimodal machine learning as described in the preceding aspect and its optional embodiments.

[0021] The beneficial effects of the technical solutions provided in this application include at least the following:

[0022] This method uses a multimodal graph representation layer to semantically associate source languages ​​of n modalities, constructing a semantic association graph. In the semantic association graph, a first connecting edge connects semantic nodes of the same modality, and a second connecting edge connects semantic nodes of different modalities. The semantic association graph fully represents the semantic associations between source languages ​​of multiple modalities. Then, a multimodal fusion encoder performs full semantic fusion on the feature vectors in the semantic association graph to obtain encoded feature vectors. After decoding the encoded feature vectors, a more accurate target sentence is obtained. This target sentence is closer to the content, emotion, and linguistic environment expressed by the multimodal source languages. Attached Figure Description

[0023] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0024] Figure 1 This is a schematic diagram of the structure of a multimodal machine translation model provided in an exemplary embodiment of this application;

[0025] Figure 2 This is a schematic diagram of the structure of a computer system provided in an exemplary embodiment of this application;

[0026] Figure 3 This is a flowchart of a translation method based on multimodal machine learning provided in an exemplary embodiment of this application;

[0027] Figure 4 This is a flowchart illustrating the construction of a semantic association graph provided in an exemplary embodiment of this application;

[0028] Figure 5 This is a flowchart of a translation method based on multimodal machine learning provided in another exemplary embodiment of this application;

[0029] Figure 6 This is a flowchart of a translation method based on multimodal machine learning provided in another exemplary embodiment of this application;

[0030] Figure 7 This is a schematic diagram of the structure of a multimodal machine translation model provided in another exemplary embodiment of this application;

[0031] Figure 8 This is a graph of the model test results provided in an exemplary embodiment of this application;

[0032] Figure 9 This is a graph of the model test results provided in another exemplary embodiment of this application;

[0033] Figure 10 This is a graph of the model test results provided in another exemplary embodiment of this application;

[0034] Figure 11 This is a block diagram of a translation apparatus based on multimodal machine learning provided in an exemplary embodiment of this application;

[0035] Figure 12 This is a schematic diagram of the structure of a server provided in an exemplary embodiment of this application. Detailed Implementation

[0036] To make the objectives, technical solutions, and advantages of this application clearer, the embodiments of this application will be described in further detail below with reference to the accompanying drawings.

[0037] The terms used in this application are explained as follows:

[0038] Artificial Intelligence (AI) is a technological science that utilizes digital computers or computers-controlled machines to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. In other words, AI is a comprehensive technology within computer science that attempts to understand the essence of intelligence and produce new intelligent machines that can react in a way similar to human intelligence. AI studies the design principles and implementation methods of various intelligent machines, enabling them to possess perception, reasoning, and decision-making capabilities.

[0039] Artificial intelligence (AI) is a comprehensive discipline encompassing a wide range of fields, including both hardware and software technologies. Fundamental AI technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interactive systems, and mechatronics. AI software technologies primarily include computer vision, speech processing, natural language processing, and machine learning / deep learning.

[0040] Natural Language Processing (NLP) is an important area within computer science and artificial intelligence. It studies the theories and methods for enabling effective communication between humans and computers using natural language. NLP is a science that integrates linguistics, computer science, and mathematics. Therefore, research in this field involves natural language—the language people use in daily life—and thus it has a close relationship with linguistic research. NLP technologies typically include text processing, semantic understanding, machine translation, question answering, and knowledge graphs.

[0041] Machine learning (ML) is a multidisciplinary field involving probability theory, statistics, approximation theory, convex analysis, and algorithm complexity theory. It specifically studies how computers can simulate or implement human learning behavior to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental way to endow computers with intelligence; its applications span all areas of artificial intelligence. Machine learning and deep learning typically include techniques such as artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and analogical learning.

[0042] This application provides a multimodal machine translation model that can accurately translate n source languages ​​of different modalities into target sentences. Here, modality refers to the way language is expressed, such as using graphical or textual representations; source language refers to the language to be translated, which includes textual sentences in the first language class and non-textual sentences in the second language class; target sentence refers to a translated sentence in textual form in the second language class, which differs from the first language class. For example, the source language includes an English sentence and its accompanying image; the multimodal machine translation model can translate the English sentence and its corresponding Chinese sentence into Chinese.

[0043] like Figure 1 The diagram shows a schematic of the structure of a multimodal machine translation model 100 provided in an exemplary embodiment of this application. The multimodal machine translation model 100 includes a multimodal graph representation layer 101, a first word vector layer 102, a multimodal fusion encoder 103, and a decoder 104.

[0044] Multimodal graph representation layer 101 is used to semantically associate source languages ​​of n modalities and construct a semantic association graph. The semantic association graph includes semantic nodes of n different modalities, a first connecting edge for connecting semantic nodes of the same modality, and a second connecting edge for connecting semantic nodes of different modalities, where n is a positive integer greater than 1.

[0045] The first word vector layer 102 is used to extract the first word vector from the semantic association graph;

[0046] A multimodal fusion encoder 103 is used to encode the first word vector to obtain an encoded feature vector;

[0047] Decoder 104 is used to decode the encoded feature vector to obtain the translated target sentence.

[0048] In some optional embodiments, the multimodal graph representation layer 101 is used to extract semantic nodes from the source language of each modality to obtain n sets of semantic nodes corresponding to the source languages ​​of n modalities.

[0049] The multimodal graph representation layer 101 is used to connect the semantic nodes of the same modality to the n groups of semantic nodes using a first connecting edge, and to connect the semantic nodes of different modalities to the n groups of semantic nodes using a second connecting edge, so as to obtain a semantic association graph.

[0050] In some alternative embodiments, the source languages ​​of the n modalities include a first source language in text form and a second source language in non-text form, and the n sets of semantic nodes include a first semantic node and a second semantic node;

[0051] The multimodal graph representation layer 101 is used to extract a first semantic node from a first source statement and extract candidate semantic nodes from a second source language; calculate a first probability distribution of candidate semantic nodes according to the semantic association between the first semantic node and the candidate semantic nodes; and determine a second semantic node from the candidate semantic nodes according to the first probability distribution.

[0052] In some optional embodiments, the multimodal graph representation layer 101 is used to determine the i-th type of first connection edge corresponding to the i-th modality, and to use the i-th type of first connection edge to connect the semantic nodes in the i-th group of semantic nodes within the same modality, where i is a positive integer less than or equal to n.

[0053] In some optional embodiments, the multimodal fusion encoder 103 includes e cascaded encoding modules 1031, each encoding module 1031 including n intramodal fusion layers 11 and n intermodal fusion layers 12 corresponding one-to-one with n modes, where e is a positive integer;

[0054] The first encoding module 1031 is used to input the first word vector into the n intramodal fusion layers 11 in the first encoding module respectively, and perform semantic fusion within the same modality on the first word vector through the n intramodal fusion layers 11 respectively to obtain n first hidden layer vectors corresponding one-to-one with the n modalities.

[0055] The first encoding module 1031 is used to input n first hidden layer vectors into each intermodal fusion layer 12 in the first encoding module, and to perform semantic fusion between different modalities on the n first hidden layer vectors through each intermodal fusion layer 12 to obtain n first intermediate vectors that correspond one-to-one with the n modalities.

[0056] The j-th encoding module 1031 is used to perform the j-th encoding process on the n first intermediate vectors until the last encoding module outputs n encoded feature vectors that correspond one-to-one with the n modes, where j is a positive integer greater than 1 and less than or equal to e.

[0057] In some optional embodiments, each encoding module 1031 further includes: n first vector transformation layers 13 corresponding one-to-one with the n modes;

[0058] The encoding module 1031 is also used to input the n first intermediate vectors into the n first vector conversion layer 13 corresponding to the respective modes for nonlinear conversion, so as to obtain the n first intermediate vectors after nonlinear conversion.

[0059] In some alternative embodiments, the hierarchical structure of each of the e cascaded encoding modules 1031 is identical.

[0060] In some alternative embodiments, different or the same self-attention functions are set in different intramodal fusion layers, and different or the same feature fusion functions are set in different intermodal fusion layers.

[0061] In some alternative embodiments, the multimodal machine translation model 100 further includes a second word vector layer 105 and a classifier 106, and the decoder 104 includes d cascaded decoding modules 1042, where d is a positive integer;

[0062] The second word vector layer 105 is used to obtain the first target word, which is the translated word in the target sentence; feature extraction is performed on the first target word to obtain the second word vector;

[0063] Decoder 104 is used to extract features by combining the second word vector with the encoded feature vector through d concatenated decoding modules 1042 to obtain the decoded feature vector;

[0064] Classifier 106 is used to calculate the probability distribution corresponding to the decoded feature vector and determine the second target word after the first target word based on the probability distribution.

[0065] In some alternative embodiments, each of the d serially connected decoding modules 1042 includes a first self-attention layer 21 and a second self-attention layer 22.

[0066] The first decoding module 1042 is used to input the second word vector into the first self-attention layer 21 in the first decoding module 1042, and extract features from the second word vector through the first self-attention layer 21 to obtain the second hidden layer vector.

[0067] The first decoding module 1042 is used to input the second hidden layer vector and the encoded feature vector into the second self-attention layer 22 in the first decoding module 1042, and to extract features by combining the second hidden layer vector and the encoded feature vector through the second self-attention layer 22 to obtain the second intermediate vector;

[0068] The kth decoding module is used to input the second intermediate vector into the kth decoding module 1042 for the kth decoding process until the last decoding module outputs the decoded feature vector, where k is a positive integer greater than 1 and less than or equal to d.

[0069] In some alternative embodiments, each decoding module 1042 further includes: a second vector conversion layer 23;

[0070] The decoding module 1042 is used to input the second intermediate vector into the second vector conversion layer 23 for nonlinear conversion to obtain the nonlinearly converted second intermediate vector.

[0071] In summary, the multimodal machine translation model provided in this embodiment performs semantic association on n source languages ​​through a multimodal graph representation layer, constructing a semantic association graph. In the semantic association graph, a first connecting edge connects semantic nodes of the same modality, and a second connecting edge connects semantic nodes of different modalities. The semantic association graph fully represents the semantic association between source languages ​​of multiple modalities. Then, a multimodal fusion encoder performs full semantic fusion on the feature vectors in the semantic association graph to obtain encoded feature vectors. After decoding the encoded feature vectors, a more accurate target sentence is obtained. This target sentence is closer to the content, emotion, and linguistic environment comprehensively expressed by the multimodal source languages.

[0072] Please refer to Figure 2 The diagram illustrates the structure of a computer system provided in an exemplary embodiment of this application, the computer system including a terminal 220 and a server 240.

[0073] Terminal 220 has an operating system installed on it; the operating system has an application installed on it that supports translation of multimodal source languages. For example, the application may include instant messaging software, financial software, game software, shopping software, video playback software, community service software, audio software, educational software, payment software, and translation software, etc., and the multimodal source language translation function is integrated into the application.

[0074] Terminal 220 and server 240 are interconnected via a wired or wireless network. Server 240 includes at least one of a single server, multiple servers, a cloud computing platform, and a virtualization center. For example, server 240 includes a processor and a memory, wherein the memory stores a computer program, and the processor reads and executes the computer program to perform multimodal source language translation.

[0075] Optionally, server 240 undertakes the main computing work and terminal 220 undertakes the secondary computing work; or, server 240 undertakes the secondary computing work and terminal 220 undertakes the main computing work; or, server 240 and terminal 220 collaborate on computing using a distributed computing architecture.

[0076] In some optional embodiments, during the implementation of the multimodal language translation function described above, server 240 provides background services for the application on terminal 220. For example, terminal 220 collects source languages ​​in n modalities and sends these n source languages ​​to server 240, where server 240 executes the multimodal machine learning-based translation method provided in this application, where n is a positive integer greater than 1.

[0077] For example, terminal 220 includes a data transmission control; terminal 220 uploads two source languages ​​of different modalities, namely the sentence to be translated and the image matching the sentence to be translated, to server 240 through the aforementioned data transmission control, and server 240 executes the translation method based on multimodal machine learning provided in this application to translate the two source languages ​​of different modalities into the target sentence.

[0078] In some alternative embodiments, the source language may include speech signals; if the source language of n modalities includes speech signals, the terminal 220 or the server 240 first converts the speech signals into text before translating the source language of the n modalities. For example, the terminal 220 collects speech signals through a microphone, or the terminal 220 receives speech signals sent by other terminals.

[0079] The aforementioned translation method based on multimodal machine learning can be applied to multimedia news translation scenarios. For example, terminal 220 uploads multimedia news including text and images to server 240, and server 240 executes the translation method based on multimodal machine learning provided in this application to translate the first language text in the multimedia news into the second language text.

[0080] The aforementioned translation method based on multimodal machine learning can be applied to foreign language document translation scenarios. For example, terminal 220 uploads the text and corresponding illustrations from the foreign language document to server 240, and server 240 executes the translation method based on multimodal machine learning provided in this application to translate the first language text in the foreign language document into the second language text.

[0081] The aforementioned translation method based on multimodal machine learning can be applied to foreign language website translation scenarios. For example, terminal 220 collects text and text images from foreign language websites, uploads the text and text images to server 240, and server 240 executes the multimodal machine learning-based translation method provided in this application to translate the first language text in the foreign language website into the second language text, thereby realizing the translation of the foreign language website.

[0082] In some alternative embodiments, the terminal 220 displays the translated text in either voice or text format.

[0083] It should be noted that, in some optional embodiments, the terminal 220 can execute the translation method based on multimodal machine learning provided in this application, and then translate the source languages ​​of n modalities.

[0084] Terminal 220 can refer to one of multiple terminals, but this embodiment only uses terminal 220 as an example. Terminal 220 may include at least one of the following: a smartphone, tablet computer, e-book reader, Moving Picture Experts Group Audio Layer III (MP3) player, Moving Picture Experts Group Audio Layer IV (MP4) player, laptop computer, desktop computer, or notebook computer. The following embodiments use smartphones and personal computer devices as examples to illustrate terminal 220.

[0085] Those skilled in the art will understand that the number of terminals 220 described above can be more or less. For example, there may be only one terminal, or there may be dozens or hundreds of terminals, or even more. This application does not limit the number or type of terminals 220 in its embodiments.

[0086] Please refer to Figure 3 The diagram illustrates a flowchart of a translation method based on multimodal machine learning provided in an exemplary embodiment of this application, which is applied to, for example... Figure 2 The computer device shown includes a multimodal fusion encoder and decoder, and the computer device may include a terminal or a server. The method includes:

[0087] Step 301: Semantically associate the source languages ​​of n modalities through a multimodal graph representation layer to construct a semantic association graph.

[0088] The semantic association graph mentioned above includes n semantic nodes of different modalities, a first connecting edge for connecting semantic nodes of the same modality, and a second connecting edge for connecting semantic nodes of different modalities, where n is a positive integer greater than 1.

[0089] The computer device extracts semantic nodes from the source language of each modality through a multimodal graph representation layer, obtaining n sets of semantic nodes corresponding to the source languages ​​of n modalities. The multimodal graph representation layer uses a first connecting edge to connect the semantic nodes within the same modality and a second connecting edge to connect the semantic nodes between different modalities, thus obtaining a semantic association graph.

[0090] Optionally, the source languages ​​of the n modalities include a first source language in text form and a second source language in non-text form, and the n sets of semantic nodes include first semantic nodes and second semantic nodes; the computer device extracts the first semantic node from the first source language and extracts candidate semantic nodes from the second source language through the multimodal graph representation layer; the multimodal graph representation layer is called to calculate the first probability distribution of the candidate semantic nodes according to the semantic association between the first semantic node and the candidate semantic node; the multimodal graph representation layer is called to determine the second semantic node from the candidate semantic node according to the first probability distribution.

[0091] In the extraction of semantic nodes from the first source language in text form, the computer device performs word segmentation on the first source language to obtain m words after segmentation. The m words are the first semantic nodes in the first source language, where m is a positive integer.

[0092] For the extraction of semantic nodes in a non-textual form of a second source language, the computer device extracts the target that corresponds to the semantics of at least one word among m words from the second source language. This target is the second semantic node in the second source language.

[0093] For example, such as Figure 4 The source languages ​​of the two modalities include the image to be translated 31 and the sentence to be translated 32. The content of the sentence to be translated 32 includes "Two boys are playing with a toy car." Each English word corresponds to a first semantic node, namely Vx1, Vx2, Vx3, Vx4, Vx5, Vx6, Vx7, and Vx8. The computer device extracts candidate images from the image to be translated 31 based on the semantics of the semantic nodes. It calculates a first probability distribution based on the semantic association between the semantic nodes and the candidate images. Based on the first probability distribution, it determines the target image 1 and target image 2 corresponding to the semantics of Vx1 and Vx2, and the target image 3 corresponding to the semantics of Vx6, Vx7, and Vx8 from the candidate images. The Vo1, Vo2, and Vo3 corresponding to the target images 1, 2, and 3 are the three second semantic nodes in the image to be translated 31. The computer device uses a first connecting edge (solid line) to connect each pair of Vx1, Vx2, Vx3, Vx4, Vx5, Vx6, Vx7, and Vx8 for intramodal semantic connection, uses a first connecting edge to connect each pair of Vo1, Vo2, and Vo3 for intramodal semantic connection, and uses a second connecting edge (dashed line) to connect the first semantic node and the second semantic node for intermodal semantic connection.

[0094] Optionally, different modalities are configured with different first connection edges; when the computer device performs intramodal connections on semantic nodes, it determines the i-th type of first connection edge corresponding to the i-th modality through the multimodal graph representation layer, and uses the i-th type of first connection edge to connect semantic nodes in the same modality to the i-th group of semantic nodes, where i is a positive integer less than or equal to n.

[0095] Optionally, the source languages ​​of the two modalities are translated. If the source languages ​​of the two modalities are text and images, respectively, the computer device uses a visual grounding tool to establish the semantic association between the source languages ​​of the two modalities and construct a semantic association graph.

[0096] Step 302: Extract the first word vector from the semantic association graph through the first word vector layer.

[0097] For example, the computer device processes the semantic association graph using word embedding to obtain a first word vector; word embedding refers to mapping words to word vectors, and optionally, the word embedding method includes at least one of the following four:

[0098] Word embedding is performed using a neural network model;

[0099] Word embedding is performed by reducing the dimensionality of the word co-occurrence matrix;

[0100] Word embedding is performed using a probabilistic model;

[0101] Word embedding is performed by considering the semantics of the context in which the word is located.

[0102] For example, one-hot encoding is used to represent words in the source language in text form, and then word embedding is performed using an embedding matrix.

[0103] Step 303: Encode the first word vector using a multimodal fusion encoder to obtain the encoded feature vector.

[0104] The computer device extracts intramodal features from the first word vector using a multimodal fusion encoder, and then performs intermodal feature fusion on the vector obtained from the feature extraction.

[0105] For example, taking n=3, the multimodal fusion encoder includes a first feature extraction function corresponding to the first modality, a second feature extraction function corresponding to the second modality, and a third feature extraction function corresponding to the third modality. The computer device extracts features from the first word vector within the first modality using the first feature extraction function, extracts features from the first word vector within the second modality using the second feature extraction function, and extracts features from the first word vector within the third modality using the third feature extraction function, ultimately obtaining three hidden layer functions. The multimodal fusion encoder also includes a first feature fusion function corresponding to the first modality, a second feature fusion function corresponding to the second modality, and a third feature fusion function corresponding to the third modality. The computer device performs intermodal feature fusion on the three hidden layer functions using the first feature fusion function, the second feature fusion function, and the third feature fusion function, obtaining the three fused hidden layer vectors, which are the encoded feature vectors.

[0106] Step 304: Call the decoder to decode the encoded feature vector to obtain the translated target sentence.

[0107] The computer device calls the decoder to decode the encoded feature vector to obtain the translated target statement, which is the statement obtained by translating the source languages ​​of n modalities into a specified language class.

[0108] In summary, the translation method based on multimodal machine learning provided in this embodiment performs semantic association on n source languages ​​through a multimodal graph representation layer, constructing a semantic association graph. In the semantic association graph, a first connecting edge connects semantic nodes of the same modality, and a second connecting edge connects semantic nodes of different modalities. The semantic association graph fully represents the semantic association between multiple source languages. Then, a multimodal fusion encoder performs full semantic fusion on the feature vectors in the semantic association graph to obtain encoded feature vectors. After decoding the encoded feature vectors, a more accurate target sentence is obtained. This target sentence is closer to the content, emotion, and linguistic environment comprehensively expressed by the multimodal source languages.

[0109] based on Figure 3 The multimodal fusion encoder includes e cascaded encoding modules, each encoding module including n intramodal fusion layers and n intermodal fusion layers corresponding one-to-one with the n modes, where e is a positive integer; therefore, step 303 may include step 3031, such as Figure 5 The steps are as follows:

[0110] Step 3031: The first word vector is fused within and between modalities e times by e concatenated encoding modules to obtain the encoded feature vector.

[0111] Intramodal fusion refers to semantic fusion between first word vectors within the same modality, while intermodal fusion refers to semantic fusion between first word vectors in different modalities.

[0112] For example, the intra-modal and inter-modal fusion of the above-mentioned encoded feature vectors can be achieved through the following steps:

[0113] 1) Input the first word vector into the n intramodal fusion layers in the first encoding module respectively. Perform semantic fusion within the same modality on the first word vector through the n intramodal fusion layers respectively to obtain n first hidden layer vectors that correspond one-to-one with the n modalities.

[0114] For example, the computer device inputs the first word vector into the first intramodal fusion layer in the first encoding module, performs intramodal semantic fusion on the first word vector through the first intramodal fusion layer, and obtains the first first hidden layer vector; inputs the first word vector into the second intramodal fusion layer in the first encoding module, performs intramodal semantic fusion on the first word vector through the second intramodal fusion layer, and obtains the second first hidden layer vector; ...; inputs the first word vector into the nth intramodal fusion layer in the first encoding module, performs intramodal semantic fusion on the first word vector through the nth intramodal fusion layer, and obtains the nth first hidden layer vector.

[0115] The intramodal fusion layer includes a feature extraction function, which optionally includes a self-attention function. Alternatively, different intramodal fusion layers may use different or the same self-attention function. It should be noted that the difference in self-attention functions refers to the difference in their intrinsic parameters; if different modalities correspond to different self-attention functions, then the intrinsic parameters of the function will also be different for each modality.

[0116] 2) Input the n first hidden layer vectors into each intermodal fusion layer in the first coding module, and perform semantic fusion between different modalities on the n first hidden layer vectors through each intermodal fusion layer to obtain n first intermediate vectors that correspond one-to-one with the n modalities.

[0117] For example, the computer device inputs n first hidden layer vectors into the first intermodal fusion layer in the first encoding module, performs intermodal semantic fusion on the n first hidden layer vectors through the first intermodal fusion layer, and obtains the first first intermediate vector corresponding to the first modality; inputs the n first hidden layer vectors into the second intermodal fusion layer in the first encoding module, performs intermodal semantic fusion on the n first hidden layer vectors through the second intermodal fusion layer, and obtains the second first intermediate vector corresponding to the second modality; ...; inputs the n first hidden layer vectors into the nth intermodal fusion layer in the first encoding module, performs intermodal semantic fusion on the n first hidden layer vectors through the nth intermodal fusion layer, and obtains the nth first intermediate vector corresponding to the nth modality.

[0118] The intermodal fusion layer is configured with a feature fusion function. Optionally, different intermodal fusion layers may use different or the same feature fusion functions. It should be noted that the difference in feature fusion functions refers to different parameters within the function, or different calculation methods.

[0119] Optionally, each encoding module further includes: n first vector conversion layers corresponding one-to-one with the n modes; after obtaining the n first intermediate vectors, the computer device also inputs the n first intermediate vectors into the n first vector conversion layers corresponding to their respective modes for nonlinear conversion to obtain the nonlinearly converted n first intermediate vectors.

[0120] 3) Input the n first intermediate vectors into the j-th encoding module for the j-th encoding process until the last encoding module outputs n encoded feature vectors that correspond one-to-one with the n modes.

[0121] The computer device inputs n intermediate vectors into the second encoding module for a second encoding process, resulting in n re-encoded first intermediate vectors; ...; inputs the n re-encoded first intermediate vectors into the j-th encoding module for the j-th encoding process, resulting in n re-encoded first intermediate vectors; ...; inputs the n re-encoded first intermediate vectors into the e-th encoding module for the e-th encoding process, resulting in n encoded feature vectors; where j is a positive integer greater than 1 and less than or equal to e. Optionally, the hierarchical structure of each of the e concatenated encoding modules is the same, that is, the j-th encoding module processes the first intermediate vectors according to the steps of the first encoding module, until the last encoding module outputs the encoded feature vectors.

[0122] For example, in this embodiment, a self-attention mechanism is used to model semantic information within the same modality. The j-th encoding module then calculates the first hidden layer vector corresponding to the text statement. The formula is:

[0123]

[0124] in, This refers to the first word vector corresponding to the text statement or the first intermediate vector output by the (j-1)th encoding module; x is used to identify the semantic nodes of the text statement and the vectors calculated from the semantic nodes of the text statement; MultiHead(Q,K,V) is the multi-attention mechanism modeling function, taking the triple (Queries,Key,Values) as input, Q as the query matrix, K as the key matrix, and V as the value matrix, where Q, K, and V are derived from... It is obtained by calculating with the parameter vector.

[0125] The j-th multimodal fusion encoder calculates the first hidden layer vector corresponding to the image. The formula is:

[0126]

[0127] in, This refers to the first word vector corresponding to the image or the first intermediate vector output by the (j-1)th encoding module;

[0128] In this embodiment, a gate-based cross-modal fusion mechanism is also used to model semantic fusion between multiple modalities. Then, the j-th encoding module calculates the first intermediate vector or encoded feature vector corresponding to the text sentence. The formula is:

[0129]

[0130]

[0131] Where A represents a set, corresponding to, It is the first semantic node The set of neighbor nodes in the semantic association graph; This represents the u-th semantic node of the text statement, where u is a positive integer; It is the semantic representation vector of the s-th semantic node of the image in the j-th encoding module; It is the semantic representation vector of the u-th semantic node of the text statement in the j-th encoding module; and `o` is the parameter matrix; `⊙` represents the XOR operation; `Sigmoid()` is an sigmoid function; `o` is used to identify the semantic nodes of the image and the vectors calculated from the semantic nodes. The first intermediate vector or encoded feature vector corresponding to the image is also calculated in the same way. I will not go into details here.

[0132] After multimodal fusion, this embodiment also uses a feedforward neural network (FFN) to generate the final encoded feature vector. The encoded feature vectors corresponding to the text sentences and the images are as follows:

[0133]

[0134]

[0135] in, {} represents a set. This represents the encoded feature vector corresponding to the u-th semantic node of the text statement in the j-th encoding module. It is the encoded feature vector corresponding to the s-th semantic node of the image in the j-th encoding module.

[0136] In summary, the translation method based on multimodal machine learning provided in this embodiment performs semantic association on n source languages ​​through a multimodal graph representation layer, constructing a semantic association graph. In the semantic association graph, a first connecting edge connects semantic nodes of the same modality, and a second connecting edge connects semantic nodes of different modalities. The semantic association graph fully represents the semantic association between multiple source languages. Then, a multimodal fusion encoder performs full semantic fusion on the feature vectors in the semantic association graph to obtain encoded feature vectors. After decoding the encoded feature vectors, a more accurate target sentence is obtained. This target sentence is closer to the content, emotion, and linguistic environment comprehensively expressed by the multimodal source languages.

[0137] The multimodal fusion encoder in this method includes e cascaded encoding modules. Each encoding module includes an intramodal fusion layer and an intermodal fusion layer. By fusing features alternately within and between modalities multiple times, a more complete semantic fusion encoding feature vector is obtained, which can then decode a more accurate target sentence corresponding to the source language of n modalities.

[0138] based on Figure 3 The decoder also includes d cascaded decoding modules, where d is a positive integer. Therefore, step 304 can include steps 3041 to 3044, such as... Figure 6 The steps are as follows:

[0139] Step 3041: Obtain the first target word through the second word vector layer.

[0140] Here, the first target word is the translated word in the target sentence. The computer device translates each word in the target sentence one by one. After translating the r-th word in the target sentence, the r-th word is used as the first target word to translate the (r+1)-th word; that is, the computer device inputs the r-th word into the second word vector layer, where r is a non-negative integer.

[0141] Step 3042: Extract features from the first target word through the second word vector layer to obtain the second word vector.

[0142] For example, the computer device performs word embedding on the first target word through a second vector layer to obtain a second word vector. Word embedding is a technique for representing words as real number vectors in a vector space; in this embodiment, word embedding refers to the word vector mapped to a word; for example, mapping "I" to obtain the word vector (0.1, 0.5, 5), (0.1, 0.5, 5) is the word vector after embedding the word "I".

[0143] Step 3043: The second word vector is combined with the encoded feature vector through d concatenated decoding modules to extract features and obtain the decoded feature vector.

[0144] The computer device calls d concatenated decoding modules to pay attention to the encoded feature vector and the second word vector, and extracts the decoded feature vector.

[0145] Optionally, each of the d cascaded decoding modules includes a first self-attention layer, a second self-attention layer, and a second vector transformation layer. For the extraction of the decoding feature vector, the computer device inputs the second word vector into the first self-attention layer of the first decoding module, and performs feature extraction on the second word vector through the first self-attention layer to obtain the second hidden layer vector. The second hidden layer vector and the encoded feature vector are input into the second self-attention layer of the first decoding module, and the second self-attention layer combines the second hidden layer vector and the encoded feature vector to perform feature extraction to obtain the second intermediate vector. The second intermediate vector is input into the k-th decoding module for the k-th decoding process, until the last decoding module outputs the decoding feature vector, where k is a positive integer greater than 1 and less than or equal to d.

[0146] The first self-attention layer is used to perform self-attention on the second word vector to extract the second hidden vector; the second self-attention layer is used to use the language class of the target sentence to pay attention to the second hidden vector and the encoded feature vector to obtain the second intermediate vector. The first self-attention layer includes a first self-attention function, and the second self-attention layer includes a second self-attention function, with different parameters for the first and second self-attention functions.

[0147] Optionally, each decoding module further includes a second vector conversion layer; after calculating the second intermediate vector, the computer device also inputs the second intermediate vector into the second vector conversion layer for nonlinear conversion to obtain the nonlinearly converted second intermediate vector.

[0148] Step 3044: Input the decoded feature vector into the classifier, calculate the probability distribution corresponding to the decoded feature vector through the classifier, and determine the second target word after the first target word based on the probability distribution.

[0149] Optionally, the classifier includes a normalization (softmax) function. The computer device calculates the probability distribution corresponding to the decoded feature vector using the softmax function, and determines the second target word after the first target word based on the probability distribution corresponding to the decoded feature vector.

[0150] In summary, the translation method based on multimodal machine learning provided in this embodiment performs semantic association on n source languages ​​through a multimodal graph representation layer, constructing a semantic association graph. In the semantic association graph, a first connecting edge connects semantic nodes of the same modality, and a second connecting edge connects semantic nodes of different modalities. The semantic association graph fully represents the semantic association between multiple source languages. Then, a multimodal fusion encoder performs full semantic fusion on the feature vectors in the semantic association graph to obtain encoded feature vectors. After decoding the encoded feature vectors, a more accurate target sentence is obtained. This target sentence is closer to the content, emotion, and linguistic environment comprehensively expressed by the multimodal source languages.

[0151] This method also uses d decoding modules to repeatedly focus on the encoded feature vector and the second hidden layer vector using the language class of the target statement, in order to decode the target statement more accurately.

[0152] It should also be noted that, by comparing the multimodal machine translation model provided in this application with previous multimodal neural machine translation (NMT) models, it is clear that the multimodal machine translation model provided in this application achieves the best translation performance. For example, taking images and text as the source languages, the above comparison will be explained in detail below:

[0153] The multimodal machine translation model provided in this application is built on an attention-based encoding and decoding framework, with the objective function being the maximization of the log-likelihood of the training data. Essentially, the multimodal fusion encoder provided in this application can be viewed as a multimodal augmented graph neural network (GNN). To construct the multimodal fusion encoder, the input image and text are represented as a corresponding multimodal graph (i.e., a semantic association graph); then, based on the aforementioned multimodal graph, multiple multimodal fusion layers are superimposed to learn node (i.e., semantic node) representations, providing the decoder with attention-based context vectors.

[0154] I. For the construction of multimodal graphs, the multimodal graph is formally undirected and can be formally represented as G = (V, E). Here, in the node set V, each node represents a text word or visual object, and the following strategy is used to construct semantic relationships between nodes:

[0155] 1. Types of nodes;

[0156] (1) To make full use of text information, all words in the text are treated as individual text nodes, for example, in Figure 4 The multimodal graph consists of a total of 8 text nodes, each corresponding to a word in the input sentence (i.e., the sentence to be translated); (2) the Stanford parser is used to identify all noun phrases in the input sentence, and then the visual landing toolkit is applied to identify the bounding box (visual object) corresponding to each noun phrase in the input image (i.e., the image to be translated). After that, all detected visual objects are treated as independent visual nodes, for example, in Figure 4 Chinese text nodes Vx1 and Vx2 correspond to visual nodes Vo1 and Vo2, and text nodes Vx6, Vx7 and Vx8 correspond to visual node Vo3.

[0157] 2. To capture the various semantic relationships between multimodal semantic units, two types of edges (i.e., connecting edges) are used to connect nodes. The two types of edges in the edge set E include: (1) any two nodes in the same modality are connected by an intramodal edge (first connecting edge); (2) any text node and its corresponding visual node are connected by an intermodal edge (second connecting edge). For example, Figure 4 Vo1 and Vo2 are connected by an inner modal edge (solid line), and Vo1 and Vx1 are connected by an intermodal edge (solid line).

[0158] Second, for the embedding layer, a word embedding layer needs to be introduced to initialize the node state before inputting the multimodal graph into the stacked multimodal fusion layer. For each text node Vx u Set its initial state Hx uDefine the sum of word embeddings and positional embeddings. For the visual node Vo s initial state Ho s Visual features need to be extracted through the fully-connected layer of the Region Of Interest pooling (ROI) layer in Faster RCNN, and then the visual features are projected onto the same space as the text representation using a multilayer perceptron with the Rectified Linear Unit (ReLU) activation function.

[0159] RCCN stands for Rich Feature Hierarchies for accurate object detection and semantic segmentation.

[0160] III. Figure 7 The encoder shown on the left has an e-layer graph-based multimodal fusion layer stacked on top of the embedding layer 402 to encode the aforementioned multimodal graph. In the multimodal fusion layer, intra-modal and inter-modal fusion are performed sequentially to update the states of all nodes. Thus, the final node state simultaneously encodes contextual information within the same modality and cross-modal semantic information. Specifically, since visual nodes and text nodes are two semantic units containing different modal information, functions with similar operations but different parameters are used to model the node state update process.

[0161] For example, in j multimodal fusion layers, the text node state and visual node state The update mainly involves the following steps:

[0162] Step 1: Intramodal Fusion. In this step, self-attention is used to fuse information between adjacent nodes within the same modality to generate a contextual representation for each node. Formally, this involves the contextual representations of all text nodes. The calculation formula is as follows:

[0163]

[0164] Here, MultiHead(Q,K,V) is the multi-head attention mechanism modeling function (also called the multi-head self-attention function), taking the query matrix Q, key matrix K, and value matrix V as input. Similarly, the context representation of all visual nodes... The calculation formula is as follows:

[0165]

[0166] In particular, since the initial state of the visual object is extracted by a deep learning algorithm (deep CNNs), a simplified multi-head self-attention is applied to represent the initial state of the visual object, where the acquired linear item values ​​and the final output are removed.

[0167] Step 2: Intermodal Fusion. Intermodal feature fusion employs a cross-modal gating mechanism with element-wise operation characteristics to learn the semantic information of each node's cross-modal neighborhood. Specifically, text nodes Vx are generated. u State representation The method is as follows:

[0168]

[0169]

[0170] in, It is node Vx u The set of neighboring nodes in a multimodal graph; and It is a parameter matrix. Similarly, it generates the state representation of the text node Vos. The method is as follows:

[0171]

[0172]

[0173] in, It is a node Vo s The set of neighboring nodes in a multimodal graph; and It is a parameter matrix.

[0174] After the aforementioned multimodal fusion process, a feedforward neural network is used to generate the final Del hidden layer representation. Formalized text node states... and image node status The calculation process is as follows:

[0175]

[0176]

[0177] in, This indicates that the states of all text nodes and image nodes have been updated.

[0178] Fourth, the decoder is similar to a traditional Transformer decoder. Since visual information has been incorporated into all text nodes through multiple graph-based multimodal fusion layers, the decoder can dynamically utilize the multimodal context by focusing only on the text node state; that is, only the text node state is input into the decoder.

[0179] like Figure 7 As shown in the right part, d identical layers are stacked to generate the target-side hidden state, where each layer consists of three sub-layers. Specifically, the first two sub-layers are the masking self-attention E... j and codec attention T j To integrate target and source language side context:

[0180] E (j) =MultiHead(S j-1 S j-1 S j-1 );

[0181]

[0182] Among them, S (j-1) This represents the target-side hidden state in the (j-1)th layer. Specifically, S (0) It is the embedding vector of the target word in the input. The hidden states at the top layer of the decoder. Then, a fully connected feedforward neural network in the position orientation is used to generate S. (j) The formula is as follows:

[0183] S (j) =FFN(T (j) );

[0184] Finally, a softmax layer is used to define the probability distribution for generating the target statement, with the hidden state of the top layer as the basis. For input:

[0185]

[0186] Where X is the input sentence to be translated, I is the input image to be translated, Y is the target sentence (i.e. the sentence to be translated), and W and b are the parameters of the softmax layer.

[0187] In the experiments, the translation task involved translating English into French and German. The dataset used was the Multi30K dataset, where each image was paired with an English description, a human-translated German, and a French text. The training, validation, and test sets contained 29,000, 1,014, and 1,000 instances, respectively. Furthermore, we evaluated various models on the WMT17 test set and the fuzzy MSCOCO test set, which contained 1,000 and 461 instances, respectively. In this experiment, we directly used preprocessed sentences, segmenting words into subwords through byte-pair encoding and 10,000 merge operations.

[0188] Visual Features: We first employ a Stanford parser to identify noun phrases from each source sentence, then use a visual landing toolkit to detect the associated visual objects of the identified noun phrases. For each phrase, we maintain the highest predicted probability for its corresponding visual object to reduce the negative impact of abundant visual objects. The average number of objects and words in each sentence is approximately 3.5 and 15.0, respectively. Finally, we compute 2048-dimensional features for these objects using a pre-trained ResNet-100 Faster R-CNN.

[0189] Setup: We use a transformer as the base. Due to the small training corpus, the trained model tends to overfit; therefore, we first perform a small grid search to obtain hyperparameters on a set of English-to-German validation sets. Specifically, the word embedding dimension and hidden size are 128 and 256, respectively. The decoder has 4 layers and 4 attention heads. The dropout rate is set to 0.5. Each batch consists of approximately 2000 source and target tokens. We apply the Adam optimizer with a predetermined learning rate to optimize various models, using the same other settings. Finally, we use the Bilingual EvaLuration Understudy (BLEU) metric and the METEOR metric to evaluate the quality of the translations. It should be noted that we run all models three times in each experiment and report the average results.

[0190] Base Model: In addition to the text-based Transformer (TF), we also leveraged our visual features and employed several effective methods for transformation, comparing our model with the Transformer:

[0191] 1. ObjectAsToken(TF). It is a variant of Transformer where all visual objects are treated as additional source code symbols and placed before the input statement.

[0192] 2. Enc-att (TF). The transformer employs an encoder-based image attention mechanism, which adds attention-based visual feature vectors to each source annotation.

[0193] 3. Double-att (TF). It is a dual-attention transformer. In each decoding layer, a cross-modal multi-head attention sublayer is inserted before the fully connected feedforward layer to generate visual context vectors based on visual features.

[0194] We also demonstrate the performance of several major multimodal neural machine translation (NMT) models, such as Doubly-att (RNN), Soft-att (RNN), Stochastlc-att (RNN), Fusion-conv (RNN), Trg-mul (RNN), VMM T (RNN), and Dellberation Network (TF). Here, RNN stands for Recurrent Neural Network.

[0195] The number of multimodal fusion layers, *e*, is an important hyperparameter that directly determines the degree of fine-grained semantic fusion in our encoder. Therefore, we first examine its impact on the English-to-German validation set. Figure 8 The experimental results show that our model achieves the optimal p-shape when e is 3. Therefore, we use e = 3 in all subsequent experiments.

[0196] Table 1

[0197]

[0198] Table 1 shows the main results of the English-to-German translation task. Compared with Fusion-conv (RNN) and Trg-mul (RNN) on METEOR, our model outperforms most previous models. Both sets of results depend on the system state on the WMT2017 test set, which is selected based on METEOR. The following conclusions can be drawn from comparisons with the base model.

[0199] First, our model outperforms ObjectAsToken(TF) by concatenating region visual features with text to form attentionable sequences and utilizing a self-attention mechanism for multimodal fusion. This is primarily due to two factors: first, modeling the semantic correspondences between semantic units of different modalities; and second, differentiating the model parameters for different modalities.

[0200] Secondly, our model also significantly outperforms Enc-att(TF). Enc-att(TF) can be considered a single-layer semantic fusion encoder. In addition to the advantages of semantic correspondence modeling, we also hypothesize that multi-layer, multimodal semantic interaction is beneficial for NMT.

[0201] Third, compared with Doubly-att(TF), which simply uses the attention mechanism to extract visual information, our model achieves significant improvements due to the full multimodal fusion provided in the encoder.

[0202] Table 2

[0203]

[0204] In addition, we divided our test set into different groups based on the length of the source sentence and the number of noun phrases, and then compared the performance of different models in each test set. Figure 9 and Figure 10 The BLEU scores for the above groupings are shown. Overall, our model consistently achieves state-of-the-art performance across all groups. Therefore, we reaffirm the effectiveness and generality of our proposed model. It is worth noting that our model's improvements over the base model are more meaningful in sentences with many phrases, typically long sentences. We hypothesize that long sentences often contain more ambiguous words. Therefore, compared to short sentences, long sentences may require better utilization of visual information as supplementary information, which can be achieved through the multimodal semantic interaction of our model.

[0205] Table 4 also shows the training and decoding speeds of our model and the base model. During training, our model can process approximately 1.1K tokens per second, which is comparable to other multimodal models. When it comes to decoding, our model translates approximately 16.7 sentences per second, a slight decrease in speed compared to Transformer. Furthermore, our model introduces only a small number of additional parameters and achieves better performance.

[0206] Table 3

[0207]

[0208] To investigate the effectiveness of different components, we conducted further experiments, comparing our model with the following variants in Table 2:

[0209] (1) Intermodal fusion. In this variant, we use two independent transformer encoders to learn semantic representations of words and visual objects respectively, and then use a dual-attention decoder to merge the textual and visual context into the decoder. The results in row 3 of Table 2 show that removing intermodal fusion leads to a significant performance degradation. This indicates that semantic interactions between multimodal semantic units are indeed useful for multimodal representation learning.

[0210] (2) Visual to Fully Connected. We fully connect words and visual objects to establish a correspondence between modalities. The results in row 4 of Table 2 show that this change leads to a significant performance degradation. The fundamental reason is that the semantic correspondence of the fully connected model introduces a lot of noise into our model.

[0211] (3) From different parameters to uniform parameters. When constructing this variant, we assign uniform parameters to update the node state in different modes. Clearly, the performance degradation reported in row 5 of Table 2 also demonstrates the effectiveness of our method using different parameters.

[0212] (4) Attention to Visual Nodes. Unlike our model that only considers text nodes, we allow this variant's decoder to use a dual-attention decoder to consider both types of nodes. From the results in row 6 of Table 2, we can observe that considering all nodes does not bring further improvement. The above results confirm our previous hypothesis that visual information has been fully incorporated into the text nodes in our encoder.

[0213] (5) Pay attention to text nodes versus visual nodes. However, when only visual nodes are considered, the model performance drops sharply, as shown in row 7 of Table 2. This is because the number of visual nodes is far less than that of text nodes, and text nodes cannot generate sufficient translation context.

[0214] For example, we also conducted experiments on the English-to-French translation dataset. As shown in Table 3, our model still achieves better performance compared to all previous models, which once again demonstrates that our model is effective and general for different language pairs in multimodal NMT.

[0215] In Table 2, the machine translation model provided in this application is compared with the machine translation model provided in the relevant multimodal NMT system and our own multimodal NMT system. It can be clearly seen from the BLEU and METEOR indicators that the machine translation model provided in this application has also achieved good results for English-French translation. Three of the four indicator values ​​are the highest values ​​(bold numbers).

[0216] Table 4

[0217] Transformer 2.6K 17.8 3.4K ObjectAsToken(TF) 1.6K 17.2 3.7K Enc-att(TF) 1.3K 16.9 3.6K Doubly-att (TF) 1.0K 12.9 3.8K Our model 1.1K 16.7 4.0K

[0218] Please refer to Figure 11 This application illustrates a translation apparatus based on multimodal machine learning, provided in an exemplary embodiment. The apparatus is part or all of a computer device, comprised of software, hardware, or a combination of both.

[0219] The semantic association module 501 is used to perform semantic association on source languages ​​of n modalities through a multimodal graph representation layer, and construct a semantic association graph. The semantic association graph includes semantic nodes of n different modalities, a first connecting edge for connecting semantic nodes of the same modality, and a second connecting edge for connecting semantic nodes of different modalities, where n is a positive integer greater than 1.

[0220] Feature extraction module 502 is used to extract the first word vector from the semantic association graph through the first word vector layer;

[0221] The vector encoding module 503 is used to encode the first word vector through a multimodal fusion encoder to obtain the encoded feature vector;

[0222] The vector decoding module 504 is used to call the decoder to decode the encoded feature vector and obtain the translated target sentence.

[0223] In some optional embodiments, the semantic association module 501 is used to extract semantic nodes from the source language of each modality through the multimodal graph representation layer to obtain n sets of semantic nodes corresponding to the source languages ​​of n modalities; the multimodal graph representation layer uses a first connecting edge to connect the semantic nodes within the same modality to the n sets of semantic nodes, and uses a second connecting edge to connect the semantic nodes between different modalities to the n sets of semantic nodes, thereby obtaining a semantic association graph.

[0224] In some alternative embodiments, the source languages ​​of the n modalities include a first source language in text form and a second source language in non-text form, and the n sets of semantic nodes include a first semantic node and a second semantic node;

[0225] The semantic association module 501 is used to extract a first semantic node from a first source statement and extract candidate semantic nodes from a second source language through a multimodal graph representation layer; call the multimodal graph representation layer to calculate a first probability distribution of candidate semantic nodes according to the semantic association between the first semantic node and the candidate semantic nodes; and call the multimodal graph representation layer to determine a second semantic node from the candidate semantic nodes according to the first probability distribution.

[0226] In some optional embodiments, the semantic association module 501 is used to determine the i-th type of first connection edge corresponding to the i-th modality through the multimodal graph representation layer, and use the i-th type of first connection edge to connect the semantic nodes in the i-th group of semantic nodes within the same modality, where i is a positive integer less than or equal to n.

[0227] In some alternative embodiments, the multimodal fusion encoder includes e cascaded encoding modules, where e is a positive integer;

[0228] The vector encoding module 503 is used to perform e intra-modal fusion and inter-modal fusion on the first word vector through e concatenated encoding modules to obtain the encoded feature vector. The intra-modal fusion refers to semantic fusion between the first word vectors in the same modality, and the inter-modal fusion refers to semantic fusion between the first word vectors in different modalities.

[0229] In some optional embodiments, each coding module includes n intramodal fusion layers and n intermodal fusion layers corresponding one-to-one with the n modalities;

[0230] The vector encoding module 503 is used to input the first word vector into the n intramodal fusion layers in the first encoding module respectively, and to perform semantic fusion within the same modality on the first word vector through the n intramodal fusion layers respectively, so as to obtain n first hidden layer vectors that correspond one-to-one with the n modalities.

[0231] The n first hidden layer vectors are input into each intermodal fusion layer in the first coding module. The n first hidden layer vectors are semantically fused between different modalities through each intermodal fusion layer to obtain n first intermediate vectors that correspond one-to-one with the n modalities.

[0232] The n first intermediate vectors are input into the j-th encoding module for the j-th encoding process, until the last encoding module outputs n encoded feature vectors that correspond one-to-one with the n modes, where j is a positive integer greater than 1 and less than or equal to e.

[0233] In some optional embodiments, each encoding module further includes: n first vector transformation layers corresponding one-to-one with the n modalities;

[0234] The vector encoding module 503 is also used to input the n first intermediate vectors into the n first vector transformation layers corresponding to their respective modes for nonlinear transformation, so as to obtain the n first intermediate vectors after nonlinear transformation.

[0235] In some alternative embodiments, the hierarchical structure is identical in each of the e cascaded coding modules.

[0236] In some alternative embodiments, different or the same self-attention functions are set in different intramodal fusion layers, and different or the same feature fusion functions are set in different intermodal fusion layers.

[0237] In some alternative embodiments, the decoder includes d cascaded decoding modules, where d is a positive integer;

[0238] The vector decoding module 504 is used to obtain the first target word through the second word vector layer. The first target word is the translated word in the target sentence. The second word vector layer is used to extract features from the first target word to obtain the second word vector.

[0239] The second word vector is combined with the encoded feature vector through d concatenated decoding modules to extract features, resulting in a decoded feature vector. The decoded feature vector is then input into a classifier, which calculates the probability distribution corresponding to the decoded feature vector and determines the second target word following the first target word based on the probability distribution.

[0240] In some alternative embodiments, each of the d cascaded decoding modules includes a first self-attention layer and a second self-attention layer;

[0241] The vector decoding module 504 is used to input the second word vector into the first self-attention layer in the first decoding module, and extract features from the second word vector through the first self-attention layer to obtain the second hidden layer vector;

[0242] The second hidden layer vector and the encoded feature vector are input into the second self-attention layer in the first decoding module. The second self-attention layer combines the second hidden layer vector and the encoded feature vector to extract features and obtain the second intermediate vector.

[0243] The second intermediate vector is input into the k-th decoding module for the k-th decoding process, until the last decoding module outputs the decoded feature vector, where k is a positive integer greater than 1 and less than or equal to d.

[0244] In some alternative embodiments, each decoding module further includes: a second vector conversion layer;

[0245] The vector decoding module 504 is also used to input the second intermediate vector into the second vector conversion layer for nonlinear conversion to obtain the nonlinearly converted second intermediate vector.

[0246] In summary, the translation device based on multimodal machine learning provided in this embodiment performs semantic association on n source languages ​​through a multimodal graph representation layer to construct a semantic association graph. In the semantic association graph, a first connecting edge connects semantic nodes of the same modality, and a second connecting edge connects semantic nodes of different modalities. The semantic association graph fully represents the semantic association between source languages ​​of multiple modalities. Then, a multimodal fusion encoder performs full semantic fusion on the feature vectors in the semantic association graph to obtain encoded feature vectors. After decoding the encoded feature vectors, a more accurate target sentence is obtained. This target sentence is closer to the content, emotion, and linguistic environment comprehensively expressed by the multimodal source languages.

[0247] Please refer to Figure 12This diagram illustrates the structure of a server according to an embodiment of this application. The server is used to implement the steps of the multimodal machine learning-based translation method provided in the above embodiments. Specifically:

[0248] The server 600 includes a CPU (Central Processing Unit) 601, a system memory 604 including RAM (Random Access Memory) 602 and ROM (Read-Only Memory) 603, and a system bus 605 connecting the system memory 604 and the central processing unit 601. The server 600 also includes a basic I / O (Input / Output) system 606 that facilitates information transfer between various devices within the computer, and a large-capacity storage device 607 for storing the operating system 613, application programs 614, and other program modules 615.

[0249] The basic input / output system 606 includes a display 608 for displaying information and an input device 609 for user input, such as a mouse or keyboard. Both the display 608 and the input device 609 are connected to the central processing unit 601 via an input / output controller 610 connected to the system bus 605. The basic input / output system 606 may also include the input / output controller 610 for receiving and processing input from multiple other devices such as a keyboard, mouse, or electronic stylus. Similarly, the input / output controller 610 also provides output to a display screen, printer, or other types of output devices.

[0250] The mass storage device 607 is connected to the central processing unit 601 via a mass storage controller (not shown) connected to the system bus 605. The mass storage device 607 and its associated computer-readable media provide non-volatile storage for the server 600. That is, the mass storage device 607 may include computer-readable media (not shown) such as a hard disk or a CD-ROM (Compact Disc Read-Only Memory) drive.

[0251] Without loss of generality, the computer-readable medium may include computer storage media and communication media. Computer storage media include volatile and non-volatile, removable and non-removable media implemented using any method or technology for storing information such as computer-readable instructions, data structures, program modules, or other data. Computer storage media include RAM, ROM, EPROM (Erasable Programmable Read-Only Memory), EEPROM (Electrically Erasable Programmable Read-Only Memory), flash memory or other solid-state storage technologies, CD-ROM, DVD (Digital Versatile Disc) or other optical storage, magnetic tape cassettes, magnetic tape, disk storage, or other magnetic storage devices. Of course, those skilled in the art will recognize that the computer storage media are not limited to the above-mentioned types. The system memory 604 and mass storage device 607 described above can be collectively referred to as memory.

[0252] According to various embodiments of this application, the server 600 can also be connected to a remote computer on a network, such as the Internet. That is, the server 600 can be connected to a network 612 via a network interface unit 611 connected to the system bus 605, or the network interface unit 611 can be used to connect to other types of networks or remote computer systems (not shown).

[0253] The sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0254] Those skilled in the art will understand that all or part of the steps of the above embodiments can be implemented by hardware or by a program instructing related hardware. The program can be stored in a computer-readable storage medium, such as a read-only memory, a disk, or an optical disk.

[0255] The above description is merely an optional embodiment of this application and is not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. A method for translation based on multi-modal machine learning, characterized in that, The method, applied in a computer device equipped with a multimodal fusion encoder and decoder, includes: A semantic association graph is constructed by semantically associating source languages ​​of n modalities through a multimodal graph representation layer. In the semantic association graph, a first connecting edge is used to connect semantic nodes of the same modality, and a second connecting edge is used to connect semantic nodes of different modalities, so as to identify the semantic association between the source languages ​​of the n modalities, where n is a positive integer greater than 1. The first word vector is extracted from the semantic association graph through the first word vector layer; The first word vector is subjected to intramodal feature extraction by a multimodal fusion encoder, and then intermodal feature fusion is performed on the vector obtained by feature extraction to obtain an encoded feature vector. The decoder is invoked to decode the encoded feature vector to obtain the translated target sentence.

2. The method according to claim 1, characterized in that, The step of semantically associating source languages ​​of n modalities through a multimodal graph representation layer to construct a semantic association graph includes: The multimodal graph representation layer extracts semantic nodes from the source language of each modality to obtain n sets of semantic nodes corresponding to the source languages ​​of the n modalities. The semantic association graph is obtained by using the first connecting edge to connect the semantic nodes of the n groups of semantic nodes within the same modality, and using the second connecting edge to connect the semantic nodes of the n groups of semantic nodes across different modalities through the multimodal graph representation layer.

3. The method according to claim 2, characterized in that, The source languages ​​of the n modalities include a first source language in text form and a second source language in non-text form, and the n sets of semantic nodes include a first semantic node and a second semantic node; The step of extracting semantic nodes from the source language of each modality through the multimodal graph representation layer to obtain n sets of semantic nodes corresponding to the source languages ​​of the n modalities includes: The first semantic node is extracted from the first source statement through the multimodal graph representation layer, and candidate semantic nodes are extracted from the second source language. The multimodal graph representation layer is invoked to calculate the first probability distribution of the candidate semantic nodes according to the semantic association between the first semantic node and the candidate semantic nodes; The multimodal graph representation layer is invoked to determine the second semantic node from the candidate semantic nodes based on the first probability distribution.

4. The method according to claim 2, characterized in that, The step of connecting the n groups of semantic nodes within the same modality using the first connecting edge through the multimodal graph representation layer includes: The i-th type of first connection edge corresponding to the i-th modality is determined by the multimodal graph representation layer, and the i-th type of first connection edge is used to connect semantic nodes in the same modality of the i-th group of semantic nodes, where i is a positive integer less than or equal to n.

5. The method according to any one of claims 1 to 4, characterized in that, The multimodal fusion encoder comprises e cascaded encoding modules, where e is a positive integer; The step of extracting intra-modal features from the first word vector using a multimodal fusion encoder, and then fusing inter-model features from the extracted vectors to obtain an encoded feature vector, includes: The first word vector is fused e times by the e concatenated encoding modules to obtain the encoded feature vector. The intra-modal fusion refers to semantic fusion between the first word vectors in the same modality, and the inter-modal fusion refers to semantic fusion between the first word vectors in different modalities.

6. The method according to claim 5, characterized in that, Each encoding module includes n intramodal fusion layers and n intermodal fusion layers, each corresponding to one of the n modalities; The step of performing e intra-modal and inter-modal semantic fusions on the first word vector through the e concatenated encoding modules to obtain the encoded feature vector includes: The first word vector is input into the n intra-modal fusion layers in the first encoding module, and the first word vector is semantically fused within the same modality by the n intra-modal fusion layers to obtain n first hidden layer vectors that correspond one-to-one with the n modalities. The n first hidden layer vectors are input into each intermodal fusion layer in the first encoding module. The intermodal fusion layer performs semantic fusion between different modalities on the n first hidden layer vectors to obtain n first intermediate vectors that correspond one-to-one with the n modalities. The n first intermediate vectors are input into the j-th encoding module for the j-th encoding process until the last encoding module outputs n encoded feature vectors that correspond one-to-one with the n modes, where j is a positive integer greater than 1 and less than or equal to e.

7. The method according to claim 6, characterized in that, Each encoding module further includes: n first vector transformation layers corresponding one-to-one with the n modes; The method further includes: The n first intermediate vectors are respectively input into the n first vector transformation layer corresponding to their respective modes for nonlinear transformation to obtain the n first intermediate vectors after nonlinear transformation.

8. The method according to claim 6, characterized in that, The hierarchical structure is the same in each of the e concatenated encoding modules.

9. The method according to claim 6, characterized in that, Different intramodal fusion layers have different or the same self-attention functions, and different intermodal fusion layers have different or the same feature fusion functions.

10. The method according to any one of claims 1 to 4, characterized in that, The decoder comprises d cascaded decoding modules, where d is a positive integer; The step of calling the decoder to decode the encoded feature vector to obtain the translated target sentence includes: The first target word is obtained through the second word vector layer, and the first target word is the translated word in the target sentence; The second word vector is obtained by extracting features from the first target word through the second word vector layer. The second word vector is combined with the encoded feature vector through the d concatenated decoding modules to extract features, thereby obtaining the decoded feature vector; The decoded feature vector is input into a classifier, the classifier calculates the probability distribution corresponding to the decoded feature vector, and the second target word following the first target word is determined based on the probability distribution.

11. The method according to claim 10, characterized in that, Each of the d cascaded decoding modules includes a first self-attention layer and a second self-attention layer. The step of extracting features by combining the second word vector with the encoded feature vector through the d concatenated decoding modules to obtain the decoded feature vector includes: The second word vector is input into the first self-attention layer in the first decoding module, and the second word vector is subjected to feature extraction through the first self-attention layer to obtain the second hidden layer vector; The second hidden layer vector and the encoded feature vector are input into the second self-attention layer in the first decoding module. Feature extraction is performed by combining the second hidden layer vector and the encoded feature vector through the second self-attention layer to obtain the second intermediate vector. The second intermediate vector is input into the k-th decoding module for the k-th decoding process until the last decoding module outputs the decoded feature vector, where k is a positive integer greater than 1 and less than or equal to d.

12. The method according to claim 11, characterized in that, Each decoding module further includes: a second vector conversion layer; The method further includes: The second intermediate vector is input into the second vector transformation layer for nonlinear transformation to obtain the nonlinearly transformed second intermediate vector.

13. A translation device based on multimodal machine learning, characterized in that, The device is equipped with a multimodal fusion encoder and a decoder, and the device includes: The semantic association module is used to perform semantic association on the source languages ​​of n modalities through the multimodal graph representation layer, and construct a semantic association graph. In the semantic association graph, a first connecting edge is used to connect semantic nodes of the same modality, and a second connecting edge is used to connect semantic nodes of different modalities, so as to identify the semantic association between the source languages ​​of the n modalities, where n is a positive integer greater than 1. The feature extraction module is used to extract the first word vector from the semantic association graph through the first word vector layer; The vector encoding module is used to extract intra-modal features from the first word vector through a multimodal fusion encoder, and then perform inter-modal feature fusion on the vector obtained by feature extraction to obtain an encoded feature vector; The vector decoding module is used to call the decoder to decode the encoded feature vector to obtain the translated target sentence.

14. The apparatus according to claim 13, characterized in that, The device includes: The semantic association module is used to extract semantic nodes from the source language of each modality through the multimodal graph representation layer to obtain n sets of semantic nodes corresponding to the source languages ​​of the n modalities. The semantic association module is used to connect the n groups of semantic nodes within the same modality using the first connecting edge and to connect the n groups of semantic nodes between different modalities using the second connecting edge, thereby obtaining the semantic association graph.

15. The apparatus according to claim 14, characterized in that, The source languages ​​of the n modalities include a first source language in text form and a second source language in non-text form, and the n sets of semantic nodes include a first semantic node and a second semantic node; The device further includes: The semantic association module is used to extract the first semantic node from the first source statement through the multimodal graph representation layer, and to extract candidate semantic nodes from the second source language. The semantic association module is used to call the multimodal graph representation layer to calculate the first probability distribution of the candidate semantic nodes according to the semantic association between the first semantic node and the candidate semantic nodes; The semantic association module is used to call the multimodal graph representation layer to determine the second semantic node from the candidate semantic nodes according to the first probability distribution.

16. The apparatus according to claim 14, characterized in that, The device further includes: The semantic association module is used to determine the i-th type of first connection edge corresponding to the i-th modality through the multimodal graph representation layer, and use the i-th type of first connection edge to connect semantic nodes in the same modality of the i-th group of semantic nodes, where i is a positive integer less than or equal to n.

17. The apparatus according to any one of claims 13 to 16, characterized in that, The multimodal fusion encoder comprises e cascaded encoding modules, where e is a positive integer; The device further includes: The vector encoding module is used to perform e intra-modal fusion and inter-modal fusion on the first word vector through the e concatenated encoding modules to obtain the encoded feature vector. The intra-modal fusion refers to semantic fusion between the first word vectors in the same modality, and the inter-modal fusion refers to semantic fusion between the first word vectors in different modalities.

18. The apparatus according to claim 17, characterized in that, Each encoding module includes n intramodal fusion layers and n intermodal fusion layers, each corresponding to one of the n modalities; The device further includes: The vector encoding module is used to input the first word vector into the n intra-modal fusion layers in the first encoding module respectively, and to perform semantic fusion within the same modality on the first word vector through the n intra-modal fusion layers respectively, so as to obtain n first hidden layer vectors that correspond one-to-one with the n modalities. The vector encoding module is used to input the n first hidden layer vectors into each intermodal fusion layer in the first encoding module, and to perform semantic fusion between different modalities on the n first hidden layer vectors through each intermodal fusion layer to obtain n first intermediate vectors that correspond one-to-one with the n modalities. The vector encoding module is used to input the n first intermediate vectors into the j-th encoding module for the j-th encoding process, until the last encoding module outputs n encoded feature vectors that correspond one-to-one with the n modes, where j is a positive integer greater than 1 and less than or equal to e.

19. The apparatus according to claim 18, characterized in that, Each encoding module further includes: n first vector transformation layers corresponding one-to-one with the n modes; The device further includes: The vector encoding module is used to input the n first intermediate vectors into the n first vector transformation layers corresponding to their respective modes for nonlinear transformation, so as to obtain the n first intermediate vectors after nonlinear transformation.

20. The apparatus according to claim 18, characterized in that, The hierarchical structure is the same in each of the e concatenated encoding modules.

21. The apparatus according to claim 18, characterized in that, Different intramodal fusion layers have different or the same self-attention functions, and different intermodal fusion layers have different or the same feature fusion functions.

22. The method according to any one of claims 13 to 16, characterized in that, The decoder in the device comprises d cascaded decoding modules, where d is a positive integer; The device further includes: The vector decoding module is used to obtain a first target word through a second word vector layer, wherein the first target word is a translated word in the target sentence; The vector decoding module is used to extract features from the first target word through the second word vector layer to obtain the second word vector; The vector decoding module is used to extract features by combining the second word vector with the encoded feature vector through the d concatenated decoding modules to obtain the decoded feature vector; The vector decoding module is used to input the decoded feature vector into a classifier, calculate the probability distribution corresponding to the decoded feature vector through the classifier, and determine the second target word after the first target word based on the probability distribution.

23. The apparatus according to claim 22, characterized in that, Each of the d cascaded decoding modules includes a first self-attention layer and a second self-attention layer. The device further includes: The vector decoding device is used to input the second word vector into the first self-attention layer in the first decoding module, and extract features from the second word vector through the first self-attention layer to obtain the second hidden layer vector; The vector decoding device is used to input the second hidden layer vector and the encoded feature vector into the second self-attention layer in the first decoding module, and to perform feature extraction by combining the second hidden layer vector and the encoded feature vector through the second self-attention layer to obtain the second intermediate vector; The vector decoding device is used to input the second intermediate vector into the k-th decoding module for the k-th decoding process until the last decoding module outputs the decoded feature vector, where k is a positive integer greater than 1 and less than or equal to d.

24. The apparatus according to claim 23, characterized in that, Each decoding module further includes: a second vector conversion layer; The device further includes: The vector decoding module is used to input the second intermediate vector into the second vector conversion layer for nonlinear conversion to obtain the nonlinearly converted second intermediate vector.

25. A computer device, characterized in that, The computer device includes: Memory; A processor connected to the memory; The processor is configured to load and execute executable instructions to implement the translation method based on multimodal machine learning as described in any one of claims 1 to 12.

26. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores at least one instruction, at least one program, code set, or instruction set; the at least one instruction, the at least one program, the code set, or the instruction set is loaded and executed by a processor to implement the translation method based on multimodal machine learning as described in any one of claims 1 to 12.

Images