Multimodal machine learning-based translation method, device, equipment, and computer program

The multimodal machine learning-based translation method constructs a semantic association diagram to connect semantic nodes of different modalities, using a fusion encoder to encode and decode feature vectors, thereby addressing the challenge of translating multimodal source languages and maintaining semantic and emotional context in the translation output.

JP7592095B2Active Publication Date: 2024-11-29TENCENT TECHNOLOGY (SHENZHEN) CO LTD
View PDF 4 Cites 0 Cited by

Patent Information

Application Number
JP2022540553
Authority / Receiving Office
JP · JP
Patent Type
Patents
Current Assignee / Owner
Priority Date
2020-05-20
Filing Date
2021-04-29
Publication Date
2024-11-29
Estimated Expiration
2041-04-29

AI Technical Summary

Technical Problem

Existing machine translation models struggle to effectively translate multimodal source languages, such as text and images, by failing to adequately fuse semantic features and maintain the content and emotional context of the source language in the translated output.

Method used

A multimodal machine learning-based translation method that constructs a semantic association diagram using a multimodal graph representation layer to connect semantic nodes of different modalities, followed by a multimodal fusion encoder to encode and decode feature vectors, ensuring accurate translation of semantic content and emotional context.

Benefits of technology

The method achieves a more accurate translation by fully expressing semantic associations and emotional context, resulting in a target statement that closely resembles the content and emotion conveyed by the multimodal source language.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 0007592095000058
    Figure 0007592095000058
  • Figure 0007592095000059
    Figure 0007592095000059
  • Figure 0007592095000060
    Figure 0007592095000060
Patent Text Reader

Abstract

This paper discloses a translation method based on multimodal machine learning, which relates to the technical field of artificial intelligence. The method performs semantic association on n source statements of different modalities to construct a semantic association diagram. In the semantic association diagram, first connection edges are used to connect semantic nodes of the same modality, and second connection edges are used to connect semantic nodes of different modalities, so that the semantic association diagram fully expresses the semantic associations between the source statements of different modalities. Subsequently, sufficient semantic fusion is performed on the feature vectors in the semantic association diagram to obtain an encoded feature vector after encoding. After decoding the encoded feature vector, a more accurate target statement is obtained. The target statement more closely reflects the overall content, sentiment, and linguistic environment of the multimodal source statements.
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] The present application relates to the technical field of artificial intelligence, and in particular to a translation method, device, apparatus, and storage medium based on multimodal machine learning.

[0002] This application claims priority to a Chinese patent application filed on May 20, 2020, bearing application number 2020104325972 and entitled "Translation method, device, apparatus and storage medium based on multimodal machine learning," the entire contents of which are incorporated herein by reference. [Background technology]

[0003] Machine translation is the process of using a computer to convert one natural language into another.

[0004] In some application scenarios, the machine translation model can translate a source language with multiple different expression formats into a target language, that is, translate a multimodal source language into a target language. For example, a picture and a corresponding English annotation are obtained, and the machine translation model performs feature extraction on the picture and the English annotation respectively, and then fuses the extracted features, and then translates according to the fused features to obtain a French annotation corresponding to the picture and the English annotation. Summary of the Invention [Problem to be solved by the invention]

[0005] The embodiments of the present application provide a translation method, device, apparatus and storage medium based on multi-modal machine learning, which can perform sufficient semantic fusion for multi-modal source languages ​​in the process of feature encoding, and make the target statement decoded by the encoding vector closer to the content, sentiment, etc. expressed by the source language. The technical means are as follows: [Means for solving the problem]

[0006] According to one aspect of the present application, there is provided a multimodal machine learning based translation method executed by a computing device, the method comprising: constructing a semantic relationship diagram based on n source statements belonging to different modalities, the semantic relationship diagram comprising: semantic nodes of n different modalities, a first connection edge used for connecting semantic nodes of the same modality, and a second connection edge used for connecting semantic nodes of different modalities, the semantic nodes are used for indicating one semantic unit of the source statements in one modality, n being a positive integer greater than 1; extracting a first plurality of word vectors from the semantic relationship diagram; encoding the first word vectors to obtain n encoded feature vectors; and decoding the n encoded feature vectors to obtain a translated target statement.

[0007] According to another aspect of the present application, there is provided a multimodal machine learning based translation device, the device comprising: A semantic association module for constructing a semantic relationship diagram based on n source statements belonging to different modalities, the semantic relationship diagram comprising: semantic nodes of n different modalities, a first connection edge for connecting the semantic nodes of the same modality, and a second connection edge for connecting the semantic nodes of the different modalities, the semantic nodes for indicating one semantic unit of the source statements in one modality, n being a positive integer greater than 1; a feature extraction module adapted to extract a first plurality of word vectors from the semantic relationship map; a vector encoding module for encoding the first word vectors to obtain n encoded feature vectors; a vector decoding module, which is used for decoding the n encoded feature vectors to obtain a translated target statement.

[0008] According to another aspect of the present application, there is provided a computing device, the computing device comprising: Memory, a processor coupled to the memory, The processor is configured to load and execute executable instructions to implement the multimodal machine learning based translation method according to the above aspect and alternative embodiments thereof.

[0009] According to another aspect of the present application, there is provided a computer-readable storage medium, storing at least one instruction, at least one segment of a program, code set or instruction set, the at least one instruction, at least one segment of a program, code set or instruction set being loaded and executed by a processor to realize the multimodal machine learning based translation method according to the above aspect and optional embodiments thereof.

[0010] In order to more clearly describe the technical means in the embodiments of the present application, the following briefly introduces the drawings that need to be used in the description of the embodiments, but it is obvious that the drawings described below are merely some embodiments of the present application, and those skilled in the art can further obtain other drawings based on these drawings without creative efforts. [Brief description of the drawings]

[0011] [Figure 1] FIG. 1 is a structural schematic diagram of a multi-modal machine translation model provided by one exemplary embodiment of the present application. [Diagram 2] FIG. 2 is a structural schematic diagram of a computer system provided by one exemplary embodiment of the present application. [Diagram 3] FIG. 3 is a flowchart of a multimodal machine learning based translation method provided by one exemplary embodiment of the present application. [Figure 4] FIG. 4 is a flow chart for constructing a semantic relationship diagram provided by one exemplary embodiment of the present application. [Diagram 5] FIG. 5 is a flowchart of a translation method based on multimodal machine learning provided by another exemplary embodiment of the present application. [Figure 6] FIG. 6 is a flowchart of a translation method based on multimodal machine learning provided by another exemplary embodiment of the present application. [Figure 7] FIG. 7 is a structural schematic diagram of a multi-modal machine translation model provided by another exemplary embodiment of the present application. [Figure 8] FIG. 8 is a curve diagram of a model test result provided by one exemplary embodiment of the present application. [Figure 9] FIG. 9 is a curve diagram of a model test result provided by another exemplary embodiment of the present application. [Figure 10] FIG. 10 is a curve diagram of a model test result provided by another exemplary embodiment of the present application. [Figure 11] FIG. 11 is a block diagram of a translation device based on multimodal machine learning provided by one exemplary embodiment of the present application. [Figure 12] FIG. 12 is a structural schematic diagram of a server provided by an exemplary embodiment of the present application. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0012] In order to make the objectives, technical means and advantages of the present application clearer, the embodiments of the present application are described in more detail below with reference to the drawings.

[0013] The nouns related to this application shall be interpreted as follows:

[0014] Artificial Intelligence (AI): A technical science of theories, methods, techniques and application systems that use digital computers or machines controlled by digital computers to imitate, expand and extend human intelligence, sense the environment, or obtain knowledge and use the knowledge to obtain optimal results. In other words, artificial intelligence is a comprehensive technology of computer science that is intended to understand the nature of intelligence and produce new intelligent machines that can react in a manner similar to human intelligence. Artificial intelligence is the study of the design principles and realization methods of various intelligent machines, giving them the functions of sensing, reasoning and decision-making.

[0015] Artificial intelligence technology is a comprehensive discipline with a wide range of related fields, including hardware-level technology and software-level technology. The basic technologies of artificial intelligence generally include technologies such as sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operating / interactive systems, and mechatronics. Artificial intelligence software technology mainly includes several major directions such as computer vision technology, voice processing technology, natural language processing technology, and machine learning / deep learning.

[0016] Here, natural language processing (NLP) is an important direction in the fields of computer science and artificial intelligence. It studies various theories and methods that can realize effective communication between humans and computers through natural language. Natural language processing is a science that integrates linguistics, computer science, and mathematics. Therefore, research in this field is related to natural language, that is, the language that people use in daily life. Therefore, it is closely related to the study of linguistics. Natural language processing technology generally includes technologies such as text processing, semantic understanding, machine translation, robot question and answering, and knowledge graph.

[0017] Machine learning (ML) is an interdisciplinary field that includes several disciplines, such as probability theory, statistics, approximation theory, convex analysis, and algorithmic complexity theory. It specializes in how computers imitate or realize human learning behavior to acquire new knowledge or skills and reorganize existing knowledge structures to continually improve their own performance. Machine learning is the core of artificial intelligence and is the fundamental method that gives computers intelligence, and its applications span various fields of artificial intelligence. Machine learning and deep learning generally include techniques such as artificial neural networks, trust networks, reinforcement learning, transfer learning, inductive learning, and analogical learning.

[0018] The present application provides a multimodal machine translation model, which can accurately translate source statements of n different modalities into target statements. Here, modality refers to the expression form of language, for example, the statements may be expressed in graphical or textual forms. The source statement refers to a statement to be translated, which includes a sentence to be translated in a first language class in text form and a language to be translated in a non-text form. The target statement refers to a translated sentence in a second language class in text form, which is different from the first language class. For example, the source statement includes an English statement and an illustration of the English statement, and the multimodal machine translation model can translate the Chinese statement corresponding to the English statement and the illustration.

[0019] FIG. 1 shows a structural diagram of a multimodal machine translation model 100 provided by an exemplary embodiment of the present application. The multimodal machine translation model 100 includes a multimodal graph representation layer 101, a first word vector layer 102, a multimodal fusion encoder 103 and a decoder 104. The multimodal graph representation layer 101 is used for performing semantic association for n modal source languages ​​to obtain a semantic association diagram. The semantic association diagram includes n different modal semantic nodes, a first connection edge used for connecting the same modal semantic nodes, and a second connection edge used for connecting the different modal semantic nodes, where n is a positive integer greater than 1. Here, one semantic node is used to represent one semantic unit of a source statement in one modal. In English, for example, one semantic node corresponds to one word, and in Chinese, for example, one semantic node corresponds to one Chinese character.

[0020] The first word vector layer 102 is used for extracting a plurality of first word vectors from the semantic relationship diagram; The multi-modal fusion encoder 103 is used for encoding the plurality of first word vectors to obtain n encoded feature vectors; The decoder 104 is used to decode the n encoded feature vectors to obtain the translated target statement.

[0021] In some alternative embodiments, the multimodal graph representation layer 101 is used for obtaining n sets of semantic nodes, where one set of semantic nodes corresponds to a source statement of one modal, and adding the first connection edge between any two of the semantic nodes of the same modality and adding the second connection edge between any two of the semantic nodes of different modality to obtain the semantic relationship diagram.

[0022] In some alternative embodiments, the multimodal graph representation layer 101 is used to extract semantic nodes from each modal source language to obtain n sets of semantic nodes corresponding to the n modal source languages; The multimodal graph representation layer 101 is used to connect semantic nodes in the same modality for the n sets of semantic nodes using the first connection edges, and connect semantic nodes between different modalities for the n sets of semantic nodes using the second connection edges, thereby obtaining a semantic relationship diagram.

[0023] In some alternative embodiments, the n modal source statements include a first source statement in a textual form and a second source statement in a non-textual form, and the n set of semantic nodes includes a first semantic node and a second semantic node; The multimodal graph representation layer 101 is used for obtaining the first semantic node, the first semantic node being obtained by processing the first source statement; obtaining a candidate semantic node, the candidate semantic node being obtained by processing the second source statement; obtaining a first probability distribution of the candidate semantic node, the first probability distribution being calculated according to a semantic association between the first semantic node and the candidate semantic node; and determining the second semantic node from the candidate semantic node, the second semantic node being determined by the multimodal graph representation layer based on the first probability distribution.

[0024] In some alternative embodiments, the multimodal graph representation layer 101 is used for extracting first semantic nodes from the first source statements and extracting candidate semantic nodes from the second source statements, calculating a first probability distribution for the candidate semantic nodes according to semantic associations between the first semantic nodes and the candidate semantic nodes, and determining a second semantic node from the candidate semantic nodes based on the first probability distribution.

[0025] In some alternative embodiments, the multimodal graph representation layer 101 is used to add an i-th type of first connection edge between any two semantic nodes of the same modality in the i-th set of semantic nodes, where the i-th type of first connection edge corresponds to the i-th modality, and i is a positive integer less than or equal to n.

[0026] That is, the multimodal graph representation layer 101 is used to determine the i-th type of first connection edge corresponding to the i-th modality, and to perform connection between semantic nodes in the same modality for the i-th set of semantic nodes using the i-th type of first connection edge, where i is a positive integer less than or equal to n.

[0027] In some alternative embodiments, the n encoded feature vectors are obtained by performing intra-modal fusion and inter-modal fusion e times on the first word vectors to obtain the encoded feature vectors, where the intra-modal fusion refers to performing semantic fusion between the first word vectors in the same modal, and the inter-modal fusion refers to performing semantic fusion between the first word vectors in different modalities, where e is a positive integer.

[0028] In some alternative embodiments, the multi-modal fusion encoder 103 includes e encoding modules 1031 connected in series, each encoding module 1031 including n intra-modal fusion layers 11 and n inter-modal fusion layers 12, each of which corresponds one-to-one to the n modalities, where e is a positive integer; The first encoding module 1031 inputs the first word vector into the n intra-modal fusion layers 11 in the first encoding module respectively, and the n intra-modal fusion layers 11 respectively perform the same modal intra-semantic fusion on the first word vector to obtain n first hidden layer vectors, where one of the first hidden layer vectors corresponds to one modal, that is, obtain n first hidden layer vectors that correspond one-to-one to the n modals; The first encoding module 1031 inputs the n first hidden layer vectors to each inter-modal fusion layer 12 in the first encoding module, and each inter-modal fusion layer 12 performs semantic fusion between different modalities on the n first hidden layer vectors to obtain n first intermediate vectors, where one of the intermediate vectors corresponds to one modality, that is, obtains n first intermediate vectors that correspond one-to-one to the n modals; The jth encoding module 1031 is used to perform the jth encoding process on the n first intermediate vectors, and continues until the last encoding module outputs n encoded feature vectors, where one encoded feature vector corresponds to one modal, that is, until the last encoding module outputs n encoded feature vectors that correspond one-to-one to the n modals, where j is a positive integer greater than 1 and less than or equal to e.

[0029] In some alternative embodiments, each encoding module 1031 further includes n first vector transformation layers 13, each of which corresponds to one modality, i.e., n first vector transformation layers 13 corresponding one-to-one to the n modals; The encoding module 1031 is further used to input the n first intermediate vectors to n first vector transformation layers 13 corresponding to the modals to which they belong, and perform nonlinear transformation to obtain n first intermediate vectors after nonlinear transformation.

[0030] In some alternative embodiments, the hierarchical structure in each of the e encoding modules 1031 connected in series is the same.

[0031] In some alternative embodiments, different or the same self-attention functions are set for different intra-modal fusion layers, and different or the same feature fusion functions are set for different inter-modal fusion layers.

[0032] In some alternative embodiments, the multimodal machine translation model 100 further includes a second word vector layer 105 and a classifier 106, and the decoder 104 includes d decoding modules 1042 connected in series, where d is a positive integer; the second word vector layer 105 is used for obtaining a first target phrase, where the first target phrase is a translated phrase in the target statement, and performing feature extraction on the first target phrase to obtain a second word vector; The decoder 104 is used for combining the second word vector and the encoded feature vector by d decoding modules 1042 connected in series to perform feature extraction to obtain a decoded feature vector; The classifier 106 is used to determine a probability distribution corresponding to the decoded feature vector, and to determine a second target phrase subsequent to the first target phrase based on the probability distribution.

[0033] In some alternative embodiments, each of the d decoding modules 1042 connected in series includes a first self-attention layer 21 and a second self-attention layer 22; The first decoding module 1042 is used for inputting the second word vector into the first self-attention layer 21 in the first decoding module 1042, and performing feature extraction on the second word vector through the first self-attention layer 21 to obtain a second hidden layer vector; The first decoding module 1042 is used for inputting the second hidden layer vector and the encoded feature vector into the second self-attention layer 22 in the first decoding module 1042, and combining the second hidden layer vector and the encoded feature vector through the second self-attention layer 22 to perform feature extraction to obtain a second intermediate vector; The kth decoding module is used to input the second intermediate vector to the kth decoding module 1042 to perform the kth decoding process, and continues until the last decoding module outputs the decoded feature vector, where k is a positive integer greater than 1 and less than or equal to d.

[0034] In some alternative embodiments, each decoding module 1042 further includes a second vector transformation layer 23; The decoding module 1042 is used for inputting the second intermediate vector to the second vector transformation layer 23 for nonlinear transformation to obtain the second intermediate vector after nonlinear transformation.

[0035] As described above, the multimodal machine translation model provided in this embodiment performs semantic association for n modal source languages ​​through a multimodal graph representation layer to obtain a semantic association diagram. In the semantic association diagram, the first connection edge is used to connect semantic nodes of the same modality, and the second connection edge is used to connect semantic nodes of different modalities, so that the semantic association diagram fully represents the semantic association between the multiple modal source languages. Then, the multimodal fusion encoder performs sufficient semantic fusion on the feature vectors in the semantic association diagram to obtain an encoded feature vector after encoding. After the encoded feature vector is further decoded, a more accurate target statement is obtained. The target statement is closer to the content, emotion, and language environment, etc., comprehensively expressed by the multimodal source statement.

[0036] Referring to FIG. 2, a structural schematic diagram of a computer system provided by an exemplary embodiment of the present application is shown, which includes a terminal 220 and a server 240 .

[0037] An operating system is installed on the terminal 220, and application programs are installed on the operating system, and the application programs support a multi-modal source language translation function. Exemplarily, the application programs may include instant messaging software, financial software, game software, shopping software, video playing software, community service software, audio software, education software, payment software, translation software, etc., and the application programs are integrated with the multi-modal source language translation function.

[0038] The terminal 220 and the server 240 are coupled to each other via a wired or wireless network. The server 240 includes at least one of a server, a plurality of servers, a cloud computing platform, and a virtualization center. Exemplarily, the server 240 includes a processor and a memory, where a computer program is stored in the memory, and the processor can read and execute the computer program to realize a multi-modal source language translation function.

[0039] Alternatively, the server 240 performs the main computational tasks and the terminal 220 performs the secondary computational tasks, or the server 240 performs the secondary computational tasks and the terminal 220 performs the main computational tasks, or a distributed computing architecture is used to perform collaborative computing between both the server 240 and the terminal 220.

[0040] In some alternative embodiments, in the process of realizing the above multimodal language translation function, the server 240 provides background services for application programs in the terminal 220. Illustratively, the terminal 220 collects n modal source statements, sends the n modal source statements to the server 240, and executes the multimodal machine learning based translation method provided by the present application through the server 240, where n is a positive integer greater than 1.

[0041] For example, the terminal 220 includes a data transmission control member, and the terminal 220 uploads the two different modal source statements, i.e., the statement to be translated and the image matching the statement to be translated, to the server 240 through the data transmission control member. The server 240 executes the translation method based on multimodal machine learning provided in the present application to translate the two modal source statements into target statements.

[0042] In some alternative embodiments, the source statement may include a voice signal. If the n-modal source statement includes a voice signal, the terminal 220 or the server 240 first converts the voice signal into written text before translating the n-modal source statement. Illustratively, the terminal 220 collects the voice signal through a microphone, or the terminal 220 receives a voice signal transmitted from another terminal.

[0043] The above-mentioned translation method based on multimodal machine learning can be applied to a multimedia news translation scene. For example, the terminal 220 uploads multimedia news including text and images to the server 240, and the server 240 executes the translation method based on multimodal machine learning provided by the present application to translate the text of a first language in the multimedia news into the text of a second language.

[0044] The above-mentioned translation method based on multimodal machine learning can be applied to the translation scene of foreign language documents. For example, the terminal 220 uploads characters and illustrations corresponding to the characters in a foreign language document to the server 240, and the server 240 executes the translation method based on multimodal machine learning provided by the present application to translate the characters in a first language in the foreign language document into the characters in a second language.

[0045] The above-mentioned translation method based on multimodal machine learning can be applied to a foreign language website translation scene. For example, the terminal 220 collects characters and character illustrations in a foreign language website, uploads the characters and character illustrations to the server 240, and the server 240 executes the translation method based on multimodal machine learning provided by the present application to translate characters of a first language in a foreign language website into characters of a second language, and further realizes translation for the foreign language website.

[0046] In some alternative embodiments, the manner in which the terminal 220 displays the translated text includes in audio form or in text form.

[0047] As should be explained, in some alternative embodiments, the terminal 220 executes the multimodal machine learning based translation method provided herein to further translate n-modal source statements.

[0048] The terminal 220 may generally refer to one of a plurality of terminals, and the present embodiment will be described using only the terminal 220 as an example. The terminal 220 may include at least one of a smartphone, a tablet computer, an e-book reader, an MPEG Audio Layer 3 (MP3) player, an MPEG Audio Layer 4 (MP4) player, a laptop pocket computer, a desktop computer, and a notebook computer. The following embodiment will be described using the terminal 220 as an example including a smartphone and a personal computer device.

[0049] As can be understood by those skilled in the art, the number of the terminals 220 may be more or less. For example, the number of the terminals 220 may be only one, or the number of the terminals may be tens or hundreds, or more. The embodiment of the present application does not limit the number and device type of the terminals 220.

[0050] Referring to Figure 3, a flowchart of a translation method based on multimodal machine learning provided by one exemplary embodiment of the present application is shown. The method is applied to the computer device shown in Figure 2, the computer device includes a terminal or a server, and the method includes:

[0051] Step 301: A computing device performs semantic associations on n modal source statements to construct a semantic association diagram.

[0052] The semantic relationship diagram includes n different modal semantic nodes, a first connection edge used to connect semantic nodes of the same modal, and a second connection edge used to connect semantic nodes of different modal, where n is a positive integer greater than 1.

[0053] Taking a modal source statement as an example, the source statement corresponds to a set of semantic nodes, which includes at least one semantic node that is used to represent a semantic unit in the source statement.

[0054] The computer device is configured with a multimodal fusion encoder and a decoder, and the computer device extracts semantic nodes from the source statements of each modality through a multimodal graph representation layer to obtain n sets of semantic nodes corresponding to the source statements of n modalities, and uses a first connection edge through the multimodal graph representation layer to perform a connection between semantic nodes in the same modality for the n sets of semantic nodes, i.e., add a first connection edge between any two semantic nodes of the same modality, and use a second connection edge to perform a connection between semantic nodes of different modalities for the n sets of semantic nodes, i.e., add a second connection edge between the semantic nodes of different modalities to obtain a semantic relationship diagram.

[0055] Alternatively, the n-modal source statements include a first source statement in a textual form and a second source statement in a non-textual form, and the n-set of semantic nodes includes a first semantic node and a second semantic node. The computing device extracts a first semantic node from the first source statement by a multimodal graph representation layer and extracts a candidate semantic node from the second source statement, invokes the multimodal graph representation layer, calculates a first probability distribution for the candidate semantic node according to a semantic association between the first semantic node and the candidate semantic node, invokes the multimodal graph representation layer, and determines a second semantic node from the candidate semantic node based on the first probability distribution.

[0056] Wherein, for extracting a semantic node in the first source statement in text form, the computer device performs a word segmentation process on the first source statement to obtain m words after word segmentation, and the m words correspond to the first semantic node in the first source statement, where m is a positive integer; For extracting a semantic node in the second source statement in non-textual form, the computing device extracts a target corresponding to the semantics of at least one of the m phrases from the second source statement, the target being a second semantic node in the second source statement.

[0057] Illustratively, as shown in FIG. 4, the bimodal source statement includes a translation target image 31 and a translation target statement 32, and the content of the translation target statement 32 includes "Two boys are playing with a toy car." Each English word corresponds to one first semantic node, which are Vx1, Vx2, Vx3, Vx4, Vx5, Vx6, Vx7, and Vx8, respectively. The computer device cuts out candidate images from the translation target image 31 according to the semantics of the semantic nodes, calculates a first probability distribution according to the semantic association between the semantic nodes and the candidate images, and determines target images 1 and 2 corresponding to the semantics of Vx1 and Vx2, and target images 3 corresponding to the semantics of Vx6, Vx7, and Vx8 from the candidate images according to the first probability distribution. Vo1, Vo2, and Vo3 corresponding to target images 1, 2, and 3, respectively, are three second semantic nodes in the translation target image 31. The computer device establishes intra-modal semantic couplings between pairs of Vx1, Vx2, Vx3, Vx4, Vx5, Vx6, Vx7, and Vx8 using first coupling edges (solid lines), intra-modal semantic couplings between pairs of Vo1, Vo2, and Vo3 using first coupling edges, and inter-modal semantic couplings between the first semantic node and the second semantic node using second coupling edges (dashed lines).

[0058] Alternatively, different first connection edges are set corresponding to different modalities. When performing intra-modal connection for semantic nodes, the computer device determines an i-th type of first connection edge corresponding to an i-th modality by a multimodal graph representation layer, and performs connection between semantic nodes in the same modality for the i-set of semantic nodes using the i-th type of first connection edge. That is, the computer device adds an i-th type of first connection edge between any two semantic nodes in the i-set of semantic nodes, where i is a positive integer less than or equal to n.

[0059] As an option, when translating two modal source statements, where the two modal source statements are text and images respectively, the computer device establishes semantic associations between the two modal source statements through visual grounding tools and constructs a semantic association diagram.

[0060] Step 302: The computing device extracts a plurality of first word vectors from the semantic relationship graph.

[0061] Illustratively, the computer device processes the semantic relationship map using a word embedding method to obtain a plurality of first word vectors. Word embedding refers to mapping words to word vectors, and optionally, the word embedding method includes: Perform word embedding using a neural network model; Performing word embedding by performing dimensionality reduction on the term co-occurrence matrix; Performing word embedding using a probabilistic model; and and performing word embeddings for words based on the semantics of the context in which the words are located.

[0062] For example, one-hot encoding is used to represent words in a textual source statement, followed by word embedding using an embedding matrix.

[0063] Step 303: The computer device encodes the multiple first word vectors to obtain n encoded feature vectors.

[0064] The computing device performs intra-modal feature extraction on the first word vector using a multi-modal fusion encoder, and then performs inter-modal feature fusion on the vector obtained by the feature extraction.

[0065] For example, take the case where the value of n is 3. The multi-modal fusion encoder includes a first feature extraction function corresponding to a first modal, a second feature extraction function corresponding to a second modal, and a third feature extraction function corresponding to a third modal. The computer device performs feature extraction in the first modal for the first word vector through the first feature extraction function, performs feature extraction in the second modal for the first word vector through the second feature extraction function, and performs feature extraction in the third modal for the first word vector through the third feature extraction function, finally obtaining three hidden layer functions. The multi-modal fusion encoder further includes a first feature fusion function corresponding to the first modal, a second feature fusion function corresponding to the second modal, and a third feature fusion function corresponding to the third modal. The computer device performs inter-modal feature fusion for the above three hidden layer functions using a first feature fusion function, performs inter-modal feature fusion for the above three hidden layer functions using a second feature fusion function, and performs inter-modal feature fusion for the above three hidden layer functions using a third feature fusion function to obtain three feature fused hidden layer vectors, i.e., encoded feature vectors.

[0066] Step 304: The computing device decodes the n encoded feature vectors to obtain a translated target statement.

[0067] The computing device calls a decoder to decode the n encoded feature vectors to obtain a translated target statement, which is a statement obtained by translating the n modal source statements into a specified language class.

[0068] As described above, the translation method based on multimodal machine learning provided in the present embodiment performs semantic association for n modal source statements by a multimodal graph representation layer to construct a semantic association diagram, and uses a first connection edge to connect semantic nodes of the same modality in the semantic association diagram, and uses a second connection edge to connect semantic nodes of different modalities in the semantic association diagram, so that the semantic association between the source statements of multiple modalities is fully represented by the semantic association diagram. Then, a multimodal fusion encoder performs sufficient semantic fusion on the feature vectors in the semantic association diagram to obtain an encoded feature vector after encoding, and a more accurate target statement is obtained after decoding the encoded feature vector. The target statement is closer to the content, emotion, and language environment, etc., comprehensively expressed by the multimodal source statements.

[0069] Based on Fig. 3, the multi-modal fusion encoder includes e encoding modules connected in series, each encoding module includes n intra-modal fusion layers and n inter-modal fusion layers, each of which corresponds to n modalities one-to-one, where e is a positive integer. Thus, step 303 may include step 3031, as in Fig. 5, the steps are as follows: Step 3031: The computer device performs e times of intra-modal fusion and inter-modal fusion on the multiple first word vectors through e serially connected encoding modules to obtain n encoded feature vectors.

[0070] Here, intra-modal fusion refers to performing semantic fusion between first word vectors in the same modality, while inter-modal fusion refers to performing semantic fusion between first word vectors in different modalities.

[0071] Exemplarily, the intra-modal and inter-modal fusion of the encoded feature vectors can be realized by the following steps.

[0072] 1) Input the first word vector to n intra-modal fusion layers in the first encoding module, respectively, and perform semantic fusion within the same modality on the first word vector through the n intra-modal fusion layers, respectively, to obtain n first hidden layer vectors, each of which corresponds to a modal, that is, obtain n first hidden layer vectors that correspond one-to-one to the n modals.

[0073] For example, the computer device inputs a first word vector to a first intra-modal fusion layer in a first encoding module, performs intra-modal semantic fusion on the first word vector by the first intra-modal fusion layer to obtain a first hidden layer vector, inputs the first word vector to a second intra-modal fusion layer in the first encoding module, and performs intra-modal semantic fusion on the first word vector by the second intra-modal fusion layer to obtain a second hidden layer vector, ..., inputs the first word vector to an n-th intra-modal fusion layer in the first encoding module, and performs intra-modal semantic fusion on the first word vector by the n-th intra-modal fusion layer to obtain an n-th hidden layer vector.

[0074] A feature extraction function is set in the intra-modal fusion layer, and optionally the feature extraction function includes a self-attention function. Optionally, different or the same self-attention functions are set in different intra-modal fusion layers. As should be explained, different self-attention functions refer to different parameters in the functions, and if the self-attention functions corresponding to different modalities are different, the parameters in the functions corresponding to different modalities are different.

[0075] 2) The n first hidden layer vectors are input to respective inter-modal fusion layers in the first encoding module, and the respective inter-modal fusion layers perform semantic fusion between different modalities on the n first hidden layer vectors to obtain n first intermediate vectors, each of which corresponds to a modal, that is, n first intermediate vectors are obtained that correspond one-to-one to the n modals.

[0076] Exemplarily, the computer device inputs the n first hidden layer vectors to a first inter-modal fusion layer in a first encoding module, performs inter-modal semantic fusion on the n first hidden layer vectors by the first inter-modal fusion layer to obtain a first first intermediate vector corresponding to a first modal, inputs the n first hidden layer vectors to a second inter-modal fusion layer in the first encoding module, and performs inter-modal semantic fusion on the n first hidden layer vectors by the second inter-modal fusion layer to obtain a second first intermediate vector corresponding to a second modal, ..., inputs the n first hidden layer vectors to an nth inter-modal fusion layer in the first encoding module, and performs inter-modal semantic fusion on the n first hidden layer vectors by the nth inter-modal fusion layer to obtain an nth first intermediate vector corresponding to the nth modal.

[0077] A feature fusion function is set in the inter-modal fusion layer, and the feature fusion functions set in different inter-modal fusion layers are optional and different or the same. As needs to be explained, different feature fusion functions refer to different parameters in the functions, or mean different calculation methods of the functions.

[0078] Optionally, each encoding module further includes n first vector transformation layers corresponding to the n modalities in a one-to-one manner. After obtaining the n first intermediate vectors, the computer device further inputs the n first intermediate vectors to the n first vector transformation layers corresponding to the modalities to which they belong, respectively, to perform nonlinear transformation, and obtain n first intermediate vectors after nonlinear transformation.

[0079] 3) The n first intermediate vectors are input to the jth encoding module for the jth encoding process, and this is continued until the last encoding module outputs n encoded feature vectors, each of which corresponds to a modal, that is, until the last encoding module outputs n encoded feature vectors that correspond one-to-one to the n modals.

[0080] The computer device inputs the n intermediate vectors to a second encoding module to perform a second encoding process to obtain n re-encoded first intermediate vectors, ..., inputs the re-encoded n first intermediate vectors to a j-th encoding module to perform a j-th encoding process to obtain n re-encoded first intermediate vectors, ..., inputs the re-encoded n first intermediate vectors to an e-th encoding module to perform an e-th encoding process to obtain n encoded feature vectors, where j is a positive integer greater than 1 and less than or equal to e. Optionally, the hierarchical structure of each of the encoding modules among the e encoding modules connected in series is the same. That is, the j-th encoding module processes according to the step of the first encoding module encoding the first intermediate vector, until the last encoding module outputs the encoded feature vector.

[0081] For example, in this embodiment, a self-attention mechanism is used to model the same modal intra-semantic information. Then, the j-th encoding module calculates the first hidden layer vector [Equation 1] corresponding to the text statement, and the formula is: [Number 2], Here, [Number 3] refers to the first word vector corresponding to the text statement or the first intermediate vector output by the (j-1)th encoding module, x is used to mark the semantic node of the text statement and the vector calculated and obtained by the semantic node of the text statement, and MultiHead(Q,K,V) is a multiple attention mechanism modeling function that takes triplets (Queries, Key, Values) as input, Q is a query matrix, K is a key matrix, and V is a value matrix, where Q, K, and V are calculated and obtained from [Number 4] and the parameter vector.

[0082]

number

number

number

number

[0083] The j-th multimodal fusion encoder computes the first hidden layer vector [Equation 5] corresponding to the image, and the formula is [Number 6],

[0084]

number

number

[0085] where [Number 7] refers to the first word vector corresponding to an image or the first intermediate vector output by the (j-1)th encoding module;

[0086]

number

[0087] In this embodiment, a cross-modal fusion mechanism based on a gating mechanism is further used to model the multi-modal semantic fusion, so that the jth encoding module calculates a first intermediate vector or encoding feature vector [Equation 8] corresponding to the text statement, and the formula is: [Number 9], [Number 10],

[0088]

number

number

number

[0089] Here, A denotes a set. Correspondingly, [Equation 11] is a set of neighboring nodes in the semantic relation diagram of the first semantic node [Equation 12]. [Equation 13] denotes the u-th semantic node of the text statement, where u is a positive integer. [Equation 14] is the semantic expression vector of the s-th semantic node of the image in the j-th encoding module, and [Equation 15] is the semantic expression vector of the u-th semantic node of the text statement in the j-th encoding module. [Equation 16] and [Equation 17] are parameter matrices, [Equation 18] denotes the exclusive NOR operation, and Sigmoid() is an s-curve type function. o is used to mark the semantic node of the image and the vector obtained by calculating the semantic node of the image. Furthermore, the first intermediate vector or the encoding feature vector [Equation 19] corresponding to the image is calculated by the same calculation method, and will not be described in detail again here.

[0090]

number

number

number

number

number

number

number

number

number

[0091] After undergoing multi-modal fusion, a FeedForward Neural (FFN) is further used in this embodiment to generate a final encoded feature vector. The encoded feature vector corresponding to the text statement and the encoded feature vector corresponding to the image are respectively: [Number 20], [Number 21],

[0092]

number

number

[0093] where [Equation 22], {} denotes a set, [Equation 23] denotes the encoding feature vector corresponding to the u-th semantic node of the text statement in the j-th encoding module, and [Equation 24] denotes the encoding feature vector corresponding to the s-th semantic node of the image in the j-th encoding module.

[0094]

number

number

number

[0095] As described above, the translation method based on multimodal machine learning provided in the present embodiment uses a multimodal graph representation layer to perform semantic association for n modal source languages ​​to construct a semantic association diagram. In the semantic association diagram, a first connection edge is used to connect semantic nodes of the same modality, and a second connection edge is used to connect semantic nodes of different modalities, so that the semantic association diagram fully represents the semantic association between the multiple modal source languages. Then, a multimodal fusion encoder performs sufficient semantic fusion on the feature vectors in the semantic association diagram to obtain an encoded feature vector after encoding, and after decoding the encoded feature vector, a more accurate target statement is obtained. The target statement is closer to the content, emotions, and language environment, etc., comprehensively expressed by the multimodal source language.

[0096] In this method, the multi-modal fusion encoder includes e encoding modules connected in series, each of which includes an intra-modal fusion layer and an inter-modal fusion layer, and the intra-modal and inter-modal feature fusion are alternately performed multiple times to obtain a more complete encoded feature vector through semantic fusion, and further decode a more accurate target statement corresponding to the n-modal source language.

[0097] Based on Fig. 3, the decoder further includes d decoding modules connected in series, where d is a positive integer. Thus, step 304 may include steps 3041 to 3044, as shown in Fig. 6, these steps are as follows:

[0098] Step 3041: The computing device obtains the first target phrase through the second word vector layer.

[0099] Here, the first target phrase is the translated phrase in the goal statement. The computer device translates the phrases in the goal statement one by one, and after translating the rth phrase in the goal statement, the rth phrase is taken as the first target phrase and used to translate the r+1th phrase. In other words, the computer device inputs the rth phrase into the second word vector layer, where r is a non-negative integer.

[0100] Step 3042: The computing device performs feature extraction on the first target phrase through a second word vector layer to obtain a second word vector.

[0101] For example, the computer device performs word embedding on the first target phrase through the second vector layer to obtain a second word vector. Word embedding is a technique for expressing words as real vectors in a vector space, and in this embodiment, word embedding refers to mapping words to word vectors. For example, if "I" is mapped to obtain a word vector (0.1,0.5,5), that is, (0.1,0.5,5) is the word vector after word embedding on "I".

[0102] Step 3043: The computer device uses the d serially connected decoding modules to combine the second word vector and the encoded feature vector to perform feature extraction, and obtain a decoded feature vector.

[0103] The computer device calls the d decoding modules connected in series to process the encoded feature vector and the second word vector based on the attention mechanism to extract a decoded feature vector.

[0104] Optionally, each of the d decoding modules connected in series includes a first self-attention layer, a second self-attention layer and a second vector conversion layer. For extracting the decoded feature vector, the computer device inputs the second word vector into the first self-attention layer of the first decoding module, performs feature extraction on the second word vector by the first self-attention layer to obtain a second hidden layer vector, inputs the second hidden layer vector and the encoded feature vector into the second self-attention layer of the first decoding module, combines the second hidden layer vector and the encoded feature vector by the second self-attention layer to perform feature extraction to obtain a second intermediate vector, inputs the second intermediate vector into the k-th decoding module to perform the k-th decoding process, and continues until the last one decoding module outputs the decoded feature vector, where k is a positive integer greater than 1 and less than or equal to d.

[0105] Wherein, the first self-attention layer is used to process the second word vector according to a self-attention mechanism to extract a second hidden layer vector, and the second self-attention layer is used to process the second hidden layer vector and the encoded feature vector using a language class of the target statement according to an attention mechanism to obtain a second intermediate vector. The first self-attention layer includes a first self-attention function, and the second self-attention layer includes a second self-attention function, and the parameters of the first self-attention function and the second self-attention function are different.

[0106] As an option, each decoding module further includes a second vector transformation layer, and after calculating and obtaining the second intermediate vector, the computer device further inputs the second intermediate vector to the second vector transformation layer to perform nonlinear transformation, and obtains a second intermediate vector after nonlinear transformation.

[0107] Step 3044: The computer device inputs the decoded feature vector into a classifier, calculates a probability distribution corresponding to the decoded feature vector through the classifier, and determines a second target phrase following the first target phrase according to the probability distribution.

[0108] As an option, the classifier includes a normalization (softmax) function, and the computing device calculates a probability distribution corresponding to the decoded feature vector through the softmax function, and determines a second target phrase after the first target phrase based on the probability distribution corresponding to the decoded feature vector.

[0109] As described above, the translation method based on multimodal machine learning provided in the present embodiment uses a multimodal graph representation layer to perform semantic association for n modal source languages ​​to construct a semantic association diagram. In the semantic association diagram, a first connection edge is used to connect semantic nodes of the same modality, and a second connection edge is used to connect semantic nodes of different modalities, so that the semantic association diagram fully represents the semantic association between the multiple modal source languages. Then, a multimodal fusion encoder performs sufficient semantic fusion on the feature vectors in the semantic association diagram to obtain an encoded feature vector after encoding, and a more accurate target statement is obtained after decoding the encoded feature vector. The target statement is closer to the content, emotion, and language environment, etc., comprehensively expressed by the multimodal source language.

[0110] The method further includes iteratively applying attention to the encoded feature vectors and the second hidden layer vectors using the linguistic class of the target statement by d decoding modules to decode a more accurate target statement.

[0111] As further explained, a test comparison was conducted between the multimodal machine translation model provided by the present application and a previous multimodal neural machine translation (NMT), and it was found that the multimodal machine translation model provided by the present application has the highest translation effect. For example, the input data is taken as two source languages, namely image and text, and the test comparison is described in detail as follows.

[0112] The multimodal machine translation model provided herein is constructed based on the attention codec framework, and the objective function is to maximize the log-likelihood of the training data. In essence, the multimodal fusion encoder provided herein may be viewed as a multimodal extended graph neural network (GNN). To construct the multimodal fusion encoder, the input image and text are represented in a multimodal graph (i.e., a semantic association diagram) in correspondence with each other, and then multiple multimodal fusion layers are stacked based on the multimodal graph to learn node (i.e., semantic node) representations, and provide the decoder with an attention-based context vector.

[0113] First, regarding the construction of a multimodal graph, formally, a multimodal graph is undirected and can be formalized as G=(V,E), where in a node set V, each node represents a text phrase or a visual object, where the node corresponding to the text is called a semantic node, and the node corresponding to the visual object is called a visual node, and the following policy is used to construct semantic associations between the nodes:

[0114] 1. Extracting nodes (1) To fully utilize the text information, every word in the text is treated as an individual text node. For example, in Fig. 4, the multimodal graph contains a total of eight text nodes, and each text node corresponds to one word in the input statement (i.e., the statement to be translated). (2) We use the Stanford parser to identify every noun phrase in the input statement, and then apply the visual grounding toolkit to identify the corresponding bounding boxes (visual objects) in the input image (i.e., the image to be translated) for every noun phrase. After that, every detected visual object is treated as an independent visual node. For example, in Fig. 4, the text nodes Vx1 and Vx2 correspond to the visual nodes Vo1 and Vo2, and the text nodes Vx6, Vx7, and Vx8 correspond to the visual node Vo3.

[0115] 2. To capture various semantic associations between multimodal semantic units, two kinds of edges (i.e., connecting edges) are used to connect semantic nodes. The two kinds of edges in edge set E include: (1) any two semantic nodes in the same modality are connected by an intra-modal edge (first connecting edge); and (2) any text node and corresponding visual node are connected by an inter-modal edge (second connecting edge). For example, as shown in FIG. 4, Vo1 and Vo2 are connected by an intra-modal edge (solid line), and Vo1 and Vx1 are connected by an inter-modal edge (second connecting edge). (Dash line ) to combine them.

[0116] Second, for the embedding layer, we need to introduce one word embedding layer to initialize the node state before inputting the multimodal graph into the stacked multimodal fusion layer. For each text node Vxu, we define its initial state Hxu as the sum of the word embedding and the position embedding. For the initial state Hos of the visual node Vos, we need to extract visual features by the fully-connected layer of the Region Of Interest pooling (ROI pool) layer in Faster-RCNN, and then project the visual features into the same space as the text representation using a multi-layer perceptron with a Rectified Linear Unit (ReLU) activation function.

[0117] Here, RCCN is a rich feature hierarchies for accurate object detection and semantic segmentation.

[0118] 3. As shown in Fig. 7, the left part shows an encoder, and a multimodal fusion layer based on the e-layer graph is stacked on top of the embedding layer 402 to encode the multimodal graph. In the multimodal fusion layer, intra-modal and inter-modal fusion are sequentially performed to update all node states. In this way, the final node state is a simultaneous encoding of context information and cross-modal semantic information in the same modality. In particular, since visual nodes and text nodes are two kinds of semantic units containing different modal information, functions with similar operations but different parameters are used to model the node state update process.

[0119] Illustratively, in j multimodal fusion layers, the updates of text node states

[25] and visual node states

[26] mainly involve the following steps:

[0120]

number

number

[0121] Step 1: Intra-modal fusion. In this step, we use self-attention to perform information fusion between adjacent nodes in the same modality to generate context representations for each node. Formally, the formula for computing the context representations of all text nodes

[27] is [Number 28],

[0122]

number

number

[0123] Here, MultiHead(Q,K,V) is a multiple attention mechanism modeling function (also called multihead self-attention function) that takes as input a query matrix Q, a key matrix K, and a value matrix V. Similarly, the formula for computing the context representation of all visual nodes [Equation 29] is [Number 30]

[0124]

number

number

[0125] In particular, the initial states of visual objects are extracted by deep learning algorithms (deep CNNs), and thus we apply a simplified multi-head self-attention to represent the initial states of visual objects, where we remove the linear item values ​​obtained and the final output.

[0126] Step 2: Intermodal fusion. When performing feature fusion between multimodal features, a kind of cross-modal gating control mechanism with element operation characteristics is used to learn the semantic information of the cross-modal neighborhood of each node. Specifically, the method for generating the state representation of the text node Vxu [Equation 31] is as follows: [Number 32], [Number 33],

[0127]

number

number

number

[0128] Here, [Equation 34] is a set of neighboring nodes in the multimodal graph of node Vxu, and [Equation 35] and [Equation 36] are parameter matrices. Similarly, the method for generating the state representation [Equation 37] of text node Vos is as follows: [Number 38] [Number 39],

[0129]

number

number

number

number

number

number

[0130] Here, [Equation 40] is a set of neighboring nodes in the multimodal graph of node Vos, and [Equation 41] and [Equation 42] are parameter matrices.

[0131]

number

number

number

[0132] After the above multimodal fusion process, a feedforward neural network is used to generate the final delta hidden layer representation. The calculation process of the text node state [Equation 43] and the image node state [Equation 44] is as follows: [Number 45] [Number 46],

[0133]

number

number

number

number

[0134] Here, [equation 47] indicates that all text node states and image node states have been updated.

[0135]

number

[0136] Fourth, the decoder is similar to a conventional Transformer decoder. Because visual information has already been fused to all text nodes by multiple graph-based multimodal fusion layers, the decoder is allowed to dynamically utilize multimodal context by focusing only on the text node state, i.e., only the text node state is input to the decoder.

[0137] As shown in the right part of Figure 7, d identical layers are stacked to generate the target-side hidden state, where each layer is composed of three sub-layers. Specifically, the top two sub-layers are masking self-attention Ej and codec attention Tj, respectively, which integrate the target and source language-side contexts, [Number 48] [Number 49],

[0138]

number

number

[0139] Here, S(j-1) denotes the target-side hidden state in the j-1th layer. In particular, S(0) is the embedding vector of the input target phrase, and [Equation 50] is the top-layer hidden state in the decoder. Then, a fully connected feedforward neural network in one position direction is used to generate S(j), and the formula is [Number 51],

[0140]

number

number

[0141] Finally, we use a softmax layer to define a probability distribution for generating the goal statements. The input to this layer is the hidden state of the top layer [Eq. 52]. [Number 53],

[0142]

number

number

[0143] where X is the input statement to be translated, I is the input image to be translated, Y is the target statement (i.e., the translated statement), and W and b are the parameters of the softmax layer.

[0144] In the experimental process, the translation task is to translate English into French and German, and the dataset is the Multi30K dataset. Here, each image in the dataset is paired with an English description and a human-translated German and French. The training, validation and test sets contain 29000, 1014 and 1000 examples, respectively. In addition, we further evaluate various models in the WMT17 test set and the fuzzy MSCOCO test set, which contain 1000 and 461 examples, respectively. In this experiment, the preprocessed statements are directly used to split words into subwords by byte pair encoding and 10000 merge operations.

[0145] Visual Features: We first use the Stanford parser to identify noun phrases from each source statement, and then use the visual grounding toolkit to find relevant visual objects for the identified noun phrases. For each phrase, we mitigate the negative effect of abundant visual objects by keeping the highest predicted probability of its corresponding visual object. In each sentence, the average number of objects and words is around 3.5 and 15.0, respectively. Finally, we use a pre-trained ResNet-100 Faster RCNN to compute 2048-dimensional features of these objects.

[0146] Settings: Transformer is used as the basis. Since the training corpus is relatively small, the model after training tends to overfit, so we first perform one small grid search to obtain the hyperparameters on a set of English to German translation validation sets. Specifically, the word embedding dimension and hidden size are 128 and 256, respectively. The decoder has four layers, and the number of attention heads is 4. The dropout rate is set as 0.5. Each lot is composed of about 2000 source code symbols and target tokens. The Adam optimizer with a given learning rate is applied to optimize various models, and the same other settings are used. Finally, the Bilingual Evaluation Understudy (BLEU) index and METEOR index are used to evaluate the quality of the translation. As it needs to be explained, every model is run three times in each experiment, and the average results are reported.

[0147] Base Model: In addition to the text-based Transformer (TransFormer, TF), we also use visual features to perform transformation using several effective methods, and compare the model provided in the embodiments of this application with the Transformer.

[0148] 1. ObjectAsToken (TF): This is a variation of the transformer in which every visual object is regarded as an additional source code symbol and placed in front of the input statement.

[0149] 2. Enc-att (TF), which uses an encoder-based image attention mechanism in the transformer to add visual feature vectors based on individual source annotations and attention.

[0150] 3. Doubly-att (TF): This is a double-attention transformer. In each decoding layer, a cross-modal multi-head attention sub-layer is inserted before the fully connected feed-followed layer, which generates a visual context vector based on visual features.

[0151] Correspondingly, the performance of several major multimodal Neural Machine Translation (NMT) models has been developed, such as Doubly-att (RNN), Soft-att (RNN), Stochastlc-att (RNN), Fusion-conv (RNN), Trg-mul (RNN), VMM T (RNN) and Delberation Network (TF), where RNN is a Recurrent Neural Network.

[0152] The number of multimodal fusion layers, e, is an important hyperparameter that directly determines the degree of fine-grained semantic fusion in the encoder. Therefore, we first examine its impact on the English to German translation validation set. Figure 8 shows the experimental results, and the model reaches the optimal p-shape when e is 3. Therefore, we used e=3 in all subsequent experiments.

[0153] [Table 1]

[0154] Table 1 shows the main results of the English to German translation task. Compared with Fusion-conv (RNN) and Trg-mul (RNN) in METEOR, the model provided by the embodiment of this application outperforms most previous models. The two sets of results are determined by the system state on the WMT2017 test set, which is selected based on METEOR. Compared with the baseline model, the following conclusions can be drawn:

[0155] First, the model provided by the embodiment of this application is superior to ObjectAsToken (TF), which combines regional visual features and text together to form attentionable sequences, and utilizes self-attention mechanism to perform multi-modal fusion. The basic reason is that it models the semantic correspondence between semantic units of different modalities, and distinguishes model parameters of different modalities.

[0156] Secondly, the model provided by the embodiment of the present application also significantly outperforms Enc-att(TF), which may be regarded as a single-layer semantic fusion encoder. In addition to the advantage of modeling semantic correspondence, it is further speculated that multi-layer multi-modal semantic interaction is also advantageous to NMT.

[0157] Third, compared to Doubly-att (TF), which extracts visual information using only attention mechanisms, the model provided by the embodiments of the present application is significantly improved because it provides sufficient multimodal fusion at the encoder.

[0158] [Table 2]

[0159] In addition, the test set is divided into different groups based on the length of the source sentence and the number of noun phrases, and then the performance of different models on the test set of each group is compared. The BLEU scores of the above groups are shown in Figures 9 and 10. In summary, the model provided by the embodiment of the present application still always reaches the optimal performance in all groups. Therefore, the effectiveness and versatility of the model provided by the embodiment of the present application are again demonstrated. It should be noted that in sentences with relatively many phrases, the sentences are generally longer, and the model provided by the embodiment of the present application has a deeper meaning than the improvement of the basic model. It is assumed that long sentences often contain relatively more ambiguous words. Therefore, compared with short sentences, long sentences may need to better utilize visual information as supplementary information, which can be achieved by the multimodal semantic interaction of the model provided by the embodiment of the present application.

[0160] Furthermore, Table 4 shows the training and decoding speeds of the model and the basic model provided by the embodiment of the present application. In the training process, the model provided by the embodiment of the present application can process about 1.1K tokens per second, which is comparable to other multi-modal models. In terms of the decoding process, the model provided by the embodiment of the present application translates about 16.7 phrases per second, which is a little slower than the transformer. In addition, the model provided by the embodiment of the present application achieves better performance with only a small amount of additional parameters introduced.

[0161] [Table 3]

[0162] Further experiments were carried out to study the effectiveness of different ingredients, comparing the model provided in the examples of this application with the following variations in Table 2:

[0163] (1) Intermodal fusion. In this variation, we use two independent transformer-based encoders to learn semantic representations of words and visual objects, respectively, and then use a dual-attention decoder to merge the text and visual context into the decoder. The results in the third row in Table 2 show that removing intermodal fusion leads to a significant degradation in performance. This indicates that semantic interactions between multimodal semantic units are useful for multimodal representation learning.

[0164] (2) From visual grounding to full connectivity. Fully connect words and visual objects together to establish inter-modal correspondence. The result in the fourth row in Table 2 shows that this change leads to a significant degradation in performance. The fundamental reason is that fully connected semantic correspondence brings a lot of noise to the model provided by the embodiment of this application.

[0165] (3) From different parameters to unified parameters. When constructing this variation, we assign unified parameters to update the node states in different modes. As is clear, the performance degradation reported in the fifth row in Table 2 also proves the effectiveness of the method using different parameters.

[0166] (4) Visual node attention. Unlike models that only consider text nodes, this variation of the decoder is allowed to consider these two types of nodes using a dual-attention decoder. As can be observed from the results in row 6 of Table 2, considering all nodes does not bring any further improvement. The above results verify the original assumption, i.e., visual information is already fully incorporated into the text nodes in the encoder.

[0167] (5) Text node attention and visual node attention. However, when only visual nodes are considered, the model's performance drops sharply, as shown in the seventh row in Table 2. This is because, although the number of visual nodes is much smaller than that of text nodes, text nodes cannot generate sufficient translation context.

[0168] For illustrative purposes, we further conduct experiments on the English to French translation dataset. As can be seen from Table 3, the model provided by the embodiment of the present application still achieves better performance than all previous models. This again proves that the model provided by the embodiment of the present application in multimodal NMT is effective and generalizable across different languages.

[0169] Table 2 compares the machine translation model provided in the related multimodal NMT system and the multimodal NMT system provided in the embodiment of the present application. As is clear from the BLEU and METEOR indices, the machine translation model provided in the present application also achieves better results for translation between English and French, with three of the four indices having the highest values ​​(bold numbers).

[0170] [Table 4]

[0171] 11, a translation device based on multimodal machine learning provided by one exemplary embodiment of the present application is shown. The device is a part or whole of a computer device by software, hardware or a combination thereof, and includes a semantic association module 501, a feature extraction module 502, a vector encoding module 503, and a vector decoding module 504.

[0172] The semantic association module 501 is used for obtaining a semantic relationship diagram based on n source statements belonging to different modalities, the semantic relationship diagram including n semantic nodes of different modalities, a first connection edge used for connecting the semantic nodes of the same modality, and a second connection edge used for connecting the semantic nodes of different modalities, the semantic nodes are used for indicating one semantic unit of the source statements in one modality, and n is a positive integer greater than 1.

[0173] Optionally, the method includes performing semantic association for the n modal source languages ​​using a multimodal graph representation layer to construct a semantic relationship diagram, the semantic relationship diagram including n different modal semantic nodes, a first connection edge for connecting the same modal semantic nodes, and a second connection edge for connecting the different modal semantic nodes, where n is a positive integer greater than 1; The feature extraction module 502 is used for extracting a plurality of first word vectors from the semantic relationship diagram, and optionally extracting the first word vectors from the semantic relationship diagram through a first word vector layer; The vector encoding module 503 is used for encoding the plurality of first word vectors to obtain n encoded feature vectors, and optionally encoding the first word vectors by a multi-modal fusion encoder to obtain the encoded feature vectors; The vector decoding module 504 is used for decoding the n encoded feature vectors to obtain the translated target statement, and optionally calls a decoder to decode the encoded feature vectors to obtain the translated target statement.

[0174] In some alternative embodiments, the semantic association module 501 is used for obtaining an n-set of semantic nodes, where one set of semantic nodes corresponds to a source statement of one modal, and adding the first connection edge between any two of the semantic nodes of the same modality and adding the second connection edge between any two of the semantic nodes of different modalities to obtain the semantic relationship diagram. Optionally, the semantic association module 501 is used for extracting semantic nodes from the source language of each modality by a multimodal graph representation layer to obtain an n-set of semantic nodes corresponding to the source languages ​​of the n modals, and performing a connection between semantic nodes within the same modality for the n-set of semantic nodes by the first connection edge and a connection between semantic nodes between different modalities for the n-set of semantic nodes by the multimodal graph representation layer to obtain the semantic relationship diagram.

[0175] In some alternative embodiments, the n-modal source languages ​​include a first source language in textual form and a second source language in non-textual form, and the n-set of semantic nodes includes a first semantic node and a second semantic node; The semantic association module 501 is used for obtaining a first semantic node, the first semantic node being obtained by a multimodal graph representation layer processing the first source statement; obtaining a candidate semantic node, the candidate semantic node being obtained by a multimodal graph representation layer processing the second source statement; obtaining a first probability distribution of the candidate semantic node, the first probability distribution being obtained by the multimodal graph representation layer calculating according to a semantic association between the first semantic node and the candidate semantic node; and determining the second semantic node from the candidate semantic node, the second semantic node being determined by the multimodal graph representation layer based on the first probability distribution.

[0176] As an option, the semantic association module 501 is used for extracting a first semantic node from the first source statement by a multimodal graph representation layer and extracting a candidate semantic node from the second source statement, invoking the multimodal graph representation layer to calculate a first probability distribution of the candidate semantic node according to the semantic association between the first semantic node and the candidate semantic node, and invoking the multimodal graph representation layer to determine a second semantic node from the candidate semantic node based on the first probability distribution.

[0177] In some alternative embodiments, the semantic association module 501 is used to add an i-th type of first connection edge between any two semantic nodes in the same modality in the i-th set of semantic nodes, where the i-th type of first connection edge corresponds to the i-th modality, and i is a positive integer less than or equal to n.

[0178] As an option, the semantic association module 501 is used to determine an i-th type of first connection edge corresponding to an i-th modality through the multimodal graph representation layer, and perform connection between semantic nodes in the same modality for the i-th set of semantic nodes using the i-th type of first connection edge, where i is a positive integer less than or equal to n.

[0179] In some alternative embodiments, the vector encoding module 503 is used for performing intra-modal fusion and inter-modal fusion e times on the first word vectors to obtain the n encoded feature vectors, where the intra-modal fusion refers to performing semantic fusion between the first word vectors in the same modality, and the inter-modal fusion refers to performing semantic fusion between the first word vectors in different modalities, where e is a positive integer.

[0180] Optionally, the multi-modal fusion encoder includes e encoding modules connected in series, where e is a positive integer; The vector encoding module 503 is used to perform intra-modal fusion and inter-modal fusion e times on the first word vector by e encoding modules connected in series to obtain an encoded feature vector, where the intra-modal fusion refers to performing semantic fusion between the first word vectors in the same modality, and the inter-modal fusion refers to performing semantic fusion between the first word vectors in different modalities. In some alternative embodiments, each encoding module includes n intra-modal fusion layers and n inter-modal fusion layers, each of which corresponds one-to-one to the n modalities; The vector encoding module 503 inputs the first word vector into n intra-modal fusion layers in the first encoding module, respectively, and performs the same intra-modal semantic fusion on the first word vector by the n intra-modal fusion layers to obtain n first hidden layer vectors, where one of the first hidden layer vectors corresponds to one modal, that is, obtains n first hidden layer vectors that correspond one-to-one to the n modals; Inputting the n first hidden layer vectors into respective inter-modal fusion layers in a first encoding module, and performing different modal semantic fusion on the n first hidden layer vectors by each inter-modal fusion layer to obtain n first intermediate vectors, where one of the intermediate vectors corresponds to one modality, i.e., obtaining n first intermediate vectors that correspond one-to-one to the n modals; The n first intermediate vectors are input to the jth encoding module to perform the jth encoding process, and this is continued until the last encoding module outputs n encoded feature vectors, where one encoded feature vector corresponds to one modal, that is, until the last encoding module outputs n encoded feature vectors that correspond one-to-one to the n modals, where j is a positive integer greater than 1 and less than or equal to e.

[0181] In some alternative embodiments, each encoding module further includes n first vector transformation layers, each of which corresponds to one modality, i.e., n first vector transformation layers corresponding one-to-one to the n modals; The vector encoding module 503 is further used for inputting the n first intermediate vectors into n first vector transformation layers corresponding to the modals to which they belong, respectively, to perform nonlinear transformation, and obtain the n first intermediate vectors after nonlinear transformation.

[0182] In some alternative embodiments, the hierarchical structure in each of the e serially connected encoding modules is the same.

[0183] In some alternative embodiments, different or the same self-attention functions are set for different intra-modal fusion layers, and different or the same feature fusion functions are set for different inter-modal fusion layers.

[0184] In some alternative embodiments, the vector decoding module 504 is used for: performing feature extraction on a first target phrase to obtain a second word vector, where the first target phrase is a translated phrase in the target statement; combining the second word vector with the encoded feature vector to perform feature extraction to obtain a decoded feature vector; determining a probability distribution corresponding to the decoded feature vector; and determining a second target phrase after the first target phrase based on the probability distribution.

[0185] Optionally, the decoder comprises d decoding modules connected in series, d being a positive integer; The vector decoding module 504 obtains a first target phrase by a second word vector layer, where the first target phrase is a translated phrase in the target statement; and performs feature extraction on the first target phrase by the second word vector layer to obtain a second word vector; The second word vector is combined with the encoded feature vector by d decoding modules connected in series to perform feature extraction to obtain a decoded feature vector; the decoded feature vector is input to a classifier, and the classifier calculates a probability distribution corresponding to the decoded feature vector, and determines a second target phrase following the first target phrase based on the probability distribution.

[0186] In some alternative embodiments, each of the d serially connected decoding modules includes a first self-attention layer and a second self-attention layer; The vector decoding module 504 inputs the second word vector into a first self-attention layer in the first decoding module, and performs feature extraction on the second word vector by the first self-attention layer to obtain a second hidden layer vector; inputting the second hidden layer vector and the encoded feature vector into a second self-attention layer in the first decoding module, and combining the second hidden layer vector and the encoded feature vector through the second self-attention layer to perform feature extraction to obtain a second intermediate vector; The second intermediate vector is input to the kth decoding module to perform the kth decoding process, and the process continues until the last decoding module outputs the decoded feature vector, where k is a positive integer greater than 1 and less than or equal to d.

[0187] In some alternative embodiments, each decoding module further includes a second vector transformation layer; The vector decoding module 504 is further used for inputting the second intermediate vector to a second vector transformation layer for nonlinear transformation to obtain a second intermediate vector after nonlinear transformation.

[0188] As described above, the translation device based on multimodal machine learning provided in this embodiment performs semantic association for n modal source languages ​​through a multimodal graph representation layer to construct a semantic association diagram, and uses a first connection edge to connect semantic nodes of the same modality in the semantic association diagram, and uses a second connection edge to connect semantic nodes of different modalities in the semantic association diagram, so that the semantic association between the multiple modal source languages ​​is fully represented by the semantic association diagram. Then, the multimodal fusion encoder performs sufficient semantic fusion on the feature vectors in the semantic association diagram to obtain an encoded feature vector after encoding, and further obtains a more accurate target statement after decoding the encoded feature vector. The target statement is closer to the content, emotion, and language environment, etc., comprehensively expressed by the multimodal source language.

[0189] Referring to Figure 12, a structural diagram of a server provided in one embodiment of the present application is shown, which is used to implement the steps of the translation method based on multimodal machine learning provided in the above embodiment. Specifically, The server 600 includes a CPU (Central Processing Unit) 601, a system memory 604 including a RAM (Random Access Memory) 602 and a ROM (Read-Only Memory) 603, and a system bus 605 coupling the system memory 604 and the CPU 601. The server 600 further includes a basic I / O (Input / Output) system 606 that assists in transmitting information between devices in the computer, and a mass storage device 607 used to store an operating system 613, application programs 614, and other program modules 615.

[0190] The basic input / output system 606 includes a display 608 used to display information and input devices 609, such as a mouse, keyboard, etc., used by a user to input information, where both the display 608 and the input devices 609 are coupled to the central processing unit 601 by an input / output controller 610 that is coupled to the system bus 605. The basic input / output system 606 may further include an input / output controller 610 for receiving and processing input from a number of other devices, such as a keyboard, mouse, or electronic stylus. Similarly, the input / output controller 610 may also provide output to a display screen, a printer, or other type of output device.

[0191] The mass storage device 607 is coupled to the central processing unit 601 by a mass storage controller (not shown) coupled to the system bus 605. The mass storage device 607 and its associated computer readable media provide non-volatile storage for the server 600. In other words, the mass storage device 607 may include a computer readable medium (not shown), such as a hard disk or a CD-ROM (Compact Disc Read-Only Memory) drive, etc.

[0192] Without loss of generality, the computer readable media may include computer storage media and communication media. Computer storage media includes volatile and non-volatile, removable and non-removable media implemented in any method or technology used for storing information such as computer readable instructions, data structures, program modules, or other data. Computer storage media includes RAM, ROM, EPROM (Erasable Programmable Read-Only Memory), EEPROM (Electrically Erasable Programmable Read-Only Memory), Flash Memory or other solid state memory technology, CD-ROM, DVD (Digital Versatile Disc) or other optical storage, tape cassettes, magnetic tape, magnetic disk storage, or other magnetic storage devices. Of course, those skilled in the art will recognize that the computer storage media is not limited to the above several types. The system memory 604 and mass storage device 607 may be collectively referred to as memory.

[0193] According to various embodiments of the present application, the server 600 can further be coupled to a remote computer in a network, such as the Internet, for example, via a network, and can operate in such a manner that the server 600 can be coupled to a network 612 by a network interface unit 611 coupled to the system bus 605, or can be coupled to another type of network or remote computer system (not shown) using the network interface unit 611.

[0194] In an exemplary embodiment, a computer-readable storage medium is further provided, for example a memory 602 including instructions, which can be executed by the processor 601 of the server 600 to complete the multimodal machine learning based translation method. Alternatively, the computer-readable storage medium may be a non-transitory storage medium, for example, the non-transitory storage medium may be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.

[0195] In an exemplary embodiment, a computer program product is further provided, which includes a computer program that may be executed by a processor of an electronic device to implement the multimodal machine learning based translation method.

[0196] As can be understood by those skilled in the art, all or part of the steps for implementing the above embodiments may be completed by hardware, or may be completed by a program issuing instructions to relevant hardware, and the above program may be stored in a type of computer-readable storage medium, and the above-mentioned storage medium may be a read-only memory, a magnetic disk, an optical disk, etc.

[0197] The above description is merely a selectable embodiment of the present application, and is not intended to limit the present application. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principle of the present application should be included within the protection scope of the present application. [Explanation of symbols]

[0198] 11 Intramodal Fusion Layer 12 Intermodal Fusion Layer 13 First Vector Transformation Layer 21 1st self-attention layer 22 Second layer of self-attention 23 Second Vector Transformation Layer 31 Images to be translated 32 Translated statements 100 Multimodal Machine Translation Models 101 Multimodal Graph Representation Layer 102 First word vector layer 103 Multimodal Fusion Encoder 104 Decoder 105 Second Word Vector Layer 106 Classifier 220 Terminals 240 Servers 502 Feature Extraction Module 503 Vector Encoding Module 504 Vector Decoding Module 600 Servers 601 Central Processing Unit 601 Processor 602 Memory 604 System Memory 605 System Bus 606 Output System 607 Mass storage equipment 608 Display 609 Input Devices 610 Input Output Controller 611 Network Interface Unit 612 Network 613 Operating Systems 614 Application Program 615 Program Module 1031 Encoding Module 1042 Decryption Module

Claims

1. 1. A multimodal machine learning based translation method executed by a computing device, the method comprising: Obtaining a semantic relationship diagram based on at least a first source statement and a second source statement among the n source statements belonging to different modalities, obtaining a first connection edge by full-mesh connection within each of the sets of semantic nodes of n different modal semantic nodes; identifying noun phrases in a first set of the n different modal semantic node sets using a predefined parser; identifying correspondences between the identified noun phrases and visual objects in a second set of the n cross-modal semantic nodes using a visual grounding toolkit and a pre-trained RCNN; combining one or more semantic nodes in the first set with one or more semantic nodes in the second set based on the identified correspondences to obtain second combining edges, the first set corresponding to the first source statement and the second set corresponding to the second source statement; Including, the first source statements are to be translated in text form and the second source statements are to be translated in non-text form; the semantic node is used to indicate one semantic unit of the source statement in one modal, where n is a positive integer greater than 1; From the semantic relationship diagram, Perform word embedding using a neural network model; performing word embedding by performing dimensionality reduction on the term co-occurrence matrix; Perform word embedding using a probabilistic model; performing word embedding for the words according to the semantics of the context in which the words are located; and at least one of defining for each of the first set a word embedding and a position embedding; projecting each visual feature of the second set into the same space as the first set using a multi-layer perceptron; extracting a plurality of first word vectors by: encoding the first plurality of word vectors to obtain n encoded feature vectors; and decoding the n encoded feature vectors to obtain a translated target statement. A multimodal machine learning based translation method.

2. The step of obtaining the first connecting edge comprises:

2. The method of claim 1, comprising the step of adding a first connection edge of an i-th type between any two semantic nodes in the same modal in an i-th set of semantic nodes, the first connection edge of the i-th type corresponding to the i-th modal, where i is a positive integer less than or equal to n.

3. The step of encoding the first word vectors to obtain n encoded feature vectors comprises:

3. The method of claim 1, further comprising: performing intra-modal fusion and inter-modal fusion e times on the plurality of first word vectors to obtain n encoded feature vectors, wherein the intra-modal fusion refers to performing semantic fusion between the first word vectors in the same modal, and the inter-modal fusion refers to performing semantic fusion between the first word vectors in a modal different from that of the first word vectors in a modal different from that of the first word vectors in the same modal, and e is a positive integer.

4. The multi-modal fusion encoder includes e encoding modules connected in series, Each of the encoding modules includes n intra-modal fusion layers and n inter-modal fusion layers, each of which corresponds to the n modalities in one-to-one correspondence; The step of performing e number of intra-modal fusion and inter-modal fusion on the plurality of first word vectors to obtain n number of encoded feature vectors includes: inputting the first word vectors into n intra-modal fusion layers in a first encoding module, respectively, and performing same-modal intra-modal semantic fusion on the first word vectors by the n intra-modal fusion layers to obtain n first hidden layer vectors, each of the first hidden layer vectors corresponding to a modal; inputting the n first hidden layer vectors into respective inter-modal fusion layers in the first encoding module, and performing semantic fusion between different modalities on the n first hidden layer vectors by the respective inter-modal fusion layers to obtain n first intermediate vectors, where one first intermediate vector corresponds to one modality; 4. The method of claim 3, further comprising: inputting the n first intermediate vectors to a j-th encoding module to perform a j-th encoding process, until a last encoding module outputs n encoded feature vectors, where one encoded feature vector corresponds to one modal, and j is a positive integer greater than 1 and less than or equal to e.

5. The method of claim 4 , wherein the hierarchical structure in each of the e serially connected encoding modules is the same.

6. The method of claim 4 , wherein different or the same self-attention functions are set for different intra-modal fusion layers, and different or the same feature fusion functions are set for different inter-modal fusion layers.

7. The step of decoding the n encoded feature vectors to obtain a translated target statement comprises: performing feature extraction on a first target phrase to obtain a second word vector, the first target phrase being a translated phrase in the target statement; performing feature extraction by combining the second word vector with the encoded feature vector to obtain a decoded feature vector; determining a probability distribution corresponding to the decoded feature vector; and determining a second target phrase after the first target phrase based on the probability distribution.

8. The decoder includes d decoding modules connected in series, where d is a positive integer, and each of the d decoding modules connected in series includes a first self-attention layer and a second self-attention layer; said step of combining said second word vector with said encoded feature vector to perform feature extraction to obtain a decoded feature vector comprises: inputting the second word vector into a first self-attention layer in a first decoding module, and performing feature extraction on the second word vector by the first self-attention layer to obtain a second hidden layer vector; inputting the second hidden layer vector and the encoded feature vector into a second self-attention layer in the first decoding module, and combining the second hidden layer vector and the encoded feature vector through the second self-attention layer to perform feature extraction to obtain a second intermediate vector; 8. The method of claim 7, further comprising: inputting the second intermediate vector into a kth decoding module to perform a kth decoding process, and continuing until a last decoding module outputs the decoded feature vector, where k is a positive integer greater than 1 and less than or equal to d.

9. A multimodal machine learning based translation device, comprising: A semantic association module used for constructing a semantic relationship diagram based on at least a first source statement and a second source statement among n source statements belonging to different modalities, The semantic association module: obtaining a first connection edge by full-mesh connection within each of the sets of semantic nodes of n different modal semantic nodes; identifying noun phrases in a first set of the n different modal semantic node sets using a parser; identifying correspondences between the identified noun phrases and visual objects in a second set of the n cross-modal semantic nodes using a visual grounding toolkit and a pre-trained RCNN; combining one or more semantic nodes in the first set with one or more semantic nodes in the second set based on the identified correspondences to obtain a second combining edge, wherein the first set corresponds to the first source statement and the second set corresponds to the second source statement. configured to run the first source statements are to be translated in text form and the second source statements are to be translated in non-text form; a semantic association module, wherein the semantic node is used to indicate one semantic unit of the source statement in one modal, where n is a positive integer greater than 1; From the semantic relationship diagram, Perform word embedding using a neural network model; performing word embedding by performing dimensionality reduction on the term co-occurrence matrix; Perform word embedding using a probabilistic model; performing word embedding for the words according to the semantics of the context in which the words are located; and at least one of defining for each of the first set a word embedding and a position embedding; projecting each visual feature of the second set into the same space as the first set using a multi-layer perceptron; a feature extraction module adapted to extract a plurality of first word vectors according to the a vector encoding module for encoding the first plurality of word vectors to obtain n encoded feature vectors; a vector decoding module, which is used for decoding the encoded feature vector to obtain a translated target statement; A translation device based on multimodal machine learning.

10. A computing device, the computing device comprising: Memory, a processor coupled to the memory, A computing device, wherein the processor is configured to implement the method for multimodal machine learning based translation according to any one of claims 1 to 8 by loading and executing executable instructions.

11. A computer program comprising at least one program segment, the at least one program segment being loaded and executed by a processor to implement the method for translation based on multimodal machine learning according to any one of claims 1 to 8.

Citation Information

Patent Citations

  • Method and system for generating input grammars for multi-modal dialog systems

    US20060123358A1

  • Exploiting the semantic web for unsupervised spoken language understanding

    US20140236570A1

  • Multi-lingual virtual personal assistant

    US20180314689A1

  • Encoder-decoder network with intercommunicating encoder agents

    US20190287012A1