Multimodal dialogue translation method and device, electronic equipment and storage medium

Abstract

The application provides a multi-modal dialogue translation method and device, electronic equipment and storage medium, and relates to the technical field of computers. After obtaining the dialogue text to be translated and the historical dialogue text based on the source language, the context relationship between the dialogue text to be translated and the historical dialogue text can be determined, the dialogue expression features of the speaker can be determined, at least one scene image corresponding to the dialogue text to be translated is extracted, the corresponding image feature set is obtained, the context relationship, the dialogue expression features and the image feature set are fused to obtain multi-modal dialogue features, and the multi-modal dialogue features are translated based on the target language to obtain the target translation dialogue text. Since the corresponding scene image is introduced and the effective image information is extracted from the scene image, more accurate target translation text can be obtained, and the obtained target translation text can better fit the current dialogue scene.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of computer technology, and in particular to a multimodal dialogue translation method, apparatus, electronic device and storage medium. Background Art

[0002] With the development of artificial intelligence technology, deep learning has achieved leap-forward development in natural language processing tasks. As international exchanges become increasingly close, large-scale cross-language communication scenarios are increasing, and the translation task volume in these scenarios is extremely large. The birth of machine translation technology provides a theoretical basis and foundation for solutions to similar scenarios.

[0003] In the related art, a Textual Chat Translation (TCT) model is usually adopted to perform dialogue translation tasks. In the process of performing dialogue translation, the TCT model obtains the translated text corresponding to the to-be-translated dialogue text according to the contextual relationship between the to-be-translated dialogue text and the historical dialogue text. However, an accurate translation result cannot be obtained only based on the contextual relationship between the to-be-translated dialogue text and the historical dialogue text. For example, if the to-be-translated dialogue text is "We're going to the bank soon" and the historical dialogue text is "Let's speed up", the translated text obtained based on the contextual relationship between the to-be-translated dialogue text and the historical dialogue text is "We are about to arrive at the bank", while the actual translated text is "We are about to arrive at the riverbank".

[0004] It can be seen that inaccurate translation results are easily obtained only according to the contextual relationship between the to-be-translated dialogue text and the historical dialogue text. Summary of the Invention

[0005] To solve the technical problems existing in the related art, the embodiments of the present application provide a multimodal dialogue translation method, apparatus, electronic device and storage medium, which can obtain more accurate dialogue translation results.

[0006] In one aspect, the embodiments of the present application provide a multimodal dialogue translation method, the method comprising:

[0007] Acquire a to-be-translated dialogue text and a historical dialogue text; wherein the historical dialogue text records the contextual dialogue content of the to-be-translated dialogue text, and both the to-be-translated dialogue text and the historical dialogue text are acquired based on a source language;

[0008] Determine the contextual relationship between the to-be-translated dialogue text and the historical dialogue text, as well as the dialogue expression feature of the speaking subject, based on the to-be-translated dialogue text and the historical dialogue text;

[0009] At least one scene image corresponding to the dialogue text to be translated is obtained, and features are extracted from the at least one scene image to obtain the corresponding image feature set;

[0010] The contextual relationships, the dialogue expression features, and the image feature set are fused to obtain multimodal dialogue features based on the source language;

[0011] The multimodal dialogue features are translated based on the target language to obtain the target translated dialogue text.

[0012] On one hand, embodiments of this application provide a multimodal dialogue translation device, the device comprising:

[0013] The text acquisition module is used to acquire the dialogue text to be translated and the historical dialogue text; wherein, the historical dialogue text records the contextual dialogue content of the dialogue text to be translated, and both the dialogue text to be translated and the historical dialogue text are acquired based on the source language;

[0014] The relationship determination module is used to determine the contextual relationship between the dialogue text to be translated and the historical dialogue text, as well as the dialogue expression characteristics of the speaker, based on the dialogue text to be translated and the historical dialogue text.

[0015] The feature extraction module is used to acquire at least one scene image corresponding to the dialogue text to be translated, and to extract features from the at least one scene image to obtain a corresponding image feature set.

[0016] The feature fusion module is used to fuse the context relationship, the dialogue expression features, and the image feature set to obtain multimodal dialogue features based on the source language;

[0017] The translation module is used to translate the multimodal dialogue features based on the target language to obtain the target translated dialogue text.

[0018] Optionally, the relationship determination module is specifically used for:

[0019] The dialogue text to be translated is vectorized to obtain the dialogue vector to be translated.

[0020] The historical dialogue text is vectorized to obtain the historical dialogue vector.

[0021] Based on a trained multimodal dialogue translation model, semantic relevance matching is performed between the dialogue vector to be translated and the historical dialogue vector to determine the contextual relationship;

[0022] Based on the multimodal dialogue translation model, a target historical dialogue vector with the same speaker as the dialogue vector to be translated is determined from the historical dialogue vector, and the dialogue expression features of the speaker are determined according to the dialogue vector to be translated and the target historical dialogue vector.

[0023] Optionally, the relationship determination module is further configured to:

[0024] Based on a trained multimodal dialogue translation model, at least one of the following is determined between the dialogue text to be translated and the historical dialogue text: dialogue topic consistency and dialogue coherence, and the determination result is used as the context relationship.

[0025] Optionally, the relationship determination module is further configured to:

[0026] Based on the dialogue vector to be translated and the target historical dialogue vector, at least one of the speaker's expression style, word usage habits, and emotional expression is determined, and the determination result is used as the speaker's dialogue expression feature.

[0027] Optionally, the feature fusion module is specifically used for:

[0028] Based on the trained multimodal dialogue translation model, the contextual relationships and the dialogue expression features are concatenated to obtain the dialogue text vector;

[0029] The dialogue text vector and the image feature set are respectively transformed into the same semantic space to obtain the corresponding target text vector and target image vector; the semantic space represents the expression form of the language;

[0030] The target text vector is matched with the target image vector to determine the similarity between the target text vector and the target image vector. Based on the similarity, the target image vector and the target text vector are weighted and merged to obtain multimodal dialogue features.

[0031] Optionally, the translation module is specifically used for:

[0032] Based on the trained multimodal dialogue translation model, the probability values ​​of the multimodal dialogue features corresponding to each preset word in the target language are determined.

[0033] Based on the probability values ​​corresponding to each preset word, a set of target words is selected;

[0034] Based on the target word set, the target translated dialogue text corresponding to the multimodal dialogue features is determined.

[0035] Optionally, the dialogue vector to be translated includes at least one of the following: word vector, word position vector, text turn vector, and role vector corresponding to the dialogue text to be translated; the historical dialogue vector includes at least one of the following: word vector, word position vector, text turn vector, and role vector corresponding to the historical dialogue text; wherein,

[0036] Each word vector represents the semantic information of each word in the corresponding text;

[0037] Each word position vector represents the position of each word in the corresponding text;

[0038] Each text round vector represents the order of the corresponding text in the semantically related text set;

[0039] Each role vector represents the identity attribute of the speaker in the corresponding text.

[0040] Optionally, the device further includes a model training module for:

[0041] Obtain a training dataset; the training dataset includes sample dialogue text and sample historical dialogue text corresponding to the source language, as well as sample real translation text of the sample dialogue text corresponding to the target language;

[0042] Based on the training dataset, the multimodal dialogue translation model is iteratively trained until the set convergence condition is met. One iteration of the training process includes:

[0043] The multimodal dialogue translation model determines the contextual relationship between the selected sample dialogue text and the corresponding sample historical dialogue text, as well as the sample dialogue expression features of the sample speaker.

[0044] Based on the contextual relationships and the expression features of the sample dialogue, the target translation dialogue text corresponding to the sample dialogue text is obtained, and the loss value determined based on the target translation dialogue text and the real translation text is used to adjust the parameters of the multimodal dialogue translation model.

[0045] Optionally, the sample dialogue text also corresponds to at least one sample scene image; the model training module is further used for:

[0046] The contextual relationship, the sample dialogue expression features, and the sample image feature set corresponding to the at least one sample scene image are fused to obtain sample multimodal dialogue features;

[0047] The multimodal dialogue features of the sample are translated to obtain the target translated dialogue text of the sample.

[0048] On one hand, embodiments of this application provide a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of the above-described multimodal dialogue translation method.

[0049] On one hand, embodiments of this application provide a computer-readable storage medium storing a computer program executable by a computer device, which, when run on the computer device, causes the computer device to perform the steps of the multimodal dialogue translation method described above.

[0050] On one hand, embodiments of this application provide a computer program product, which includes a computer program stored on a computer-readable storage medium. The computer program includes program instructions that, when executed by a computer, cause the computer to perform the steps of the above-described multimodal dialogue translation method.

[0051] The beneficial effects of this application are as follows:

[0052] This application provides a multimodal dialogue translation method, apparatus, electronic device, and storage medium. After obtaining the dialogue text to be translated and historical dialogue text, as well as at least one scene image corresponding to the dialogue text to be translated, based on the source language, the contextual relationship between the dialogue text to be translated and the historical dialogue text, and the dialogue expression features of the speakers can be determined. Feature extraction is performed on at least one scene image to obtain a corresponding image feature set. The contextual relationship, dialogue expression features, and image feature set are fused to obtain multimodal dialogue features. These multimodal dialogue features are then translated based on the target language to obtain the target translated dialogue text. Because corresponding scene images are introduced during the translation of the dialogue text to be translated, and effective image features are extracted from the scene images and fused with the contextual relationship between the dialogue text to be translated and the historical dialogue text, as well as the dialogue expression features of the speakers, a more accurate dialogue translation result can be obtained. Furthermore, the obtained dialogue translation result not only expresses the meaning of the source language but also better fits the current dialogue scenario.

[0053] Other features and advantages of this application will be set forth in the description which follows, and will be apparent in part from the description, or may be learned by practicing the application. The objectives and other advantages of this application may be realized and obtained by means of the structures particularly pointed out in the written description, claims, and drawings. Attached Figure Description

[0054] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0055] Figure 1a This application provides an illustration of an application scenario for a multimodal dialogue translation method.

[0056] Figure 1b An application scenario diagram for another multimodal dialogue translation method provided in the embodiments of this application;

[0057] Figure 2a A flowchart illustrating a multimodal dialogue translation method provided in this application embodiment;

[0058] Figure 2b A schematic diagram of a dialogue scenario provided in an embodiment of this application;

[0059] Figure 2c A schematic diagram illustrating a dialogue expression feature provided in an embodiment of this application;

[0060] Figure 2d A schematic diagram of a dialogue video clip provided in an embodiment of this application;

[0061] Figure 3a A flowchart illustrating the training process of a multimodal dialogue translation model provided in this application embodiment;

[0062] Figure 3b A schematic diagram illustrating another process for training a multimodal dialogue translation model provided in this application embodiment;

[0063] Figure 4a This is a schematic diagram of the structure of a multimodal dialogue translation model provided in an embodiment of this application;

[0064] Figure 4b A schematic diagram illustrating a vector transformation process provided in an embodiment of this application;

[0065] Figure 4c This is a schematic diagram of the structure of an encoding layer provided in an embodiment of this application;

[0066] Figure 4d This is a schematic diagram of the structure of a decoding layer provided in an embodiment of this application;

[0067] Figure 4e This application provides a schematic diagram illustrating the information fusion process in a text-image fusion sublayer.

[0068] Figure 5a A flowchart illustrating another multimodal dialogue translation method provided in this application embodiment;

[0069] Figure 5b A schematic diagram illustrating another dialogue expression feature provided in an embodiment of this application;

[0070] Figure 6 A schematic diagram of the structure of a multimodal dialogue translation device provided in this application embodiment;

[0071] Figure 7 A schematic diagram of another multimodal dialogue translation device provided in this application embodiment;

[0072] Figure 8 This is a schematic diagram of the structure of an electronic device according to an embodiment of this application;

[0073] Figure 9 This is a schematic diagram of the structure of another electronic device in an embodiment of this application. Detailed Implementation

[0074] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of this application will be clearly and completely described below with reference to the accompanying drawings of the embodiments of this application. Obviously, the described embodiments are only some embodiments of the technical solutions of this application, and not all embodiments. Based on the embodiments recorded in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the technical solutions of this application.

[0075] The terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such use of data can be interchanged where appropriate so that embodiments of the invention described herein can be implemented in sequences other than those illustrated or described herein.

[0076] The following explanations of some terms used in the embodiments of this application are provided to facilitate understanding by those skilled in the art.

[0077] Semantic space: the world of linguistic meaning. Generally speaking, information is a unity of meaning and symbol. The inherent meaning can only be expressed through certain external forms (actions, expressions, words, sounds, pictures, images, etc.). Therefore, every symbol system, in a broad sense, is a language that conveys meaning, and the meanings they express constitute a specific semantic space. The dissemination of information takes place both in social space and semantic space. A prerequisite for successful dissemination is that both the sender and receiver must share a common semantic space, that is, a shared understanding of the meaning of symbols or a common cultural background. Otherwise, the dissemination process itself cannot succeed; it may fail to communicate or lead to misunderstandings.

[0078] The word “exemplary” as used below means “serving as an example, embodiment, or illustration.” Any embodiment illustrated as an “exemplary” need not be construed as superior to or better than other embodiments.

[0079] The terms "first" and "second" used in this document are for descriptive purposes only and should not be construed as indicating relative importance or implying the number of technical features indicated. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of the embodiments of this application, unless otherwise stated, "multiple" means two or more.

[0080] The design concept of the embodiments of this application is briefly introduced below:

[0081] With the development of artificial intelligence technology, deep learning has made leaps and bounds in natural language processing tasks. As international exchanges become increasingly frequent, large-scale cross-language communication scenarios are becoming more common, resulting in enormous translation tasks. The emergence of machine translation technology provides a theoretical basis and foundation for solutions to such scenarios.

[0082] In related technologies, the TCT model is commonly used to perform dialogue translation tasks. The TCT model obtains the translated text corresponding to the dialogue text to be translated based on the contextual relationship between the dialogue text to be translated and the historical dialogue text. However, relying solely on the contextual relationship between the dialogue text to be translated and the historical dialogue text cannot yield an accurate translation result.

[0083] In view of this, embodiments of this application propose a multimodal dialogue translation method, apparatus, electronic device, and storage medium. In these embodiments, corresponding scene images are introduced during the translation of the dialogue text to be translated, and features are extracted from these scene images to obtain effective image features. These image features are then fused with the contextual relationship between the dialogue text to be translated and historical dialogue texts, as well as the dialogue expression features of the speakers. This results in a more accurate dialogue translation, which not only expresses the meaning of the source language but also better fits the current dialogue scenario.

[0084] The preferred embodiments of this application are described below with reference to the accompanying drawings. It should be understood that the preferred embodiments described herein are for illustration and explanation only and are not intended to limit this application. Furthermore, the embodiments and features in the embodiments of this application can be combined with each other without conflict.

[0085] See Figure 1a The diagram illustrates an application scenario in this embodiment. The application scenario includes a terminal device 100 and a server 200. The terminal device 100 and the server 200 can communicate via a communication network. Optionally, the communication network can be a wired network or a wireless network. The terminal device 100 and the server 200 can be directly or indirectly connected via wired or wireless communication, which is not limited herein.

[0086] In this embodiment, the terminal device 100 is an electronic device used by a user, which may be a personal computer, mobile phone, tablet computer, laptop, e-book reader, smart home device, vehicle terminal, etc. The server 200 may be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery network (CDN), and big data and artificial intelligence platforms.

[0087] For example, when a user receives a video in which the dialogue is in the source language and needs to translate it into the target language, they can send the video to the server 200 via terminal device 100. The server 200 can then obtain the dialogue text to be translated and the historical dialogue text corresponding to the source language in the video, determine the contextual relationship between the dialogue text to be translated and the historical dialogue text, as well as the dialogue expression characteristics of the speakers. It then extracts features from at least one scene image corresponding to the dialogue text to be translated, obtaining corresponding image feature sets. The server 200 then fuses the contextual relationship, dialogue expression features, and image feature sets to obtain multimodal dialogue features. Finally, it translates the multimodal dialogue features to obtain the target translated dialogue text in the target language. After translating the dialogue in the video from the source language to the target language, the server 200 can send the video to the terminal device 100 so that the user can watch the video.

[0088] In one embodiment, such as Figure 1b As shown, User A and User B are engaged in a dialogue. User A uses the source language, while User B uses the target language. Both users communicate with each other in their respective languages. Terminal device 100 can acquire the dialogue between User A and User B in real time, as well as capture images of User A and User B, and send these data to server 200. Server 200 can translate User A's dialogue and image to obtain translated dialogue A in the target language, and translate User B's dialogue and image to obtain translated dialogue B in the target language. Server 200 then sends both translated dialogue A and translated dialogue B to terminal device 100, enabling barrier-free communication between User A and User B.

[0089] It should be noted that, Figure 1a and Figure 1b This is an example of the application scenarios of the multimodal dialogue translation method of this application. However, the application scenarios that the method in the actual embodiments of this application can be applied to are not limited to this.

[0090] To further illustrate the technical solutions provided in the embodiments of this application, a detailed description is provided below in conjunction with the accompanying drawings and specific implementation methods. Although the embodiments of this application provide method operation steps as shown in the following embodiments or drawings, the method may include more or fewer operation steps based on conventional or non-inventive methods. In steps where there is no logically necessary causal relationship, the execution order of these steps is not limited to the execution order provided in the embodiments of this application. In actual processing or when the device executes the method, it may be executed sequentially or in parallel according to the method shown in the embodiments or drawings.

[0091] Figure 2a The flowchart shown is a multimodal dialogue translation method provided in an embodiment of this application. This method can be executed by a server, for example... Figure 1a Server 200 in the middle can also be executed by terminal devices, for example Figure 1a Terminal device 100. The following is in conjunction with the appendix. Figure 2a The multimodal dialogue translation process in the embodiments of this application is described in detail.

[0092] Step S201: Obtain the dialogue text to be translated and the historical dialogue text.

[0093] The historical dialogue text records the contextual dialogue content of the dialogue text to be translated, and both the dialogue text to be translated and the historical dialogue text are obtained based on the source language. It is understood that the source language and the target language in this embodiment are different languages, specifically Chinese, English, French, Spanish, Russian, Arabic, etc.

[0094] For example, if the source language is Chinese, the target language can be English; if the source language is English, the target language can be French; if the source language is Spanish, the target language can be Chinese.

[0095] The dialogue text to be translated is the text generated during a bilingual dialogue that requires translation. The historical dialogue text is the text generated before the dialogue text to be translated during the bilingual dialogue. Both the historical dialogue text and the dialogue text to be translated are in the same language, derived from the source language. The historical dialogue text can include directly generated dialogue text based on the source language, or it can include translated dialogue text based on the target language, resulting in a translated dialogue text based on the source language.

[0096] In one embodiment, after obtaining the current dialogue text of the source language speaker, the current dialogue text can be used as the dialogue text to be translated, and the historical dialogue text corresponding to the dialogue text to be translated can be obtained.

[0097] For example, a dialogue video can be obtained, showing a conversation between speaker A and speaker B in the source language. Speaker A's dialogue text includes X1, X2, X3, X4, and X5, while speaker B's dialogue text includes Y1, Y2, Y3, and Y4. The dialogue scenario between speaker A and speaker B can be described as follows: Figure 2b As shown.

[0098] Assuming that X5 in the dialogue text of speaker A is taken as the to-be-translated dialogue text, all of X1, X2, X3, X4, Y1, Y2, Y3 and Y4 can be taken as historical dialogue texts.

[0099] Optionally, if X5 in the dialogue text of speaker A is taken as the to-be-translated dialogue text, only X1, X2, X3 and X4 may be taken as historical dialogue texts.

[0100] Step S202: Based on the to-be-translated dialogue text and the historical dialogue texts, a context relationship between the to-be-translated dialogue text and the historical dialogue texts, and a dialogue expression feature of the speaker are determined.

[0101] The context relationship is used to represent at least one of dialogue topic consistency and dialogue coherence between the to-be-translated dialogue text and the historical dialogue texts. The context relationship refers to the preceding and following relationship within a word, phrase, sentence, paragraph or text, and is used to help understand the specific meaning of language components such as words, phrases and sentences.

[0102] In the embodiments of the present application, the context relationship between the to-be-translated dialogue text and the historical dialogue texts refers to the preceding and following relationship between words, phrases and sentences in the to-be-translated dialogue text and words, phrases and sentences in the historical dialogue texts. It can be understood that when performing dialogue translation, the use of the context relationship between the to-be-translated dialogue text and the historical dialogue texts can help understand the specific meaning of language components such as words, phrases and sentences in the to-be-translated dialogue text, so that more accurate translation can be performed on the to-be-translated dialogue text.

[0103] The dialogue expression feature is used to represent the dialogue characteristic of the speaker who sends the to-be-translated dialogue text, that is, the language expression characteristic of the speaker of the source language during language expression, which may specifically be at least one of expression style, word usage habit and expression emotion.

[0104] For example, as Figure 2c shown, in terms of word usage habits, for the synonyms "teng" (ache) and "tong" (pain), some speakers are accustomed to using "teng", such as "my head tengs badly", while some other speakers are accustomed to using "tong", such as "my head tongs badly"; in terms of expression style, for expressing the same meaning, some speakers will use "the rain outside is very heavy" to describe that it is raining heavily outside, while some other speakers will use "outside the rain is very heavy" to describe that it is raining heavily outside; in terms of expression emotion, for describing the same thing that happened, some speakers will use "a major event happened there", while some other speakers will use "a super huge event happened there".

[0105] In one embodiment, after obtaining the dialogue text to be translated and the historical dialogue text, the dialogue text to be translated can be vectorized to obtain the dialogue vector to be translated, and the historical dialogue text can be vectorized to obtain the historical dialogue vector. Then, the dialogue vector to be translated and the historical dialogue vector are input into a trained multimodal dialogue translation model. The multimodal dialogue translation model performs semantic relevance matching on the dialogue vector to be translated and the historical dialogue vector to determine the contextual relationship between the dialogue text to be translated and the historical dialogue text, and determines the target historical dialogue vector with the same speaker as the dialogue vector to be translated from the historical dialogue vector. Based on the dialogue vector to be translated and the target historical dialogue vector, the dialogue expression features of the speaker are determined.

[0106] The dialogue vector to be translated includes at least one of the following: word vectors, word position vectors, text turn vectors, and role vectors corresponding to the dialogue text to be translated. The historical dialogue vector includes at least one of the following: word vectors, word position vectors, text turn vectors, and role vectors corresponding to the historical dialogue text. Each word vector represents the semantic information of each word in the corresponding text, each word position vector represents the arrangement position of each word in the corresponding text, each text turn vector represents the order of the corresponding text in the semantically related text set, and each role vector represents the identity attribute of the speaker in the corresponding text.

[0107] Step S203: Obtain at least one scene image corresponding to the dialogue text to be translated, and extract features from each scene image to obtain the corresponding image feature set.

[0108] The dialogue text to be translated can correspond to at least one scene image. For each scene image, the following operations can be performed:

[0109] Feature extraction is performed on the scene image to obtain the image features corresponding to the scene image.

[0110] Based on the image features corresponding to at least one scene image, a corresponding set of image features can be obtained.

[0111] Step S204: The contextual relationships, dialogue expression features, and image feature sets are fused to obtain multimodal dialogue features based on the source language.

[0112] A multimodal dialogue translation model can concatenate contextual relationships and dialogue expression features to obtain dialogue text vectors. These text vectors and image feature sets are then transformed into the same semantic space to obtain the corresponding target text vectors and target image vectors. The semantic space represents the expressive form of the language.

[0113] Then, the target text vector is matched with the target image vector to determine the similarity between the target text vector and the target image vector. Based on the similarity, the target image vector and the target text vector are weighted and merged to obtain multimodal dialogue features.

[0114] Step S205: Translate the multimodal dialogue features based on the target language to obtain the target translated dialogue text.

[0115] By using a multimodal dialogue translation model, the probability values ​​of the multimodal dialogue features corresponding to each preset word in the target language are determined. Based on the probability values ​​of each preset word, a set of target words is selected. Based on the set of target words, the target translation dialogue text corresponding to the multimodal dialogue features is determined.

[0116] The multimodal dialogue translation method proposed in this application effectively incorporates image information during the translation process, thereby enabling the target translated dialogue text to not only accurately express the meaning of the source language but also better fit the current dialogue scenario.

[0117] For example, Figure 2d For a video clip of a conversation, such as Figure 2d As shown, the dialogue video clip contains at least three dialogue texts X1, X2, and X3, and each dialogue text corresponds to a scene image. If dialogue text X1 is taken as the dialogue text to be translated, then at least dialogue texts X2 and X3 can be taken as the corresponding historical dialogue texts. If dialogue text X2 is taken as the dialogue text to be translated, then at least dialogue texts X1 and X3 can be taken as the corresponding historical dialogue texts. If dialogue text X3 is taken as the dialogue text to be translated, then at least dialogue texts X1 and X2 can be taken as the corresponding historical dialogue texts.

[0118] Assuming the dialogue text X1 is taken as the dialogue text to be translated, image features related to pants can be extracted from the scene image corresponding to dialogue text X1. By combining the dialogue text X1 and the image features related to pants in the scene image, the dialogue text X1: Where you think you're going with those? can be accurately translated into the corresponding target translation dialogue text Y1: Where are you going with these pants?

[0119] Specifically, the training process of a multimodal dialogue translation model can be as follows: Figure 3a As shown below, in conjunction with Figure 3a The training process of the multimodal dialogue translation model in the embodiments of this application is described in detail.

[0120] Step S301: Obtain the training dataset.

[0121] The acquired training dataset may include sample dialogue texts and sample historical dialogue texts corresponding to the source language, as well as sample real translation texts of the target language corresponding to the sample dialogue texts.

[0122] Step S302: Extract sample dialogue text and corresponding sample historical dialogue text from the training dataset.

[0123] When training a multimodal dialogue translation model, sample dialogue texts and corresponding historical dialogue texts can be extracted from the training dataset as training sample data.

[0124] Step S303: Input the extracted sample dialogue text and sample historical dialogue text into the multimodal dialogue translation model. Through the multimodal dialogue translation model, determine the contextual relationship between the sample dialogue text and the sample historical dialogue text, as well as the sample dialogue expression features of the sample speakers.

[0125] The extracted sample dialogue text and sample historical dialogue text are input into the multimodal dialogue translation model. The multimodal dialogue translation model performs semantic correlation matching on the sample dialogue vector corresponding to the sample dialogue text and the sample historical dialogue vector corresponding to the sample historical dialogue text to determine the contextual relationship between the sample dialogue text and the sample historical dialogue text.

[0126] Furthermore, through a multimodal dialogue translation model, the target historical dialogue vector with the same speaker as the sample dialogue vector is determined from the sample historical dialogue vector, and the sample dialogue expression features of the speaker are determined based on the sample dialogue vector and the target historical dialogue vector.

[0127] Step S304: Based on the context and the expression features of the sample dialogue, obtain the target translation dialogue text corresponding to the sample dialogue text.

[0128] Based on the contextual relationship between the sample dialogue text and the sample historical dialogue text, as well as the sample dialogue expression characteristics of the sample speakers, the sample dialogue text is translated to obtain the sample target translation dialogue text corresponding to the target language.

[0129] Step S305: Determine the corresponding loss value based on the target translation dialogue text and the actual translation text of the sample.

[0130] When calculating the loss value, a pre-defined loss function can be used. This function can be the cross-entropy loss function, such as the sigmoid function. Other loss functions used include, but are not limited to, multi-class cross-entropy loss, contrastive loss (related to metric learning), or triplet loss. Generally, the loss value determines how closely the actual output matches the expected output. A smaller loss value indicates that the actual output is closer to the expected output.

[0131] Step S306: Determine whether the loss value has converged to the preset target value; if not, proceed to step S307; if yes, proceed to step S308.

[0132] To determine whether the loss value has converged to the preset target value, if the loss value is less than or equal to the preset target value, or if the change in the loss value obtained from N consecutive training iterations is less than or equal to the preset target value, then the loss value is considered to have converged to the preset target value, indicating that the loss value has converged; otherwise, it indicates that the loss value has not yet converged.

[0133] Step S307: Adjust the parameters of the multimodal dialogue translation model based on the determined loss value.

[0134] If the loss value does not converge, the model parameters are adjusted. After adjusting the model parameters, the process returns to step S302 to continue the next round of training.

[0135] Step S308: End training to obtain the trained multimodal dialogue translation model.

[0136] If the loss value converges, the current multimodal dialogue translation model is used as the trained multimodal dialogue translation model.

[0137] The sample dialogue texts in the training dataset also correspond to at least one sample scene image. Feature extraction can be performed on each of these at least one scene image to obtain the corresponding image feature set. After determining the image feature set corresponding to at least one sample scene image, the specific process in step S304 above, which involves obtaining the sample target translation dialogue text based on contextual relationships and sample dialogue expression features, can also be as follows: Figure 3b As shown, it includes the following steps:

[0138] Step S3041: The contextual relationships, sample dialogue expression features, and sample image feature set are fused to obtain sample multimodal dialogue features.

[0139] Specifically, after obtaining the contextual relationship between the sample dialogue text and the sample historical dialogue text, as well as the sample dialogue expression features of the sample speaker, the contextual relationship and the sample dialogue expression features can be concatenated to obtain the sample dialogue text vector.

[0140] The sample dialogue text vectors and sample image feature sets are transformed into the same semantic space to obtain the corresponding sample target text vectors and sample target image vectors. The sample target text vectors and sample target image vectors are matched to determine the similarity between them. Based on the similarity, the sample target image vectors and sample target text vectors are weighted and merged to obtain the sample multimodal dialogue features.

[0141] Step S3042: Translate the multimodal dialogue features of the sample to obtain the target translated dialogue text of the sample.

[0142] The probability values ​​of each sample preset word in the target language corresponding to the sample multimodal dialogue features are determined. Based on the probability values ​​corresponding to each sample preset word, a sample target word set is selected. Based on the sample target word set, the sample target translation dialogue text corresponding to the sample multimodal dialogue features is determined.

[0143] In some embodiments, the structural diagram of the multimodal dialogue translation model provided in this application can be as follows: Figure 4a As shown. Figure 4a As shown, the dialogue translation model may include a word embedding layer, an encoder, and a decoder. The encoder may include N encoding layers, and the decoder may include N decoding layers.

[0144] After obtaining the dialogue text to be translated and the historical dialogue text corresponding to the source language, the dialogue text to be translated and the historical dialogue text can be input into the word embedding layer respectively. Based on the word embedding layer, the dialogue text to be translated can be vectorized to obtain the dialogue vector to be translated, and the historical dialogue text can be vectorized to obtain the historical dialogue vector.

[0145] Specifically, after inputting the dialogue text to be translated and the historical dialogue text into the word embedding layer, the word embedding layer performs vector transformation on the dialogue text to be translated and the historical dialogue text as follows: Figure 4b As shown, the word embedding layer, when performing vector transformation on the dialogue text to be translated and the historical dialogue text, can obtain the word vectors, word position vectors, text turn vectors, and role vectors corresponding to the dialogue text to be translated and the historical dialogue text, respectively. Then, it adds the word vectors, word position vectors, text turn vectors, and role vectors to obtain the corresponding dialogue vectors to be translated and historical dialogue vectors. Among these, Figure 4bThe ⊕ symbol in the diagram represents addition. By adding the word vector, word position vector, text turn vector, and role vector together, the corresponding dialogue vector can be obtained.

[0146] Each word vector represents the semantic information of each word in the corresponding text, each word position vector represents the arrangement position of each word in the corresponding text, each text round vector represents the order of the corresponding text in the semantically related text set, and each role vector represents the identity attribute of the speaker in the corresponding text.

[0147] After obtaining the dialogue vector to be translated corresponding to the dialogue text to be translated and the historical dialogue vector corresponding to the historical dialogue text through the word embedding layer, the dialogue vector to be translated and the historical dialogue vector can be concatenated to obtain the corresponding concatenated vector.

[0148] like Figure 4c As shown, each encoding layer can include a multi-head attention sublayer, a feed-forward network (FFN) sublayer, and a text-vision fusion sublayer. The multi-head attention sublayer performs self-attention processing on the data input to that sublayer, the feed-forward network sublayer performs linear transformation on the data input to that sublayer, and the text-vision fusion sublayer fuses the image and text features input to that sublayer.

[0149] Attention refers to the ability of a person's mental activity to be directed and focused on a certain thing. The attention mechanism is based on the human visual attention mechanism. Simply put, the visual attention mechanism is that when people observe an image, they first notice a certain part of the image, rather than browsing the entire content, and then adjust the focus of their attention during the observation process. Therefore, the attention mechanism in the field of neural networks is derived from this principle.

[0150] Specifically, after obtaining the concatenated vector containing the contextual relationship between the dialogue text to be translated and the historical dialogue text, as well as the dialogue expression features of the speaker, the concatenated vector can be input into the encoder. The multi-head attention sublayer in the encoder can perform self-attention processing on the concatenated vector to obtain the corresponding hidden state vector. The feedforward network sublayer in the encoder can perform linear transformation on the hidden state vector to obtain the corresponding dialogue text vector. The image-text fusion sublayer in the encoder can fuse the dialogue text vector with the corresponding image feature set to obtain multimodal dialogue features.

[0151] like Figure 4dAs shown, each decoding layer can include a masked multi-head attention sublayer, a multi-head attention sublayer, and a feed-forward network (FFN) sublayer. The masked multi-head attention sublayer performs masking on the input data, the multi-head attention sublayer performs self-attention on the input data, and the feed-forward network sublayer performs linear transformation on the input data.

[0152] Specifically, after obtaining the multimodal dialogue features, these features can be input into the decoder. The mask multi-head attention sublayer in the decoder can perform mask occlusion processing on other features after the target feature in the multimodal dialogue features to obtain the corresponding mask hidden state vector. The multi-head attention sublayer in the decoder can perform self-attention processing on the mask hidden state vector to obtain the corresponding hidden state vector. The feedforward network sublayer in the decoder can perform linear transformation on the hidden state vector to obtain the corresponding target translated dialogue text.

[0153] Optionally, first obtain the dialogue text to be translated and the historical dialogue text corresponding to the source language. Then, input the dialogue text to be translated and the historical dialogue text into the word embedding layer to obtain the corresponding token representation, i.e., the vectorized representation, and denote the obtained vector as X. text .

[0154] At least one scene image corresponding to the dialogue text to be translated is obtained. Then, the external tool Faster R-CNN is used to extract features from each scene image to obtain the object information corresponding to the scene image, i.e., image features. The resulting set of image features is denoted as X. img .

[0155] X text The input is fed into the encoder, where a standard Multi-Head Attention operation is first performed, namely:

[0156] Z t =FFN(MultiHead(X) text ,X text ,X text ))

[0157] After obtaining the dialogue text vector Z t Then, the dialogue text vector Z can be... t and image feature set X img The input is fed into the image-text fusion sublayer in the encoder. For example... Figure 4e The image shows the text fusion sublayer that handles the dialogue text vector Z. tand image feature set X img The process of information fusion. Among them, Z... v For image feature set X img Z′ t These are the multimodal dialogue features obtained after fusion. (From...) Figure 4e It can be seen that in the image-text fusion sublayer, the image feature set Z is first processed. v After performing a linear transformation, it is then compared with the dialogue text vector Z. t Cross-modal dot-product attention is applied to obtain the multimodal dialogue features Z′. t .

[0158] Specifically, the image-text fusion sublayer will convert the dialogue text vector Z... t and image feature set Z v The fusion process can be illustrated by the following formula:

[0159]

[0160]

[0161] Z′ t =Concat(Z) t ,AZ v W2

[0162] Where W1 and W2 are matrices, and the Softmax function is used to determine the dialogue text vector Z. t and image feature set Z v The similarity between them is used by the Concat function to calculate the similarity between the dialogue text vectors Z and Z. t and image feature set Z v We perform weighted merging to obtain multimodal dialogue features.

[0163] The obtained multimodal dialogue features are input into the decoder to generate the t-th word y. t At that time, the first part was the historical section y 1：t-1 Encoding learning using self-attention mechanisms:

[0164] H y =MultiHead(y,y,y)

[0165] Where y is y 1：t-1 The word embedding vector representation.

[0166] Subsequently, the encoder representation Z is obtained through another self-attention mechanism. t ′ and content reply indicate H y The interactive representation, namely:

[0167]

[0168] Finally, decoding it yields the final target translated dialogue text, namely:

[0169]

[0170] Among them, W o and b o These are the parameters to be learned.

[0171] Furthermore, the loss function for training a multimodal dialogue translation model can be expressed as follows:

[0172]

[0173] In the process of training a multimodal dialogue translation model, the model can first be initialized using mBART weights, and then the model can be trained based on the loss function L mentioned above.

[0174] Optionally, the MultiHead in the embodiments of this application can also be replaced by a single-head self-attention mechanism.

[0175] In some embodiments, the multimodal dialogue translation method proposed in this application can also be implemented according to... Figure 5a The process shown can be implemented by... Figure 1a The server 200 can execute the commands, or it can be executed by a terminal device or other electronic device. For example... Figure 5a As shown, it may include the following steps:

[0176] Step S501: Obtain the dialogue text to be translated and the historical dialogue text corresponding to the source language.

[0177] Step S502: Perform vector transformation on the dialogue text to be translated to obtain the dialogue vector.

[0178] The dialogue vector to be translated includes at least one of the following: word vectors, word position vectors, text turn vectors, and role vectors corresponding to the dialogue text to be translated. Each word vector represents the semantic information of each word in the corresponding text, each word position vector represents the arrangement position of each word in the corresponding text, each text turn vector represents the order of the corresponding text in the semantically related text set, and each role vector represents the identity attribute of the speaker in the corresponding text.

[0179] Step S503: Perform vector transformation on the historical dialogue text to obtain the historical dialogue vector.

[0180] The historical dialogue vector includes at least one of the following: word vectors, word position vectors, text round vectors, and role vectors corresponding to the historical dialogue text. Each word vector represents the semantic information of each word in the corresponding text, each word position vector represents the arrangement position of each word in the corresponding text, each text round vector represents the order of the corresponding text in the semantically related text set, and each role vector represents the identity attribute of the speaker in the corresponding text.

[0181] Step S504: Concatenate the dialogue vector to be translated and the historical dialogue vector to obtain the concatenated vector.

[0182] Step S505: Input the concatenated vector into the encoder of the multimodal dialogue translation model to obtain the dialogue text vector.

[0183] Specifically, the encoder can perform semantic relevance matching between the dialogue vector to be translated and the historical dialogue vector to determine the contextual relationship between the dialogue text to be translated and the historical dialogue text. The contextual relationship can include at least one of dialogue topic consistency and dialogue coherence.

[0184] The encoder can identify a target historical dialogue vector from the historical dialogue vectors that has the same speaker as the dialogue vector to be translated, and determine the dialogue expression features of the speaker based on the dialogue vector to be translated and the target historical dialogue vector. These dialogue expression features can include at least one of expression style, word choice habits, and emotional expression.

[0185] In other words, the encoder can obtain a dialogue text vector that includes contextual relationships and dialogue expression features by encoding the concatenated vector.

[0186] Step S506: Obtain at least one scene image corresponding to the dialogue text to be translated, and extract features from each scene image to obtain the corresponding image feature set.

[0187] Step S507: The dialogue text vector and image feature set are fused by the encoder to obtain multimodal dialogue features.

[0188] The encoder can first transform the dialogue text vector and the image feature set into the same semantic space to obtain the corresponding target text vector and target image vector. Then, it matches the target text vector and the target image vector to determine the similarity between the target text vector and the target image vector. Based on the similarity, it performs a weighted merging of the target image vector and the target text vector to obtain multimodal dialogue features.

[0189] Step S508: Input the multimodal dialogue features into the decoder of the multimodal dialogue translation model, and obtain the target translated dialogue text corresponding to the target language through the decoder.

[0190] The decoder can determine the probability values ​​of each preset word in the target language corresponding to the multimodal dialogue features. Based on the probability values ​​of each preset word, a set of target words is selected. Finally, based on the set of target words, the target translated dialogue text corresponding to the multimodal dialogue features is determined.

[0191] The multimodal dialogue translation method proposed in this application, after obtaining the dialogue text to be translated and the historical dialogue text corresponding to the source language, as well as at least one scene image corresponding to the dialogue text to be translated, can determine the contextual relationship between the dialogue text to be translated and the historical dialogue text, as well as the dialogue expression features of the speakers. It then extracts features from each of the at least one scene image to obtain corresponding image feature sets. Furthermore, it fuses the contextual relationship, dialogue expression features, and image feature sets to obtain multimodal dialogue features, and translates these multimodal dialogue features to obtain the target translated dialogue text corresponding to the target language. This not only fully utilizes existing multilingual pre-trained models but also fully models image and text information, learning the most useful information for the current dialogue text to be translated from dialogue history and image information. This results in a more accurate target translated text that better fits the current dialogue scenario, is more practical, and significantly improves the user experience.

[0192] See Figure 5b As shown below, a specific application scenario will be used to further illustrate the above embodiments in detail:

[0193] Figure 5b A video clip containing dialogues has at least seven dialogue texts: X1, X2, X3, X4, X5, X6, and X7. Each dialogue text corresponds to a scene image. Assuming dialogue text X5 is taken as the dialogue text to be translated, then dialogue texts X1, X2, X3, X4, X6, and X7 can be taken as the corresponding historical dialogue texts.

[0194] Based on the historical dialogue texts X1, X2, X3, X4, X6, X7 and the dialogue text to be translated X5, it can be determined that the entire scene dialogue is related to a ship, and the relevant dialogue takes place on the ship. By extracting features from the scene image corresponding to the dialogue text to be translated X5 and fusing it with the dialogue text, image features related to the sails can be extracted.

[0195] Based on the scene image corresponding to the dialogue text X5 to be translated and the historical dialogue texts X1, X2, X3, X4, X6, and X7, the dialogue text X5 to be translated, "Mr. Lawrence, hold our course," can be accurately translated into the corresponding target translation dialogue text Y5: "Mr. Lawrence, keep our course."

[0196] Optionally, the multimodal dialogue translation method proposed in this application can be compared with multimodal chat translation (MCT) methods in related technologies. The comparison results are shown in Table 1 below:

[0197] Table 1

[0198] MCT 26.54 22.00 VG-MCT 27.83 23.21

[0199] Among them, VG-MCT is the multimodal dialogue translation method proposed in this application, En-Zh represents the translation from English to Chinese, Zh-En represents the translation from Chinese to English, and BLEU value is used to measure the accuracy of the translation. The higher the BLEU value, the more accurate the translation.

[0200] As can be seen from Table 1, the VG-MCT proposed in this application has higher BLEU values ​​in the two language directions (En->Zh, Zh->En) than the existing MCT model architecture, and the VG-MCT proposed in this application has higher translation accuracy.

[0201] and Figure 2a The multimodal dialogue translation method shown is based on the same inventive concept. This application also provides a multimodal dialogue translation device, which can be deployed in a server or terminal device. Since this device corresponds to the multimodal dialogue translation method of this application, and the principle by which this device solves the problem is similar to that of the method, the implementation of this device can refer to the implementation of the above method; repeated details will not be elaborated further.

[0202] Figure 6 This illustration shows a structural schematic diagram of a multimodal dialogue translation device provided in an embodiment of this application, as shown below. Figure 6 As shown, the multimodal dialogue translation device includes a text acquisition module 601, a relationship determination module 602, a feature extraction module 603, a feature fusion module 604, and a translation module 605.

[0203] The text acquisition module 601 is used to acquire the dialogue text to be translated and the historical dialogue text. The historical dialogue text records the contextual dialogue content of the dialogue text to be translated, and both the dialogue text to be translated and the historical dialogue text are acquired based on the source language.

[0204] The relationship determination module 602 is used to determine the contextual relationship between the dialogue text to be translated and the historical dialogue text, as well as the dialogue expression characteristics of the speaker, based on the dialogue text to be translated and the historical dialogue text.

[0205] The feature extraction module 603 is used to acquire at least one scene image corresponding to the dialogue text to be translated, and to extract features from the at least one scene image to obtain the corresponding image feature set.

[0206] The feature fusion module 604 is used to fuse contextual relationships, dialogue expression features, and image feature sets to obtain multimodal dialogue features based on the source language;

[0207] Translation module 605 is used to translate multimodal dialogue features based on the target language to obtain the target translated dialogue text.

[0208] Optionally, the relationship determination module 602 is specifically used for:

[0209] The dialogue text to be translated is transformed into a vector to obtain the dialogue vector.

[0210] Perform vector transformation on the historical dialogue text to obtain the historical dialogue vector;

[0211] Based on the trained multimodal dialogue translation model, semantic relevance matching is performed between the dialogue vector to be translated and the historical dialogue vector to determine the contextual relationship;

[0212] Based on a multimodal dialogue translation model, a target historical dialogue vector with the same speaker as the dialogue vector to be translated is identified from the historical dialogue vectors. Then, based on the dialogue vector to be translated and the target historical dialogue vector, the dialogue expression features of the speaker are determined.

[0213] Optionally, the relationship determination module 602 is also used for:

[0214] Based on a trained multimodal dialogue translation model, at least one of the following is determined between the dialogue text to be translated and the historical dialogue text: topic consistency and dialogue coherence. The determined result is used as the context relationship.

[0215] Optionally, the relationship determination module 602 is also used for:

[0216] Based on the dialogue vector to be translated and the target historical dialogue vector, at least one of the speaker's expression style, word usage habits, and emotional expression is determined, and the determined result is used as the speaker's dialogue expression feature.

[0217] Optionally, the feature fusion module 604 is specifically used for:

[0218] Based on the trained multimodal dialogue translation model, the contextual relationships and dialogue expression features are concatenated to obtain the dialogue text vector;

[0219] The dialogue text vectors and image feature sets are transformed into the same semantic space to obtain the corresponding target text vectors and target image vectors; the semantic space represents the expression form of the language.

[0220] The target text vector is matched with the target image vector to determine the similarity between them. Based on the similarity, the target image vector and the target text vector are weighted and merged to obtain multimodal dialogue features.

[0221] Optionally, translation module 605 is specifically used for:

[0222] Based on the trained multimodal dialogue translation model, the probability values ​​of the multimodal dialogue features corresponding to each preset word in the target language are determined.

[0223] The target word set is selected based on the probability value of each preset word.

[0224] Based on the target word set, the target translated dialogue text corresponding to the multimodal dialogue features is determined.

[0225] Optionally, the dialogue vector to be translated includes at least one of the following: word vector, word position vector, text turn vector, and role vector corresponding to the dialogue text to be translated; the historical dialogue vector includes at least one of the following: word vector, word position vector, text turn vector, and role vector corresponding to the historical dialogue text.

[0226] in,

[0227] Each word vector represents the semantic information of each word in the corresponding text;

[0228] Each word position vector represents the position of each word in the corresponding text;

[0229] Each text round vector represents the order of the corresponding text in the semantically related text set;

[0230] Each role vector represents the identity attribute of the speaker in the corresponding text.

[0231] Optionally, such as Figure 7 As shown, the above-mentioned device may further include a model training module 701, used for:

[0232] Obtain the training dataset; the training dataset includes sample dialogue texts and sample historical dialogue texts corresponding to the source language, as well as sample real translation texts of the sample dialogue texts corresponding to the target language;

[0233] Based on the training dataset, the multimodal dialogue translation model is iteratively trained until the set convergence condition is met. One iteration of the training process includes:

[0234] By using a multimodal dialogue translation model, the contextual relationship between the selected sample dialogue text and the corresponding sample historical dialogue text, as well as the sample dialogue expression features of the sample speakers, are determined.

[0235] Based on the context and the expressive features of the sample dialogue, the target translation dialogue text corresponding to the sample dialogue text is obtained, and the loss value determined based on the target translation dialogue text and the sample real translation text is used to adjust the parameters of the multimodal dialogue translation model.

[0236] Optionally, the sample dialogue text also corresponds to at least one sample scene image; the model training module 701 is further used for:

[0237] By fusing the contextual relationships, the sample dialogue expression features, and the sample image feature set corresponding to at least one sample scene image, the multimodal dialogue features of the samples are obtained.

[0238] The multimodal dialogue features of the samples are translated to obtain the target translated dialogue text of the samples.

[0239] In one embodiment, the electronic device may be a server, such as... Figure 1a The server 200 is shown. In this embodiment, the structure of the electronic device can be as follows: Figure 8 As shown, it includes a memory 801, a communication module 803, and one or more processors 802.

[0240] The memory 801 is used to store computer programs executed by the processor 802. The memory 801 may mainly include a program storage area and a data storage area. The program storage area may store the operating system and programs required to run instant messaging functions, etc.; the data storage area may store various instant messaging information and operation instruction sets, etc.

[0241] Memory 801 may be volatile memory, such as random-access memory (RAM); memory 801 may also be non-volatile memory, such as read-only memory, flash memory, hard disk drive (HDD), or solid-state drive (SSD); or memory 801 may be any other medium capable of carrying or storing desired program code in the form of instructions or data structures and accessible by a computer, but is not limited thereto. Memory 801 may be a combination of the above-described memories.

[0242] Processor 802 may include one or more central processing units (CPUs) or digital processing units, etc. Processor 802 is used to implement the above-described multimodal dialogue translation method when calling computer programs stored in memory 801.

[0243] The communication module 803 is used to communicate with terminal devices and other electronic devices. If the electronic device is a server, the server can receive the dialogue text to be translated and the historical dialogue text sent by the terminal device through the communication module 803.

[0244] This application does not limit the specific connection medium between the memory 801, the communication module 803, and the processor 802 described above. This disclosure embodiment... Figure 8 The memory 801 and the processor 802 are connected via a bus 804, and the bus 804 is in Figure 8 The connections between other components are indicated by thick lines and are for illustrative purposes only, not as limiting information. The 804 bus can be divided into address bus, data bus, control bus, etc. For ease of illustration, Figure 8 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0245] In another embodiment, the electronic device can be any electronic device such as a mobile phone, tablet computer, POS (Point of Sales), in-vehicle computer, smart wearable device, or PC. The electronic device can also be... Figure 1a The terminal device 100 shown is shown.

[0246] Figure 9 A structural block diagram of an electronic device provided in an embodiment of this application is shown. Figure 9 As shown, the electronic device includes: a radio frequency (RF) circuit 910, a memory 920, an input unit 930, a display unit 940, a sensor 950, an audio circuit 960, a wireless fidelity (WiFi) module 970, a processor 980, and other components. Those skilled in the art will understand that... Figure 9 The electronic device structure shown does not constitute a limitation on the electronic device and may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0247] The following is combined with Figure 9 A detailed introduction to each component of the electronic device:

[0248] The RF circuit 910 can be used to receive and transmit signals during information transmission or calls. In particular, it receives downlink information from the base station and processes it with the processor 980; in addition, it transmits uplink data to the base station.

[0249] The memory 920 can be used to store software programs and modules, such as the adversarial training method and device for the retrieval-based multi-turn dialogue model in this application embodiment, and the corresponding program instructions / modules. The processor 980 executes various functional applications and data processing of the electronic device by running the software programs and modules stored in the memory 920, such as the multimodal dialogue translation method provided in this application embodiment. The memory 920 may mainly include a program storage area and a data storage area. The program storage area may store the operating system, application programs of at least one application, etc.; the data storage area may store data created according to the use of the electronic device, etc. In addition, the memory 920 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other volatile solid-state storage device.

[0250] The input unit 930 can be used to receive numeric or character information input by the user, and to generate key signal inputs related to user settings and function control of the terminal.

[0251] Optionally, the input unit 930 may include a touch panel 931 and other input devices 932.

[0252] The touch panel 931, also known as a touch screen, can collect user touch operations on or near it (such as user actions using fingers, styluses, or any suitable object or accessory on or near the touch panel 931), and perform corresponding operations according to a pre-set program, such as the user clicking on a shortcut icon of a function module. Optionally, the touch panel 931 may include two parts: a touch detection device and a touch controller. The touch detection device detects the user's touch position and the signal generated by the touch operation, and transmits the signal to the touch controller; the touch controller receives touch information from the touch detection device, converts it into touch point coordinates, and sends it to the processor 980, and can also receive and execute commands from the processor 980. Furthermore, the touch panel 931 can be implemented using various types of touch technologies, such as resistive, capacitive, infrared, and surface acoustic wave.

[0253] Optionally, other input devices 932 may include, but are not limited to, one or more of the following: physical keyboard, function keys (such as volume control buttons, power buttons, etc.), trackball, mouse, joystick, etc.

[0254] Display unit 940 can be used to display information input by the user or interface information displayed to the user, as well as various menus of electronic devices. Display unit 940 is the display system of the terminal device, used to present the interface, such as displaying the desktop, the operation interface of an application, or the operation interface of a live streaming application.

[0255] The display unit 940 may include a display panel 941. Optionally, the display panel 941 may be configured as a liquid crystal display (LCD), an organic light-emitting diode (OLED), or the like.

[0256] Furthermore, the touch panel 931 can cover the display panel 941. When the touch panel 931 detects a touch operation on or near it, it transmits the information to the processor 980 to determine the type of touch event. Subsequently, the processor 980 provides a corresponding interface output on the display panel 941 according to the type of touch event.

[0257] Although Figure 9 In this embodiment, the touch panel 931 and the display panel 941 are two separate components to realize the input and output functions of the electronic device. However, in some embodiments, the touch panel 931 and the display panel 941 can be integrated to realize the input and output functions of the terminal.

[0258] The electronic device may also include at least one sensor 950, such as a light sensor, a motion sensor, and other sensors. Specifically, the light sensor may include an ambient light sensor and a proximity sensor. The ambient light sensor can adjust the brightness of the display panel 941 according to the ambient light level, and the proximity sensor can turn off the backlight of the display panel 941 when the electronic device is moved to the ear. As a type of motion sensor, an accelerometer sensor can detect the magnitude of acceleration in various directions (generally three axes). When stationary, it can detect the magnitude and direction of gravity and can be used for applications that identify the posture of the electronic device (such as landscape / portrait switching, related games, magnetometer posture calibration), vibration recognition related functions (such as pedometers, taps), etc. Other sensors that may be configured in the electronic device, such as gyroscopes, barometers, hygrometers, thermometers, and infrared sensors, will not be described in detail here.

[0259] Audio circuit 960, speaker 961, and microphone 962 provide an audio interface between the user and electronic device. Audio circuit 960 converts received audio data into electrical signals and transmits them to speaker 961, where speaker 961 converts them into sound signals for output. On the other hand, microphone 962 converts collected sound signals into electrical signals, which are received by audio circuit 960, converted into audio data, and then output to processor 980 for processing. The audio data is then transmitted via RF circuit 910 to, for example, another electronic device, or output to memory 920 for further processing.

[0260] WiFi is a short-range wireless transmission technology. Electronic devices using the WiFi module 970 can help users send and receive emails, browse web pages, and access streaming media, providing users with wireless broadband internet access. Although Figure 9 WiFi module 970 is shown, but it is understood that it is not a necessary component of an electronic device and can be omitted as needed without changing the nature of the invention.

[0261] The processor 980 is the control center of the electronic device, connecting various parts of the device via various interfaces and lines. It executes software programs and / or modules stored in the memory 920, and calls data stored in the memory 920 to perform various functions and process data. Optionally, the processor 980 may include one or more processing units; optionally, the processor 980 may integrate an application processor and a modem processor. The application processor mainly processes software programs such as the operating system, applications, and functional modules within the applications, such as the multimodal dialogue translation method provided in this embodiment. The modem processor mainly processes wireless communication. It is understood that the modem processor may not be integrated into the processor 980.

[0262] Understandable. Figure 9 The structure shown is for illustrative purposes only; the electronic device may also include components that are more advanced than those shown. Figure 9 The more or fewer components shown, or having the same Figure 9 The different configurations shown. Figure 9 The components shown can be implemented using hardware, software, or a combination thereof.

[0263] According to one aspect of this application, a computer program product or computer program is provided, comprising computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the multimodal dialogue translation method described above.

[0264] The program product may employ any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may be, for example—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples (a non-exhaustive list) of readable storage media include: electrical connections having one or more wires, portable disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0265] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application.

Claims

1. A multimodal dialogue translation method, characterized in that, include: Obtain the dialogue text to be translated and the historical dialogue text; wherein, the historical dialogue text records the contextual dialogue content of the dialogue text to be translated, and both the dialogue text to be translated and the historical dialogue text are obtained based on the source language; Based on the dialogue text to be translated and the historical dialogue text, the contextual relationship between the dialogue text to be translated and the historical dialogue text, as well as the dialogue expression characteristics of the speaker, are determined; wherein, the contextual relationship represents at least one of the consistency of dialogue topic and dialogue coherence between the dialogue text to be translated and the historical dialogue text; the dialogue expression characteristics represent at least one of the speaker's expression style, word usage habits, and emotional expression; at least one scene image corresponding to the dialogue text to be translated is obtained, and feature extraction is performed on the at least one scene image to obtain the corresponding image feature set; Based on the trained multimodal dialogue translation model, the contextual relationships and the dialogue expression features are concatenated to obtain the dialogue text vector; The dialogue text vector and the image feature set are respectively transformed into the same semantic space to obtain the corresponding target text vector and target image vector; the semantic space represents the expression form of the language; The target text vector is matched with the target image vector to determine the similarity between the target text vector and the target image vector. Based on the similarity, the target image vector and the target text vector are weighted and merged to obtain multimodal dialogue features. The multimodal dialogue features are translated based on the target language to obtain the target translated dialogue text.

2. The method as described in claim 1, characterized in that, The step of determining the contextual relationship between the dialogue text to be translated and the historical dialogue text, as well as the dialogue expression characteristics of the speakers, based on the dialogue text to be translated and the historical dialogue text, includes: The dialogue text to be translated is vectorized to obtain the dialogue vector to be translated. The historical dialogue text is vectorized to obtain the historical dialogue vector. Based on a trained multimodal dialogue translation model, semantic relevance matching is performed between the dialogue vector to be translated and the historical dialogue vector to determine the contextual relationship; Based on the multimodal dialogue translation model, a target historical dialogue vector with the same speaker as the dialogue vector to be translated is determined from the historical dialogue vector, and the dialogue expression features of the speaker are determined according to the dialogue vector to be translated and the target historical dialogue vector.

3. The method as described in claim 2, characterized in that, The trained multimodal dialogue translation model performs semantic relevance matching between the dialogue vector to be translated and the historical dialogue vector to determine the contextual relationship, including: Based on a trained multimodal dialogue translation model, at least one of the following is determined between the dialogue text to be translated and the historical dialogue text: dialogue topic consistency and dialogue coherence, and the determination result is used as the context relationship.

4. The method as described in claim 2, characterized in that, The step of determining the dialogue expression features of the speaker based on the dialogue vector to be translated and the target historical dialogue vector includes: Based on the dialogue vector to be translated and the target historical dialogue vector, at least one of the speaker's expression style, word usage habits, and emotional expression is determined, and the determination result is used as the speaker's dialogue expression feature.

5. The method as described in claim 1, characterized in that, The process of translating the multimodal dialogue features based on the target language to obtain the target translated dialogue text includes: Based on the trained multimodal dialogue translation model, the probability values ​​of the multimodal dialogue features corresponding to each preset word in the target language are determined. Based on the probability values ​​corresponding to each preset word, a set of target words is selected; Based on the target word set, the target translated dialogue text corresponding to the multimodal dialogue features is determined.

6. The method as described in claim 2, characterized in that, The dialogue vector to be translated includes at least one of the following: word vector, word position vector, text turn vector, and role vector corresponding to the dialogue text to be translated; the historical dialogue vector includes at least one of the following: word vector, word position vector, text turn vector, and role vector corresponding to the historical dialogue text; wherein... Each word vector represents the semantic information of each word in the corresponding text; Each word position vector represents the position of each word in the corresponding text; Each text round vector represents the order of the corresponding text in the semantically related text set; Each role vector represents the identity attribute of the speaker in the corresponding text.

7. The method according to any one of claims 2 to 6, characterized in that, The training process of the multimodal dialogue translation model includes: Obtain the training dataset; the training dataset includes sample dialogue text and sample historical dialogue text corresponding to the source language, as well as sample real translation text of the sample dialogue text corresponding to the target language; Based on the training dataset, the multimodal dialogue translation model is iteratively trained until the set convergence condition is met. One iteration of the training process includes: The multimodal dialogue translation model determines the contextual relationship between the selected sample dialogue text and the corresponding sample historical dialogue text, as well as the sample dialogue expression features of the sample speaker. Based on the contextual relationships and the expression features of the sample dialogue, the target translation dialogue text corresponding to the sample dialogue text is obtained, and the loss value determined based on the target translation dialogue text and the sample real translation text is used to adjust the parameters of the multimodal dialogue translation model.

8. The method as described in claim 7, characterized in that, The sample dialogue text also corresponds to at least one sample scene image; The step of obtaining the target translation dialogue text corresponding to the sample dialogue text based on the contextual relationship and the sample dialogue expression features includes: The contextual relationship, the sample dialogue expression features, and the sample image feature set corresponding to the at least one sample scene image are fused to obtain sample multimodal dialogue features; The multimodal dialogue features of the sample are translated to obtain the target translated dialogue text of the sample.

9. A multimodal dialogue translation device, characterized in that, include: The text acquisition module is used to acquire the dialogue text to be translated and the historical dialogue text; wherein, the historical dialogue text records the context dialogue content of the dialogue text to be translated, and both the dialogue text to be translated and the historical dialogue text are acquired based on the source language; The relationship determination module is used to determine the contextual relationship between the dialogue text to be translated and the historical dialogue text, as well as the dialogue expression characteristics of the speaker, based on the dialogue text to be translated and the historical dialogue text; wherein, the contextual relationship represents at least one of the following: the consistency of dialogue topic and the coherence of dialogue between the dialogue text to be translated and the historical dialogue text; the dialogue expression characteristics represent at least one of the following: the speaker's expression style, word usage habits, and emotional expression. The feature extraction module is used to acquire at least one scene image corresponding to the dialogue text to be translated, and to extract features from the at least one scene image to obtain a corresponding image feature set. The feature fusion module is used to concatenate the contextual relationships and dialogue expression features based on a trained multimodal dialogue translation model to obtain a dialogue text vector; convert the dialogue text vector and the image feature set into the same semantic space to obtain corresponding target text vectors and target image vectors; the semantic space represents the expression form of the language; match the target text vectors with the target image vectors to determine the similarity between the target text vectors and the target image vectors, and based on the similarity, perform weighted merging of the target image vectors and target text vectors to obtain multimodal dialogue features; The translation module is used to translate the multimodal dialogue features based on the target language to obtain the target translated dialogue text.

10. A computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the steps of the method according to any one of claims 1 to 8.

11. A computer-readable storage medium, characterized in that, It stores a computer program executable by a computer device, which, when run on the computer device, causes the computer device to perform the steps of the method according to any one of claims 1 to 8.

12. A computer program product, characterized in that, The computer program product includes a computer program stored on a computer-readable storage medium, the computer program including program instructions that, when executed by a computer, cause the computer to perform the steps of any one of the methods of claims 1-8.

Images