Translation method, device, readable medium and electronic device

Abstract

The present disclosure relates to a translation method, apparatus, readable medium, and electronic device. The method includes: determining a source text to be translated and a source-associated image corresponding to the source text; inputting the source text and the source-associated image into a pre-generated target multimodal translation model to obtain a target translated text output by the target multimodal translation model; wherein the target multimodal translation model is a model generated by training the target multimodal translation model based on sample data, the sample data including at least two of multimodal multilingual data, unimodal multilingual data, and multimodal monolingual data; the multimodal multilingual data including a first source language text, a first target language text, and a first image corresponding to the first source language text; the unimodal multilingual data including a second source language text and a second target language text; and the multimodal monolingual data including a third target language text and a second image.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, and in particular, to a translation method, device, readable medium, and electronic device. Background Art

[0002] With the advancement of computer technology, machine translation has become an important research topic in natural language text processing. Machine translation refers to the process of translating text in a source language into semantically equivalent text in a target language using a computer or other electronic device. Multimodal machine translation can further improve translation accuracy. Multimodal machine translation refers to a machine translation model that integrates information from multiple modalities, such as text, images, and sound. Compared to pure text machine translation models, multimodal machine translation models can use modal information other than text to assist and improve translation results, making up for the shortcomings of single-modal machine translation and improving machine translation accuracy. For example, image information can help eliminate ambiguity and improve accuracy, which is particularly useful in scenarios such as e-commerce and conversations.

[0003] However, in related technologies, there is a problem that the translation accuracy of multimodal machine translation models is not high, which affects the application of multimodal machine translation. Summary of the Invention

[0004] This summary is provided to briefly introduce concepts that will be described in detail in the detailed description below. This summary is not intended to identify key features or essential features of the claimed technical solution, nor is it intended to limit the scope of the claimed technical solution.

[0005] According to a first aspect of an embodiment of the present disclosure, a translation method is provided, the method comprising:

[0006] Determining a source text to be translated and a source-associated image corresponding to the source text;

[0007] Inputting the source text and the source-associated image into a pre-generated target multimodal translation model to obtain a target translation text output by the target multimodal translation model;

[0008] The target multimodal translation model is a model generated after training a target multimodal translation model based on sample data, the sample data including at least two types of multimodal multilingual data, unimodal multilingual data, and multimodal monolingual data; the multimodal multilingual data includes a first source language text, a first target language text, and a first image corresponding to the first source language text; the unimodal multilingual data includes a second source language text and a second target language text; the multimodal monolingual data includes a third target language text and a second image, the first image being an associated image corresponding to the first source language text, and the second image being an associated image corresponding to the third target language text; the first source language text, the second source language text, and the source text are of the same language type, and the first target language text, the second target language text, the third target language text, and the target translation text are of the same language type.

[0009] According to a second aspect of an embodiment of the present disclosure, a translation device is provided, the device comprising:

[0010] a determination module, configured to determine a source text to be translated and a source-associated image corresponding to the source text;

[0011] a translation module, configured to input the source text and the source associated image into a pre-generated target multimodal translation model to obtain a target translated text output by the target multimodal translation model;

[0012] The target multimodal translation model is a model generated after training a target multimodal translation model based on sample data, the sample data including at least two types of multimodal multilingual data, unimodal multilingual data, and multimodal monolingual data; the multimodal multilingual data includes a first source language text, a first target language text, and a first image corresponding to the first source language text; the unimodal multilingual data includes a second source language text and a second target language text; the multimodal monolingual data includes a third target language text and a second image, the first image being an associated image corresponding to the first source language text, and the second image being an associated image corresponding to the third target language text; the first source language text, the second source language text, and the source text are of the same language type, and the first target language text, the second target language text, the third target language text, and the target translation text are of the same language type.

[0013] According to a third aspect of an embodiment of the present disclosure, a computer-readable medium is provided, on which a computer program is stored. When the computer program is executed by a processing device, the steps of the method described in the first aspect of the present disclosure are implemented.

[0014] According to a fourth aspect of the embodiments of the present disclosure, there is provided an electronic device, including:

[0015] a storage device having a computer program stored thereon;

[0016] A processing device is used to execute the computer program in the storage device to implement the steps of the method described in the first aspect of the present disclosure.

[0017] Using the above technical solution, a source text to be translated and a source-associated image corresponding to the source text are determined; the source text and the source-associated image are input into a pre-generated target multimodal translation model to obtain a target translated text output by the target multimodal translation model; wherein the target multimodal translation model is a model generated by training the predetermined multimodal translation model based on sample data, the sample data including at least two types of multimodal multilingual data, unimodal multilingual data, and multimodal monolingual data; the multimodal multilingual data includes a first source language text, a first target language text, and a first image corresponding to the first source language text; the unimodal multilingual data includes a second source language text and a second target language text; the multimodal monolingual data includes a third target language text and a second image, the first image being an associated image corresponding to the first source language text, and the second image being an associated image corresponding to the third target language text; the first source language text, the second source language text, and the source text are of the same language type, and the first target language text, the second target language text, the third target language text, and the target translated text are of the same language type. In this way, by using at least two types of sample data such as multimodal multilingual data, monomodal multilingual data and multimodal monolingual data, the target multimodal translation model is generated after training the predetermined multimodal translation model, which can increase the number of samples used for model training, thereby improving the translation accuracy of the trained target multimodal translation model.

[0018] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] The above and other features, advantages and aspects of the various embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. Throughout the drawings, the same or similar reference numerals represent the same or similar elements. It should be understood that the drawings are schematic and that the originals and elements are not necessarily drawn to scale. In the drawings:

[0020] Figure 1 The figure is a flowchart of a translation method according to an exemplary embodiment.

[0021] Figure 2 The figure is a schematic structural diagram of a target multimodal translation model according to an exemplary embodiment.

[0022] Figure 3The present invention is a flowchart of a method for generating a target multimodal translation model according to an exemplary embodiment.

[0023] Figure 4 The figure is a schematic diagram of a data flow for training a target multimodal translation model according to an exemplary embodiment.

[0024] Figure 5 is a schematic structural diagram of another target multimodal translation model according to an exemplary embodiment.

[0025] Figure 6 The figure is a flowchart of another method for generating a target multimodal translation model according to an exemplary embodiment.

[0026] Figure 7 FIG. 4 is a schematic diagram of a data flow for training another target multimodal translation model according to an exemplary embodiment.

[0027] Figure 8 is a schematic structural diagram of another target multimodal translation model according to an exemplary embodiment.

[0028] Figure 9 The figure is a block diagram of a translation device according to an exemplary embodiment.

[0029] Figure 10 It is a block diagram of another translation device according to an exemplary embodiment.

[0030] Figure 11 It is a block diagram of an electronic device according to an exemplary embodiment. DETAILED DESCRIPTION

[0031] The following describes embodiments of the present disclosure in more detail with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as limited to the embodiments described herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are for illustrative purposes only and are not intended to limit the scope of protection of the present disclosure.

[0032] It should be understood that the various steps described in the method embodiments of the present disclosure may be performed in different orders and / or in parallel. In addition, the method embodiments may include additional steps and / or omit the steps shown. The scope of the present disclosure is not limited in this respect.

[0033] As used herein, the term "including" and its variations are open-ended, i.e., "including but not limited to." The term "based on" means "based, at least in part, on." The term "one embodiment" means "at least one embodiment," the term "another embodiment" means "at least one additional embodiment," and the term "some embodiments" means "at least some embodiments." Other terms are defined in the following description.

[0034] It should be noted that the concepts of "first" and "second" mentioned in this disclosure are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units.

[0035] It should be noted that the modifications of "one" and "multiple" mentioned in the present disclosure are illustrative rather than restrictive. Those skilled in the art should understand that, unless otherwise clearly indicated in the context, they should be understood as "one or more". In the description of the present disclosure, unless otherwise specified, "multiple" refers to two or more than two, and other quantifiers are similar; "at least one item (individual)", "one (individual) or multiple items (individuals)" or similar expressions refer to any combination of these items (individuals), including any combination of single items (individuals) or plural items (individuals). For example, at least one item (individual) a can represent any number of a; for another example, one (individual) or multiple items (individuals) among a, b and c can represent: a, b, c, ab, ac, bc, or abc, where a, b, c can be single or multiple; "and / or" is a kind of association relationship that describes the associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone, where A and B can be singular or plural.

[0036] Although operations or steps are described in a particular order in the drawings in the embodiments of the present disclosure, this should not be understood as requiring that these operations or steps be performed in the particular order shown or in a serial order, or that all of the operations or steps shown be performed to obtain a desired result. In the embodiments of the present disclosure, these operations or steps may be performed serially; these operations or steps may also be performed in parallel; or some of these operations or steps may be performed.

[0037] The names of the messages or information exchanged between multiple devices in the embodiments of the present disclosure are only used for illustrative purposes and are not used to limit the scope of these messages or information.

[0038] It is understandable that before using the technical solutions disclosed in the various embodiments of this disclosure, the type, scope of use, usage scenarios, etc. of the personal information involved in this disclosure should be informed to the user and the user's authorization should be obtained in an appropriate manner in accordance with relevant laws and regulations.

[0039] For example, in response to a user's active request, a prompt message is sent to the user to clearly inform the user that the operation requested will require the acquisition and use of the user's personal information. This allows the user to independently choose whether to provide personal information to the electronic device, application, server, storage medium, or other software or hardware that performs the operations of the disclosed technical solution based on the prompt message.

[0040] As an optional but non-limiting implementation, in response to receiving a user's active request, the prompt information may be sent to the user in the form of a pop-up window, in which the prompt information may be presented in text form. Furthermore, the pop-up window may also contain a selection control for the user to select "agree" or "disagree" to provide personal information to the electronic device.

[0041] It is understandable that the above notification and user authorization process are merely illustrative and do not limit the implementation of the present disclosure. Other methods that comply with relevant laws and regulations may also be applied to the implementation of the present disclosure.

[0042] At the same time, it is understandable that the data involved in this technical solution (including but not limited to the data itself, the acquisition or use of the data) must comply with the requirements of relevant laws, regulations and relevant provisions.

[0043] First, the application scenarios of the present disclosure are described. The present disclosure can be applied to machine translation scenarios, particularly multimodal machine translation scenarios. Multimodal machine translation models can leverage information in modalities other than text to assist and improve translation results, addressing the shortcomings of single-modal machine translation and enhancing its accuracy. For example, image information can help eliminate ambiguity and improve accuracy, making it particularly useful in scenarios such as e-commerce and conversations.

[0044] In related technologies, multimodal machine translation can use triplets consisting of source language text, target language text, and images as training and test data. However, since this type of data is very scarce and difficult to label, the translation accuracy of the trained model is not high.

[0045] To address the above-mentioned issues, the present disclosure provides a translation method, apparatus, readable medium, and electronic device. By employing at least two types of sample data, such as multimodal multilingual data, monomodal multilingual data, and multimodal monolingual data, a target multimodal translation model is generated after training a predetermined multimodal translation model. This can increase the number of samples used for model training, thereby improving the translation accuracy of the trained target multimodal translation model.

[0046] The present disclosure is described below with reference to specific embodiments.

[0047] Figure 1 This is a flowchart of a translation method according to an exemplary embodiment. The method can be applied to electronic devices, which may include terminal devices such as smart phones, smart wearable devices, smart speakers, smart tablets, PDAs (Personal Digital Assistants), CPEs (Customer Premise Equipment), personal computers, and vehicle-mounted terminals. The electronic devices may also include servers, such as local servers or cloud servers. Figure 1 As shown, the method may include:

[0048] S101: Determine a source text to be translated and a source-associated image corresponding to the source text.

[0049] The source text may be text in a source language, and the source-associated image may be an image used to indicate all or part of the semantics expressed by the source text. For example, if the source text is a black hat, the source-associated image may be an image of a black hat, an image of a hat of any color, or an image of any black object.

[0050] It should be noted that the source text may be a word, a sentence, a paragraph or an article, and this disclosure does not limit this.

[0051] S102: Input the source text and the source associated image into a pre-generated target multimodal translation model to obtain a target translation text output by the target multimodal translation model.

[0052] The target multimodal translation model is a model generated after training the target multimodal translation model based on sample data, and the sample data may include at least two types of data among multimodal multilingual data, unimodal multilingual data and multimodal monolingual data.

[0053] The multimodal multilingual data may include a first source language text, a first target language text, and a first image corresponding to the first source language text; the monomodal multilingual data may include a second source language text and a second target language text; and the multimodal monolingual data may include a third target language text and a second image; the first source language text, the second source language text, and the source text may be of the same language type, for example, all Chinese; the first target language text, the second target language text, the third target language text, and the target translation text may be of the same language type, for example, all English.

[0054] In some embodiments, the sample data may include multimodal multilingual data, unimodal multilingual data, and multimodal monolingual data. In other embodiments, the sample data may include unimodal multilingual data and multimodal monolingual data. In other embodiments, the sample data may include multimodal multilingual data and multimodal monolingual data. In still other embodiments, the sample data may include multimodal multilingual data and unimodal multilingual data.

[0055] It should be noted that the above-mentioned first image can be an associated image corresponding to the first source language text, and the first image can be used to prompt part or all of the semantics of the first source language text. The above-mentioned second image can be an associated image corresponding to the third target language text, and the second image can be used to prompt part or all of the semantics of the third target language text.

[0056] Using the above method, a source text to be translated and a source-associated image corresponding to the source text are determined; the source text and the source-associated image are input into a pre-generated target multimodal translation model to obtain a target translated text output by the target multimodal translation model; wherein the target multimodal translation model is a model generated by training the predetermined multimodal translation model based on sample data, the sample data including at least two types of multimodal multilingual data, unimodal multilingual data, and multimodal monolingual data; the multimodal multilingual data includes a first source language text, a first target language text, and a first image corresponding to the first source language text; the unimodal multilingual data includes a second source language text and a second target language text; the multimodal monolingual data includes a third target language text and a second image, the first image being an associated image corresponding to the first source language text, and the second image being an associated image corresponding to the third target language text; the first source language text, the second source language text, and the source text are of the same language type, and the first target language text, the second target language text, the third target language text, and the target translated text are of the same language type. In this way, by using at least two types of sample data such as multimodal multilingual data, monomodal multilingual data and multimodal monolingual data, the target multimodal translation model is generated after training the predetermined multimodal translation model, which can increase the number of samples used for model training, thereby improving the translation accuracy of the trained target multimodal translation model.

[0057] Figure 2 FIG. 1 is a schematic diagram showing the structure of a target multimodal translation model according to an exemplary embodiment. Figure 2 As shown, the target multimodal translation model includes an image encoder 201, a first text encoder 202, and a first text decoder 203. The output of the image encoder 201 and the output of the first text encoder 202 are weighted and then input into the first text decoder 203.

[0058] It should be noted that Figure 2The target multimodal translation model shown can also be called a fusion multimodal translation model. The image encoder can generate source-related image features. These source-related image features are then fused with the text features of the source text to assist in translating the source text. Based on this target multimodal translation model, step S102 above can include the following sub-steps:

[0059] First, the source-associated image is input into the image encoder to obtain the source-associated image features output by the image encoder.

[0060] For example, the image encoder can be an image encoder (ImageEncoder) in a CLIP (Contrastive Language-Image Pre-Training) model. The CLIP model is a pre-trained neural network model that can be used to match images and texts. The image encoder can include ResNet (Deepresidual network) and / or ViT (Vision Transformer). The CLIP model can directly use a large amount of Internet data for pre-training to improve the accuracy of image and text encoding.

[0061] It should be noted that the image encoder can also be a neural network used for image encoding in related technologies, such as convolutional neural network (CNN), recurrent neural network (RNN), deep belief network (DBN) and generative adversarial network (GAN).

[0062] Secondly, the source text is input into the first text encoder to obtain the first source text features output by the first text encoder.

[0063] For example, the first text encoder may also be the text encoder in the above-mentioned CLIP model, or the first text encoder may be a neural network for encoding text in related technologies.

[0064] Thirdly, the first source text feature and the source associated image feature are weighted to obtain the target feature.

[0065] For example, the target feature can be obtained by performing a weighted summation of the first source text feature and the source-associated image feature. The weights of the first source text feature and the source-associated image feature can be preset. For example, the weights of the first source text feature and the source-associated image feature can be set to 50% each; or the weight of the first source text feature can be set to 70% and the weight of the source-associated image feature can be set to 30%.

[0066] Finally, the target features are input into the first text decoder to obtain the target translation text output by the first text decoder.

[0067] It should be noted that the first text decoder may also be a decoder used for machine translation in related technologies, and the present disclosure does not limit the specific structure of the first text decoder.

[0068] In this way, the source text and the source-associated image can be processed by the image encoder, the first text encoder, and the first text decoder in the target multimodal translation model to obtain the translated target translation text.

[0069] Figure 3 FIG is a flow chart of a method for generating a target multimodal translation model according to an exemplary embodiment. Figure 3 As shown, the target multimodal translation model can be obtained by training the pending multimodal translation model. The structure of the pending multimodal translation model is the same as that of the target multimodal translation model. For example, the pending multimodal translation model can also include Figure 2 The image encoder, the first text encoder and the first text decoder shown. The method may include:

[0070] S301: Obtain sample data.

[0071] S302: Looping through the first training step according to the sample data until it is determined that the trained pending multimodal translation model meets a first preset stop iteration condition, and using the trained pending multimodal translation model as the target multimodal translation model.

[0072] The first training step may include:

[0073] S11 . Determine a to-be-determined image sample feature, a first to-be-determined source text, and a first to-be-determined target text according to the sample data.

[0074] The sample data may include at least two of the multimodal multilingual data, the monomodal multilingual data, and the multimodal monolingual data. Different types of sample data may be processed differently to determine the undetermined image sample features, the first undetermined source text, and the first undetermined target text.

[0075] In some embodiments, when the sample data includes multimodal and multilingual data, the image features output after the first image is input into the image encoder can be used as the undetermined image sample features, the first source language text can be used as the first undetermined source text, and the first target language text can be used as the first undetermined target text. In this way, the undetermined image sample features, the first undetermined source text, and the first undetermined target text for training can be obtained.

[0076] In other embodiments, when the sample data includes unimodal multilingual data, the preset image sample features can be used as the pending image sample features, the second source language text can be used as the first pending source text, and the second target language text can be used as the first pending target text.

[0077] For example, the preset image sample feature may be 0, that is, the undetermined image sample feature may be set to zero, and the undetermined multimodal translation model may be regarded as an ordinary text Transformer model.

[0078] In some other embodiments, when the sample data includes multimodal monolingual data, the image features output after the second image is input into the image encoder can be used as the pending image sample features, the text obtained after masking the third target language text can be used as the first pending source text, and the third target language text can be used as the first pending target text.

[0079] The masking process may include masking the third target language text by using a preset mask token. For example, a portion or all of the text in the third target language text may be randomly masked, and the masked text may be used as the first pending source text.

[0080] In some further embodiments, when the sample data includes two or three of multimodal multilingual data, monomodal multilingual data, and multimodal monolingual data, different data may be processed separately according to the above-described method to obtain the undetermined image sample features, the first undetermined source text, and the first undetermined target text.

[0081] S12: Input the first undetermined source text into the first text encoder to obtain first text sample features output by the first text encoder.

[0082] S13. Determine a first multimodal sample feature according to the first text sample feature and the first image sample feature.

[0083] For example, the first text sample feature and the first image sample feature may be weightedly summed to obtain the first multimodal sample feature.

[0084] S14. Input the first multimodal sample feature into a first text decoder to obtain a first translated text output by the first text decoder;

[0085] S15. Determine a first loss value based on the first translated text and the first undetermined target text. If it is determined that the undetermined multimodal translation model does not meet a first preset stop iteration condition based on the first loss value, update the parameters of the undetermined multimodal translation model based on the first loss value to obtain a trained undetermined multimodal translation model, and use the trained undetermined multimodal translation model as a new undetermined multimodal translation model.

[0086] In some embodiments, the first preset iteration stopping condition may include any one or more of the following conditions:

[0087] The first loss value is less than or equal to a preset first loss threshold;

[0088] The change in the first loss value within a preset number of iterations is less than or equal to a preset first preset change threshold, and the preset number of iterations can be any preset value, such as the last 2 iterations or the last 10 iterations.

[0089] In other embodiments, the first preset condition for stopping iteration may also be other conditions for stopping iteration in related technologies, such as the total number of training iterations being greater than or equal to a preset threshold number of iterations, which is not limited in this disclosure.

[0090] In addition, when it is determined based on the first loss value that the pending multimodal translation model meets the above-mentioned first preset stop iteration condition, the first training step can be stopped and the trained pending multimodal translation model can be used as the target multimodal translation model.

[0091] In this way, through the above method, the target multimodal translation model can be trained and generated.

[0092] Figure 4 FIG. 1 is a schematic diagram of a data flow for training a target multimodal translation model according to an exemplary embodiment. Figure 4 As shown, the data for training the target multimodal translation model may include sample data 41 , which may include at least two items of the multimodal multilingual data 411 , the unimodal multilingual data 412 , and the multimodal monolingual data 413 .

[0093] like Figure 4 As shown, the multimodal multilingual data 411 can be represented in the form of "(x1, i1)→y1", where x1 represents the first source language text 4111 (e.g. Figure 4 In the example, i1 represents the first image 4112 (e.g., the black mask image 4112 in the figure), y1 represents the first target language text (e.g., “black mask”), and the multimodal multilingual data 411 can also be called triple data. The unimodal multilingual data 412 can be represented in the form of “(x2)→y2”, where x2 represents the second source language text 4121 and y2 represents the second target language text. The unimodal multilingual data 412 can also be called bilingual text data. The multimodal monolingual data 413 can be represented in the form of “(y3 *, i2)→y3”, where i2 represents the second image 4132, y3 represents the third target language text, and y3 * The text 4131 is obtained after masking the third target language text. In the figure, [MASK] indicates that the text here is masked.

[0094] In some embodiments, an image encoder (e.g., an image encoder in a CLIP model) can be used to extract the first image and / or the second image as image features, and a first text encoder (e.g., a text encoder in a CLIP model) can be used to extract the pending text as text features. The pending text can include at least two of the first source language text 4111, the second source language text 4121, and the text 4131 obtained after masking the third target language text. The image features and text features are weighted and summed to obtain target features, which are then input into the first text decoder for translation. For example:

[0095] For the multimodal multilingual data, the image features of the first image can be extracted by an image encoder, and the text features of the first source language text can be extracted by a first text encoder. The image features and the text features are weighted and summed as the final target features. Machine translation is performed based on the target features, that is, the target features are input into the first text decoder for translation.

[0096] For the above-mentioned unimodal multilingual data, the image features can be set to zero. In this case, the model can be degenerated into an ordinary text Transformer model. The text features of the second source language text are directly extracted through the first text encoder, and the text features are used as target features. Machine translation is performed based on the target features, that is, the target features are input into the first text decoder for translation.

[0097] For the above-mentioned multimodal monolingual data, a mechanism similar to self-supervised learning can be used. A preset mask (mask token) is used to randomly mask part of the original text of the third target language text. The text features of the third target language text after the masked original text are extracted through the first text encoder. The image features of the second image are extracted through the image encoder. The image features and the text features are then weighted and summed as the target features. Machine translation is then performed based on the target features, that is, the target features are input into the first text decoder for translation, for example, to restore the original text.

[0098] In this way, fused multimodal translation can be achieved through the target multimodal translation model, which introduces more types of sample data in training and improves the translation accuracy of the trained model.

[0099] Figure 5FIG is a schematic diagram showing the structure of another target multimodal translation model according to an exemplary embodiment. Figure 5 As shown, the target multimodal translation model includes an image converter 501, a second text encoder 502, and a second text decoder 503. The output of the image converter 501 can be used as the input of the second text encoder 502, and the output of the second text encoder 502 can be used as the input of the second text decoder 503.

[0100] It should be noted that Figure 5 The target multimodal translation model shown can also be called a prompt multimodal translation model, which generates target prompt text through an image converter to assist in translating the source text. Based on this target multimodal translation model, the above step S102 may include the following sub-steps:

[0101] First, the source associated image is input into the image converter, and the target prompt text is output by the image converter.

[0102] For example, the image converter may be a pre-generated image annotation model (eg, Caption Model), which is used to extract prompt keywords from an input image and output a target prompt text so as to assist machine translation through the target prompt text.

[0103] Secondly, the source text and the target prompt text are input into the second text encoder to obtain the second source text features output by the text encoder.

[0104] For example, the second text encoder can also be the text encoder in the CLIP model described above, or a neural network used in related art to encode text. The source text and the target prompt text can be concatenated and then input into the second text encoder, or the source text and the target prompt text can be input into the second text encoder separately.

[0105] Finally, the second source text features are input into the second text decoder to obtain the target translation text output by the text decoder.

[0106] It should be noted that the second text decoder may be a decoder used for machine translation in related technologies, and the present disclosure does not limit the specific structure of the second text decoder.

[0107] In this way, the source text and the source-associated image can be processed by the image converter, the second text encoder, and the second text decoder in the target multimodal translation model to obtain the translated target translation text.

[0108] Figure 6 FIG is a flow chart of another method for generating a target multimodal translation model according to an exemplary embodiment. Figure 6As shown, the target multimodal translation model can be obtained by training the pending multimodal translation model. The structure of the pending multimodal translation model is the same as that of the target multimodal translation model. For example, the pending multimodal translation model can also include Figure 4 The image converter, the second text encoder and the second text decoder shown. The method may include:

[0109] S601: Obtain sample data.

[0110] S602: Looping through the second training step according to the sample data until it is determined that the trained pending multimodal translation model meets a second preset stop iteration condition, and using the trained pending multimodal translation model as the target multimodal translation model.

[0111] The second training step includes:

[0112] S21 . Determine a pending prompt text, a second pending source text, and a second pending target text according to the sample data.

[0113] The sample data may also include at least two of the multimodal multilingual data, the monomodal multilingual data, and the multimodal monolingual data. Different types of sample data may be processed differently to determine the pending prompt text, the second pending source text, and the second pending target text.

[0114] In some embodiments, when the sample data includes multimodal and multilingual data, the text output after the first image is input into the image converter is used as the pending prompt text, the first source language text is used as the second pending source text, and the first target language text is used as the second pending target text.

[0115] In other embodiments, when the sample data includes unimodal multilingual data, the preset prompt text is used as the pending prompt text, the second source language text is used as the second pending source text, and the second target language text is used as the second pending target text.

[0116] For example, the preset prompt text may be empty text.

[0117] In some other embodiments, when the sample data includes multimodal monolingual data, the text output after the second image is input into the image converter is used as the pending prompt text, the text obtained after masking the third target language text is used as the second pending source text, and the third target language text is used as the second pending target text.

[0118] Likewise, the masking process may include masking the third target language text using a preset mask token. For example, a portion or all of the third target language text may be randomly masked, and the masked text may be used as the second pending source text.

[0119] In some further embodiments, when the sample data includes two or three of multimodal multilingual data, monomodal multilingual data, and multimodal monolingual data, different data may be processed separately according to the above-described method to obtain a pending prompt text, a second pending source text, and a second pending target text.

[0120] S22: Input the second undetermined source text and the undetermined prompt text into a second text encoder to obtain second text sample features output by the second text encoder.

[0121] In this step, the second pending source text and the pending prompt text may be spliced ​​and then input into the second text encoder, or the second pending source text and the pending prompt text may be input into the second text encoder separately.

[0122] S23: Input the second text sample feature into the second text decoder to obtain a second translated text output by the second text decoder.

[0123] S24. Determine a second loss value based on the second translated text and the second undetermined target text. If it is determined that the undetermined multimodal translation model does not meet the second preset stop iteration condition based on the second loss value, update the parameters of the undetermined multimodal translation model based on the second loss value to obtain a trained undetermined multimodal translation model, and use the trained undetermined multimodal translation model as a new undetermined multimodal translation model.

[0124] In some embodiments, the second preset iteration stopping condition may include any one or more of the following conditions:

[0125] The second loss value is less than or equal to a preset second loss threshold;

[0126] The change in the second loss value within a preset number of iterations is less than or equal to a preset second preset change threshold, and the preset number of iterations can be any preset value, such as the last 2 iterations or the last 10 iterations.

[0127] In other embodiments, the second preset condition for stopping iteration may also be other conditions for stopping iteration in related technologies, for example, the total number of training iterations is greater than or equal to a preset threshold number of iterations, etc., which is not limited in this disclosure.

[0128] In addition, when it is determined based on the second loss value that the pending multimodal translation model meets the above-mentioned first preset stop iteration condition, the first training step can be stopped and the trained pending multimodal translation model can be used as the target multimodal translation model.

[0129] In this way, through the above method, the target multimodal translation model can be trained and generated.

[0130] Figure 7 FIG is a schematic diagram of another data flow for training a target multimodal translation model according to an exemplary embodiment. Figure 7 As shown, the data for training the target multimodal translation model may include sample data 41 , which may include at least two items of the multimodal multilingual data 411 , the unimodal multilingual data 412 , and the multimodal monolingual data 413 .

[0131] For the specific representation of the above sample data, please refer to the disclosure of Figure 4 It is described in [1] and will not be repeated here.

[0132] based on Figure 7 , the first image and / or the second image can be converted into pending prompt text (such as prompt keywords) through an image converter (such as CaptionModel), and auxiliary translation is performed based on the pending prompt text. For example:

[0133] For the above-mentioned multimodal multilingual data, the first image can be directly converted into a pending prompt text through an image converter, and the pending prompt text and the first source language text can be concatenated as new input text to train the second text encoder and the second text decoder in the pending multimodal translation model.

[0134] For the aforementioned unimodal multilingual data, the model can be directly trained on the parallel corpus. In this case, the model degenerates into a standard text Transformer model. The second text encoder and second text decoder in the proposed multimodal translation model are trained based on the second source language text and the second target language text.

[0135] For the above-mentioned multimodal monolingual data, a mechanism similar to self-supervised learning can be used. A preset mask (mask token) is used to randomly cover part of the original text of the third target language text. The first image is directly converted into a pending prompt text through an image converter. The pending prompt text and the third target language text with the masked original text are concatenated as the new input text to train the second text encoder and the second text decoder in the pending multimodal translation model.

[0136] In this way, prompted multimodal translation can be achieved through the target multimodal translation model, which introduces more types of sample data in training and improves the translation accuracy of the trained model.

[0137] Figure 8 FIG is a schematic diagram showing the structure of another target multimodal translation model according to an exemplary embodiment. Figure 8 As shown, the target multimodal translation model includes an image encoder 201, an image converter 501, a text encoder 802, and a text decoder 803. The output of the image converter 501 can be used as the input of the text encoder 502. The output of the image encoder 201 and the output of the text encoder 802 can be input into the text decoder 803 after weighted processing.

[0138] It should be noted that, for the specific implementation of the above-mentioned image encoder, image converter, text encoder and text decoder, reference can be made to the description in the aforementioned embodiments of the present disclosure, which will not be repeated here.

[0139] In this way, the above-mentioned fusion and prompt models can be combined to further improve the translation accuracy of the target multimodal translation model.

[0140] In order to verify the effect of the target multimodal translation model in the embodiments of the present disclosure, the BLEU (Bilingual Evaluation Understudy) indicators of various models were compared and verified. The BLEU indicators obtained by multimodal translation verification of different models are shown in Table 1:

[0141]

[0142] Table 1

[0143] The selective attention translation model in Table 1 is a multimodal translation model used in related technologies, and the fusion multimodal translation model is a multimodal translation model used in the embodiments of the present disclosure. Figure 2 The target multimodal translation model shown in the figure is the prompt multimodal translation model in the embodiment of the present disclosure. Figure 5 The target multimodal translation model shown in FIG. 1 is a fusion + prompt multimodal translation model in the embodiment of the present disclosure. Figure 8 The target multimodal translation model shown.

[0144] As shown in Table 1, when using multimodal and multilingual data as training sample data, the BLEU index of the selective attention translation model is 36.39, the BLEU index of the fusion multimodal translation model is 37.18, the BLEU index of the prompt multimodal translation model is 37.20, and the BLEU index of the fusion + prompt multimodal translation model is 37.03;

[0145] When using both multimodal and multilingual data and unimodal and multilingual data as training samples, the BLEU index of the fusion multimodal translation model is 39.63, the BLEU index of the prompt multimodal translation model is 38.22, and the BLEU index of the fusion + prompt multimodal translation model is 40.10.

[0146] When using "multimodal multilingual data + monomodal multilingual data + multimodal monolingual data" as training sample data, the BLEU index of the fusion multimodal translation model is 41.49, the BLEU index of the prompt multimodal translation model is 40.79, and the BLEU index of the fusion + prompt multimodal translation model is 41.62.

[0147] It can be seen that the target multimodal translation model and translation method provided by the embodiments of the present disclosure can greatly improve the effect of multimodal translation.

[0148] Figure 9 is a block diagram of a translation device 1100 according to an exemplary embodiment. Figure 9 As shown, the apparatus 1100 may include:

[0149] A determination module 1101 is configured to determine a source text to be translated and a source-associated image corresponding to the source text;

[0150] A translation module 1102 is configured to input the source text and the source associated image into a pre-generated target multimodal translation model to obtain a target translated text output by the target multimodal translation model;

[0151] The target multimodal translation model is a model generated after training a target multimodal translation model based on sample data, the sample data including at least two types of multimodal multilingual data, unimodal multilingual data, and multimodal monolingual data; the multimodal multilingual data includes a first source language text, a first target language text, and a first image corresponding to the first source language text; the unimodal multilingual data includes a second source language text and a second target language text; the multimodal monolingual data includes a third target language text and a second image, the first image being an associated image corresponding to the first source language text, and the second image being an associated image corresponding to the third target language text; the first source language text, the second source language text, and the source text are of the same language type, and the first target language text, the second target language text, the third target language text, and the target translation text are of the same language type.

[0152] According to one or more embodiments of the present disclosure, the target multimodal translation model includes an image encoder, a first text encoder, and a first text decoder; the translation module 1102 is configured to input the source-associated image into the image encoder to obtain source-associated image features output by the image encoder; input the source text into the first text encoder to obtain first source text features output by the first text encoder; obtain target features after weighted processing of the first source text features and the source-associated image features; and input the target features into the first text decoder to obtain target translated text output by the first text decoder.

[0153] Figure 10 is a block diagram of another translation device 1100 according to an exemplary embodiment. The undetermined multimodal translation model includes the image encoder, the first text encoder and the first text decoder; Figure 10 As shown, the apparatus 1100 may further include:

[0154] A generation module 1103 is configured to obtain the sample data; cyclically execute the first training step based on the sample data until it is determined that the trained candidate multimodal translation model satisfies a first preset stopping condition, and use the trained candidate multimodal translation model as the target multimodal translation model;

[0155] Among them, the first training step includes: determining the pending image sample features, the first pending source text and the first pending target text according to the sample data; inputting the first pending source text into the first text encoder to obtain the first text sample features output by the first text encoder; determining the first multimodal sample features according to the first text sample features and the first image sample features; inputting the first multimodal sample features into the first text decoder to obtain the first translated text output by the first text decoder; determining a first loss value according to the first translated text and the first pending target text, and when it is determined according to the first loss value that the pending multimodal translation model does not meet the first preset stopping iteration condition, updating the parameters of the pending multimodal translation model according to the first loss value to obtain the trained pending multimodal translation model, and using the trained pending multimodal translation model as the new pending multimodal translation model.

[0156] According to one or more embodiments of the present disclosure, the generation module 1103 is configured to, when the sample data includes the multimodal multilingual data, use the image features output after the first image is input into the image encoder as the pending image sample features, use the first source language text as the first pending source text, and use the first target language text as the first pending target text; or, when the sample data includes the monomodal multilingual data, use preset image sample features as the pending image sample features, use the second source language text as the first pending source text, and use the second target language text as the first pending target text; or, when the sample data includes the multimodal monolingual data, use the image features output after the second image is input into the image encoder as the pending image sample features, use the text obtained after masking the third target language text as the first pending source text, and use the third target language text as the first pending target text.

[0157] According to one or more embodiments of the present disclosure, the target multimodal translation model includes an image converter, a second text encoder, and a second text decoder; the translation module 1102 is configured to input the source-associated image into the image converter to obtain a target prompt text output by the image converter; input the source text and the target prompt text into the second text encoder to obtain second source text features output by the text encoder; and input the second source text features into the second text decoder to obtain a target translated text output by the text decoder.

[0158] According to one or more embodiments of the present disclosure, the undetermined multimodal translation model includes the image converter, the second text encoder, and the second text decoder; the generation module 1103 is configured to obtain the sample data; and loop through the second training step based on the sample data until it is determined that the trained undetermined multimodal translation model satisfies a second preset stopping condition, and the trained undetermined multimodal translation model is used as the target multimodal translation model.

[0159] The second training step includes: determining a pending prompt text, a second pending source text, and a second pending target text based on the sample data; inputting the second pending source text and the pending prompt text into the second text encoder to obtain a second text sample feature output by the second text encoder; inputting the second text sample feature into the second text decoder to obtain a second translated text output by the second text decoder; determining a second loss value based on the second translated text and the second pending target text, and when it is determined that the pending multimodal translation model does not meet the second preset stopping iteration condition based on the second loss value, updating the parameters of the pending multimodal translation model based on the second loss value to obtain a trained pending multimodal translation model, and using the trained pending multimodal translation model as a new pending multimodal translation model.

[0160] According to one or more embodiments of the present disclosure, the generation module 1103 is configured to, when the sample data includes the multimodal multilingual data, use the text output after the first image is input into the image converter as the pending prompt text, use the first source language text as the second pending source text, and use the first target language text as the second pending target text; or, when the sample data includes the monomodal multilingual data, use the preset prompt text as the pending prompt text, use the second source language text as the second pending source text, and use the second target language text as the second pending target text; or, when the sample data includes the multimodal monolingual data, use the text output after the second image is input into the image converter as the pending prompt text, use the text obtained after masking the third target language text as the second pending source text, and use the third target language text as the second pending target text.

[0161] Regarding the apparatus in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0162] Reference below Figure 11, which shows a schematic diagram of the structure of an electronic device 2000 (e.g., a terminal device or server) suitable for implementing the embodiments of the present disclosure. The terminal device in the embodiments of the present disclosure may include, but is not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), in-vehicle terminals (e.g., in-vehicle navigation terminals), etc., as well as fixed terminals such as digital TVs and desktop computers. The server in the embodiments of the present disclosure may include, but is not limited to, local servers, cloud servers, single servers, distributed servers, etc. Figure 11 The electronic device shown is only an example and should not limit the functions and scope of use of the embodiments of the present disclosure.

[0163] like Figure 11 As shown, the electronic device 2000 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 2001, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 2002 or a program loaded from a storage device 2008 into a random access memory (RAM) 2003. Various programs and data required for the operation of the electronic device 2000 are also stored in the RAM 2003. The processing device 2001, the ROM 2002, and the RAM 2003 are connected to each other via a bus 2004. An input / output (I / O) interface 2005 is also connected to the bus 2004.

[0164] Typically, the following devices may be connected to the input / output interface 2005: an input device 2006 including, for example, a touch screen, a touchpad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 2007 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 2008 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 2009. The communication device 2009 may allow the electronic device 2000 to communicate with other devices wirelessly or by wire to exchange data. Although Figure 11 The electronic device 2000 is shown with various devices, but it should be understood that it is not required to implement or possess all of the devices shown. More or fewer devices may be implemented or possessed instead.

[0165] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a non-transitory computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication device 2009, or installed from the storage device 2008, or installed from the ROM 2002. When the computer program is executed by the processing device 2001, the above-mentioned functions defined in the method of the embodiment of the present disclosure are performed.

[0166] It should be noted that the computer-readable medium mentioned above in the present disclosure may be a computer-readable signal medium or a computer-readable storage medium, or any combination of the two. Computer-readable storage media may include, but are not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or components, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present disclosure, a computer-readable storage medium may be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, device, or component. In the present disclosure, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, which carries computer-readable program code. This propagated data signal may take a variety of forms, including, but not limited to, an electromagnetic signal, an optical signal, or any suitable combination of the above. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium that can transmit, propagate, or transport a program for use by or in conjunction with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium may be transmitted using any suitable medium, including but not limited to wires, optical cables, RF (radio frequency), etc., or any suitable combination thereof.

[0167] In some embodiments, the client and server can communicate using any currently known or future developed network protocol, such as HTTP (HyperText Transfer Protocol), and can be interconnected with any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network ("LAN"), a wide area network ("WAN"), an internet (e.g., the Internet), and a peer-to-peer network (e.g., an ad hoc peer-to-peer network), as well as any currently known or future developed network.

[0168] The computer-readable medium may be included in the electronic device, or may exist independently without being incorporated into the electronic device.

[0169] The computer-readable medium carries one or more programs. When the one or more programs are executed by the electronic device, the electronic device: determines a source text to be translated and a source-associated image corresponding to the source text; inputs the source text and the source-associated image into a pre-generated target multimodal translation model to obtain a target translated text output by the target multimodal translation model; wherein the target multimodal translation model is a model generated by training the target multimodal translation model based on sample data, wherein the sample data includes at least two types of data selected from multimodal multilingual data, unimodal multilingual data, and multimodal monolingual data; the multimodal multilingual data packet The multilingual data includes a first source language text, a first target language text, and a first image corresponding to the first source language text; the monomodal multilingual data includes a second source language text and a second target language text; the multimodal monolingual data includes a third target language text and a second image; the first image is an associated image corresponding to the first source language text, and the second image is an associated image corresponding to the third target language text; the first source language text, the second source language text, and the source text are of the same language type; the first target language text, the second target language text, the third target language text, and the target translation text are of the same language type.

[0170] Computer program code for performing the operations of the present disclosure may be written in one or more programming languages, or a combination thereof, including, but not limited to, object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a stand-alone software package, partially on the user's computer and partially on a remote computer, or entirely on the remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., through the Internet using an Internet service provider).

[0171] The flowcharts and block diagrams in the accompanying drawings illustrate the possible implementation architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each box in the flowchart or block diagram can represent a module, program segment, or a part of code, and the module, program segment, or a part of code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order than that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of the boxes in the block diagram and / or flowchart, can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0172] The modules described in the embodiments of the present disclosure may be implemented in software or hardware. In some cases, the name of a module does not necessarily limit the module itself. For example, a determination module may be described as a "module for determining a source text to be translated and a source-associated image corresponding to the source text."

[0173] The functions described above herein may be performed, at least in part, by one or more hardware logic components. For example, and without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chip (SOCs), complex programmable logic devices (CPLDs), and the like.

[0174] In the context of the present disclosure, a machine-readable medium can be a tangible medium that can contain or store a program for use by or in conjunction with an instruction execution system, device or equipment. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium can include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0175] According to one or more embodiments of the present disclosure, a translation method includes:

[0176] Determining a source text to be translated and a source-associated image corresponding to the source text;

[0177] Inputting the source text and the source-associated image into a pre-generated target multimodal translation model to obtain a target translation text output by the target multimodal translation model;

[0178] The target multimodal translation model is a model generated after training a target multimodal translation model based on sample data, the sample data including at least two types of multimodal multilingual data, unimodal multilingual data, and multimodal monolingual data; the multimodal multilingual data includes a first source language text, a first target language text, and a first image corresponding to the first source language text; the unimodal multilingual data includes a second source language text and a second target language text; the multimodal monolingual data includes a third target language text and a second image, the first image being an associated image corresponding to the first source language text, and the second image being an associated image corresponding to the third target language text; the first source language text, the second source language text, and the source text are of the same language type, and the first target language text, the second target language text, the third target language text, and the target translation text are of the same language type.

[0179] According to one or more embodiments of the present disclosure, the target multimodal translation model includes an image encoder, a first text encoder, and a first text decoder; inputting the source text and the source-associated image into a pre-generated target multimodal translation model to obtain a target translated text output by the target multimodal translation model includes:

[0180] Inputting the source-associated image into the image encoder to obtain source-associated image features output by the image encoder;

[0181] Inputting the source text into the first text encoder to obtain first source text features output by the first text encoder;

[0182] Obtaining a target feature by weighting the first source text feature and the source-associated image feature;

[0183] The target features are input into the first text decoder to obtain a target translation text output by the first text decoder.

[0184] According to one or more embodiments of the present disclosure, the pending multimodal translation model includes the image encoder, the first text encoder, and the first text decoder; the target multimodal translation model is generated by:

[0185] obtaining the sample data;

[0186] Looping through the first training step according to the sample data until it is determined that the trained undetermined multimodal translation model satisfies a first preset stopping condition, and using the trained undetermined multimodal translation model as the target multimodal translation model;

[0187] The first training step includes:

[0188] Determining a to-be-determined image sample feature, a first to-be-determined source text, and a first to-be-determined target text according to the sample data;

[0189] Inputting the first undetermined source text into the first text encoder to obtain first text sample features output by the first text encoder;

[0190] Determining a first multimodal sample feature based on the first text sample feature and the first image sample feature;

[0191] Inputting the first multimodal sample feature into the first text decoder to obtain a first translated text output by the first text decoder;

[0192] A first loss value is determined based on the first translation text and the first pending target text. When it is determined based on the first loss value that the pending multimodal translation model does not satisfy the first preset stop iteration condition, the parameters of the pending multimodal translation model are updated based on the first loss value to obtain a trained pending multimodal translation model, and the trained pending multimodal translation model is used as a new pending multimodal translation model.

[0193] According to one or more embodiments of the present disclosure, determining the undetermined image sample features, the first undetermined source text, and the first undetermined target text according to the sample data includes:

[0194] In a case where the sample data includes the multimodal multilingual data, image features output after the first image is input into the image encoder are used as the undetermined image sample features, the first source language text is used as the first undetermined source text, and the first target language text is used as the first undetermined target text, or,

[0195] In a case where the sample data includes the unimodal multilingual data, using a preset image sample feature as the undetermined image sample feature, using the second source language text as the first undetermined source text, and using the second target language text as the first undetermined target text; or

[0196] When the sample data includes the multimodal monolingual data, image features output after the second image is input into the image encoder are used as the undetermined image sample features, text obtained after masking the third target language text is used as the first undetermined source text, and the third target language text is used as the first undetermined target text.

[0197] According to one or more embodiments of the present disclosure, the target multimodal translation model includes an image converter, a second text encoder, and a second text decoder; inputting the source text and the source-associated image into a pre-generated target multimodal translation model to obtain a target translated text output by the target multimodal translation model includes:

[0198] Inputting the source associated image into the image converter to obtain the target prompt text output by the image converter;

[0199] Inputting the source text and the target prompt text into the second text encoder to obtain a second source text feature output by the text encoder;

[0200] The second source text feature is input into the second text decoder to obtain a target translation text output by the text decoder.

[0201] According to one or more embodiments of the present disclosure, the pending multimodal translation model includes the image converter, the second text encoder, and the second text decoder; the target multimodal translation model is generated by:

[0202] obtaining the sample data;

[0203] Looping through the second training step based on the sample data until it is determined that the trained undetermined multimodal translation model satisfies a second preset stopping condition, and using the trained undetermined multimodal translation model as the target multimodal translation model;

[0204] The second training step includes:

[0205] Determining a pending prompt text, a second pending source text, and a second pending target text according to the sample data;

[0206] Inputting the second undetermined source text and the undetermined prompt text into the second text encoder to obtain second text sample features output by the second text encoder;

[0207] Inputting the second text sample feature into the second text decoder to obtain a second translated text output by the second text decoder;

[0208] A second loss value is determined based on the second translated text and the second pending target text. When it is determined based on the second loss value that the pending multimodal translation model does not satisfy the second preset stop iteration condition, the parameters of the pending multimodal translation model are updated based on the second loss value to obtain a trained pending multimodal translation model, and the trained pending multimodal translation model is used as a new pending multimodal translation model.

[0209] According to one or more embodiments of the present disclosure, determining the pending prompt text, the second pending source text, and the second pending target text according to the sample data includes:

[0210] In a case where the sample data includes the multimodal multilingual data, the text output after the first image is input into the image converter is used as the pending prompt text, the first source language text is used as the second pending source text, and the first target language text is used as the second pending target text; or

[0211] In the case where the sample data includes the unimodal multilingual data, a preset prompt text is used as the pending prompt text, the second source language text is used as the second pending source text, and the second target language text is used as the second pending target text; or

[0212] When the sample data includes the multimodal monolingual data, the text output after the second image is input into the image converter is used as the pending prompt text, the text obtained after masking the third target language text is used as the second pending source text, and the third target language text is used as the second pending target text.

[0213] According to one or more embodiments of the present disclosure, a translation device is provided, comprising:

[0214] a determination module, configured to determine a source text to be translated and a source-associated image corresponding to the source text;

[0215] a translation module, configured to input the source text and the source associated image into a pre-generated target multimodal translation model to obtain a target translated text output by the target multimodal translation model;

[0216] The target multimodal translation model is a model generated after training a target multimodal translation model based on sample data, the sample data including at least two types of multimodal multilingual data, unimodal multilingual data, and multimodal monolingual data; the multimodal multilingual data includes a first source language text, a first target language text, and a first image corresponding to the first source language text; the unimodal multilingual data includes a second source language text and a second target language text; the multimodal monolingual data includes a third target language text and a second image, the first image being an associated image corresponding to the first source language text, and the second image being an associated image corresponding to the third target language text; the first source language text, the second source language text, and the source text are of the same language type, and the first target language text, the second target language text, the third target language text, and the target translation text are of the same language type.

[0217] The above description is merely a preferred embodiment of the present disclosure and an illustration of the technical principles employed. Those skilled in the art should understand that the scope of disclosure involved in the present disclosure is not limited to the technical solutions formed by the specific combination of the above-mentioned technical features, but also includes other technical solutions formed by any combination of the above-mentioned technical features or their equivalents without departing from the above-mentioned disclosed concepts. For example, a technical solution formed by replacing the above-mentioned features with (but not limited to) technical features with similar functions disclosed in this disclosure.

[0218] In addition, although each operation is described in a specific order, this should not be understood as requiring these operations to be performed in the specific order shown or in a sequential order. Under certain circumstances, multitasking and parallel processing may be advantageous. Similarly, although some specific implementation details have been included in the above discussion, these should not be interpreted as limiting the scope of the present disclosure. Some features described in the context of a separate embodiment can also be implemented in a single embodiment in combination. On the contrary, the various features described in the context of a single embodiment can also be implemented in multiple embodiments individually or in any suitable sub-combination mode.

[0219] Although the subject matter has been described using language specific to structural features and / or methodological logical acts, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or acts described above. Rather, the specific features and acts described above are merely example forms of implementing the claims. Regarding the apparatus in the above-described embodiments, the specific manner in which each module performs operations has been described in detail in the embodiments related to the method and will not be elaborated upon here.

Claims

1. A translation method, characterized in that: The method comprises: Determining a source text to be translated and a source-associated image corresponding to the source text; the source-associated image is used to indicate all or part of the semantics expressed by the source text; Inputting the source text and the source-associated image into a pre-generated target multimodal translation model to obtain a target translation text output by the target multimodal translation model; the target multimodal translation model extracting prompt keywords from the source-associated image to obtain a target prompt text, wherein the target prompt text is used to assist in translating the source text; The target multimodal translation model is a model generated after training a target multimodal translation model based on sample data, the sample data including multimodal multilingual data, unimodal multilingual data, and multimodal monolingual data; the multimodal multilingual data includes a first source language text, a first target language text, and a first image corresponding to the first source language text; the unimodal multilingual data includes a second source language text and a second target language text; the multimodal monolingual data includes a third target language text and a second image, the first image being an associated image corresponding to the first source language text, and the second image being an associated image corresponding to the third target language text; the first source language text, the second source language text, and the source text are of the same language type, and the first target language text, the second target language text, the third target language text, and the target translation text are of the same language type; Wherein, when the sample data includes the multimodal monolingual data, the third target language text is masked to obtain a first pending source text or a second pending source text, and both the first pending source text and the second pending source text are used to input the pending multimodal translation model to train the target multimodal translation model.

2. The method according to claim 1, characterized in that The target multimodal translation model includes an image encoder, a first text encoder, and a first text decoder; inputting the source text and the source-associated image into a pre-generated target multimodal translation model to obtain a target translated text output by the target multimodal translation model includes: Inputting the source-associated image into the image encoder to obtain source-associated image features output by the image encoder; Inputting the source text into the first text encoder to obtain first source text features output by the first text encoder; Obtaining a target feature by weighting the first source text feature and the source-associated image feature; The target features are input into the first text decoder to obtain a target translation text output by the first text decoder.

3. The method according to claim 2, characterized in that The pending multimodal translation model includes the image encoder, the first text encoder, and the first text decoder; the target multimodal translation model is generated by: obtaining the sample data; Looping through the first training step according to the sample data until it is determined that the trained undetermined multimodal translation model satisfies a first preset stopping condition, and using the trained undetermined multimodal translation model as the target multimodal translation model; The first training step includes: Determining a to-be-determined image sample feature, a first to-be-determined source text, and a first to-be-determined target text according to the sample data; Inputting the first undetermined source text into the first text encoder to obtain first text sample features output by the first text encoder; Determining a first multimodal sample feature based on the first text sample feature and the first image sample feature; Inputting the first multimodal sample feature into the first text decoder to obtain a first translated text output by the first text decoder; A first loss value is determined based on the first translation text and the first pending target text. When it is determined based on the first loss value that the pending multimodal translation model does not satisfy the first preset stop iteration condition, the parameters of the pending multimodal translation model are updated based on the first loss value to obtain a trained pending multimodal translation model, and the trained pending multimodal translation model is used as a new pending multimodal translation model.

4. The method according to claim 3, characterized in that The determining of the undetermined image sample features, the first undetermined source text, and the first undetermined target text according to the sample data includes: In a case where the sample data includes the multimodal multilingual data, image features output after the first image is input into the image encoder are used as the undetermined image sample features, the first source language text is used as the first undetermined source text, and the first target language text is used as the first undetermined target text, or, In the case where the sample data includes the unimodal multilingual data, using a preset image sample feature as the undetermined image sample feature, using the second source language text as the first undetermined source text, and using the second target language text as the first undetermined target text; or When the sample data includes the multimodal monolingual data, image features output after the second image is input into the image encoder are used as the undetermined image sample features, text obtained after masking the third target language text is used as the first undetermined source text, and the third target language text is used as the first undetermined target text.

5. The method according to claim 1, characterized in that The target multimodal translation model includes an image converter, a second text encoder, and a second text decoder; the step of inputting the source text and the source-associated image into a pre-generated target multimodal translation model to obtain a target translated text output by the target multimodal translation model includes: Inputting the source associated image into the image converter to obtain the target prompt text output by the image converter; Inputting the source text and the target prompt text into the second text encoder to obtain a second source text feature output by the text encoder; The second source text feature is input into the second text decoder to obtain a target translation text output by the text decoder.

6. The method according to claim 5, characterized in that The pending multimodal translation model includes the image converter, the second text encoder, and the second text decoder; the target multimodal translation model is generated by: obtaining the sample data; Looping through the second training step based on the sample data until it is determined that the trained undetermined multimodal translation model satisfies a second preset stopping condition, and using the trained undetermined multimodal translation model as the target multimodal translation model; The second training step includes: Determining a pending prompt text, a second pending source text, and a second pending target text according to the sample data; Inputting the second undetermined source text and the undetermined prompt text into the second text encoder to obtain second text sample features output by the second text encoder; Inputting the second text sample feature into the second text decoder to obtain a second translated text output by the second text decoder; A second loss value is determined based on the second translated text and the second pending target text. When it is determined based on the second loss value that the pending multimodal translation model does not satisfy the second preset stop iteration condition, the parameters of the pending multimodal translation model are updated based on the second loss value to obtain a trained pending multimodal translation model, and the trained pending multimodal translation model is used as a new pending multimodal translation model.

7. The method according to claim 6, characterized in that Determining the pending prompt text, the second pending source text, and the second pending target text according to the sample data includes: In a case where the sample data includes the multimodal multilingual data, the text output after the first image is input into the image converter is used as the pending prompt text, the first source language text is used as the second pending source text, and the first target language text is used as the second pending target text; or In the case where the sample data includes the unimodal multilingual data, a preset prompt text is used as the pending prompt text, the second source language text is used as the second pending source text, and the second target language text is used as the second pending target text; or When the sample data includes the multimodal monolingual data, the text output after the second image is input into the image converter is used as the pending prompt text, the text obtained after masking the third target language text is used as the second pending source text, and the third target language text is used as the second pending target text.

8. A translation device, characterized in that: The device comprises: a determination module, configured to determine a source text to be translated and a source-associated image corresponding to the source text; the source-associated image is configured to indicate all or part of the semantics expressed by the source text; a translation module configured to input the source text and the source-associated image into a pre-generated target multimodal translation model to obtain a target translated text output by the target multimodal translation model; the target multimodal translation model extracts prompt keywords from the source-associated image to obtain a target prompt text, wherein the target prompt text is used to assist in translating the source text; The target multimodal translation model is a model generated after training a target multimodal translation model based on sample data, the sample data including multimodal multilingual data, unimodal multilingual data, and multimodal monolingual data; the multimodal multilingual data includes a first source language text, a first target language text, and a first image corresponding to the first source language text; the unimodal multilingual data includes a second source language text and a second target language text; the multimodal monolingual data includes a third target language text and a second image, the first image being an associated image corresponding to the first source language text, and the second image being an associated image corresponding to the third target language text; the first source language text, the second source language text, and the source text are of the same language type, and the first target language text, the second target language text, the third target language text, and the target translation text are of the same language type; Wherein, when the sample data includes the multimodal monolingual data, the third target language text is masked to obtain a first pending source text or a second pending source text, and both the first pending source text and the second pending source text are used to input the pending multimodal translation model to train the target multimodal translation model.

9. A computer-readable medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processing device, the steps of the method according to any one of claims 1 to 7 are implemented.

10. An electronic device, characterized in that: include: a storage device having a computer program stored thereon; A processing device, configured to execute the computer program in the storage device to implement the steps of the method according to any one of claims 1 to 7.

Images