Training method, device and electronic device for multi-language multi-modal pre-training model

Abstract

The present disclosure relates to a training method, device and electronic device for a multilingual multimodal pre-training model, and in particular to the field of machine learning technology. The method comprises: obtaining a paired sample set, wherein the paired sample set comprises: image-text paired data of an image and a target text, and parallel paired data of a first language text and a second language text; pre-training the paired sample set based on a unified multilingual multimodal model framework to obtain a multilingual multimodal pre-training model; wherein the target text is of any language type, and the language types of the first language text and the second language text are different.

Description

Technical Field

[0001] The present disclosure relates to the technical field of machine learning, and in particular to a training method, device and electronic device for a multi-language multi-modal pre-training model. Background Art

[0002] Image-text pairs in some languages ​​with rich data resources are sufficient for multimodal pre-training. However, it is difficult to collect enough training data in some languages ​​with less data resources, making it impossible to apply multimodal pre-training technology to various languages. A model with multimodal understanding capabilities in multiple languages ​​is urgently needed. Summary of the invention

[0003] In order to solve the above technical problems or at least partially solve the above technical problems, the present disclosure provides a training method, device and electronic device for a multi-language multi-modal pre-training model, and the multi-language multi-modal pre-training model can realize the multi-language multi-modal understanding ability. In order to achieve the above purpose, the technical solutions provided by the embodiments of the present disclosure are as follows:

[0004] In a first aspect, a training method for a multi-language multi-modal pre-training model is provided, comprising:

[0005] Acquire a paired sample set, wherein the paired sample set includes: image-text paired data of an image and a target text, and parallel paired data of a first language text and a second language text;

[0006] Pre-training the paired sample set based on a unified multilingual multimodal model framework to obtain a multilingual multimodal pre-trained model;

[0007] The target text is in any language type, and the first language text and the second language text are in different language types.

[0008] As an optional implementation of the embodiment of the present disclosure, the paired sample set is pre-trained based on a unified multi-language multi-modal model framework to obtain a multi-language multi-modal pre-trained model, including:

[0009] The image-text paired data and the parallel paired data are pre-trained alternately based on a unified multilingual multimodal model framework to obtain the multilingual multimodal pre-trained model.

[0010] As an optional implementation of the embodiment of the present disclosure, the training objectives of pre-training the paired sample set based on the unified multilingual multimodal model framework include:

[0011] Contrastive learning on target paired data;

[0012] and / or,

[0013] The target paired data includes masked text, and the masked text in the target paired data is restored;

[0014] Wherein, the target pairing data includes the image-text pairing data or the parallel pairing data.

[0015] As an optional implementation of the embodiment of the present disclosure, the pre-training based on the unified multi-language multi-modal model framework through the paired sample set to obtain a multi-language multi-modal pre-training model includes:

[0016] Encoding the target paired data to obtain first encoded information and second encoded information, wherein the target paired data includes the image-text paired data or the parallel paired data;

[0017] Inputting the first encoded information and the second encoded information into an N-layer feature fusion model, where N is an integer greater than or equal to 1;

[0018] Inputting the first output result of the N-layer feature fusion model into a linear layer for processing to obtain a first processing result, and determining a first loss parameter based on the first processing result and a first loss function;

[0019] Based on the first loss parameter, the model parameters of the unified multilingual multimodal model framework are updated.

[0020] As an optional implementation of the embodiment of the present disclosure, the first loss function includes: a matching loss function and / or a CMLM loss function.

[0021] As an optional implementation of the embodiment of the present disclosure, the method further includes:

[0022] Determine a second loss parameter based on the first encoding information, the second encoding information and a second loss function;

[0023] Based on the second loss parameter, the model parameters of the unified multilingual multimodal model framework are updated.

[0024] As an optional implementation of the embodiment of the present disclosure, the second loss function is a contrastive learning loss function.

[0025] As an optional implementation of the embodiment of the present disclosure, the target pairing data is the image-text pairing data, the first encoding information is the encoding information of the image, and the second encoding information is the encoding information of the target text;

[0026] The target paired data is the parallel paired data, the first encoding information is the encoding information of the first language text, and the second encoding information is the encoding information of the second language text.

[0027] In a second aspect, a training device for a multi-language multi-modal pre-training model is provided, comprising:

[0028] An acquisition module, used to acquire a paired sample set, wherein the paired sample set includes: image-text paired data of an image and a target text, and parallel paired data of a first language text and a second language text;

[0029] A pre-training module, used for pre-training the paired sample set based on a unified multi-language multi-modal model framework to obtain a multi-language multi-modal pre-training model;

[0030] The target text is in any language type, and the first language text and the second language text are in different language types.

[0031] As an optional implementation of the embodiment of the present disclosure, the pre-training module is specifically used for:

[0032] The image-text paired data and the parallel paired data are pre-trained alternately based on a unified multilingual multimodal model framework to obtain the multilingual multimodal pre-trained model.

[0033] As an optional implementation of the embodiment of the present disclosure, the training objectives of pre-training the paired sample set based on the unified multilingual multimodal model framework include:

[0034] Contrastive learning on target paired data;

[0035] and / or,

[0036] The target paired data includes masked text, and the masked text in the target paired data is restored;

[0037] Wherein, the target pairing data includes the image-text pairing data or the parallel pairing data.

[0038] As an optional implementation of the embodiment of the present disclosure, the pre-training module includes:

[0039] An encoder, used to encode target paired data to obtain first encoded information and second encoded information, wherein the target paired data includes the image-text paired data or the parallel paired data;

[0040] An N-layer feature fusion model, used for receiving the first encoded information and the second encoded information as input, and outputting a first input result, where N is an integer greater than or equal to 1;

[0041] A linear layer, configured to receive a first output result and process the first output result to obtain a first processing result;

[0042] A loss calculation module, used for determining a first loss parameter based on the first processing result and a first loss function;

[0043] An updating module is used to update the model parameters of the unified multilingual multimodal model framework based on the first loss parameter.

[0044] As an optional implementation of the embodiment of the present disclosure, the first loss function includes: a matching loss function and / or a CMLM loss function.

[0045] As an optional implementation of the embodiment of the present disclosure, the loss calculation module is further used to: determine a second loss parameter based on the first encoding information, the second encoding information and a second loss function;

[0046] The updating module is further used to: update the model parameters of the unified multilingual multimodal model framework based on the second loss parameter.

[0047] As an optional implementation of the embodiment of the present disclosure, the second loss function is a contrastive learning loss function.

[0048] As an optional implementation of the embodiment of the present disclosure, the target pairing data is the image-text pairing data, the first encoding information is the encoding information of the image, and the second encoding information is the encoding information of the target text;

[0049] The target paired data is the parallel paired data, the first encoding information is the encoding information of the first language text, and the second encoding information is the encoding information of the second language text.

[0050] In a third aspect, an electronic device is provided, comprising: a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein when the computer program is executed by the processor, the training method for a multilingual multimodal pre-training model as described in the first aspect or any one of its optional embodiments is implemented.

[0051] In a fourth aspect, a computer-readable storage medium is provided, comprising: a computer program is stored on the computer-readable storage medium, and when the computer program is executed by a processor, the training method of the multilingual multimodal pre-training model as described in the first aspect or any one of its optional embodiments is implemented.

[0052] In a fifth aspect, a computer program product is provided, characterized in that it includes: when the computer program product is run on a computer, the computer implements the training method of the multilingual multimodal pre-training model as described in the first aspect or any one of its optional embodiments.

[0053] The training method of the multilingual multimodal pre-training model provided by the embodiment of the present disclosure first obtains a paired sample set, wherein the paired sample set includes: image-text paired data of an image and a target text, and parallel paired data of a first language text and a second language text; and then the paired sample set is pre-trained based on a unified multilingual multimodal model framework to obtain a multilingual multimodal pre-training model wherein the target text is of any language type, and the language type of the first language text and the second language text are different. Through this scheme, since the image-text paired data includes images and texts with related meanings, and the parallel paired data includes two different language texts with similar semantics, it can be seen that the image-text paired data and the parallel paired data are paired data for two representations of the same substance, so the image-text paired data and the parallel paired data are pre-trained based on a unified multilingual multimodal model framework to obtain a multilingual multimodal pre-training model, so that if the paired sample set includes text corresponding to a language type with rich data resources, the multimodal understanding ability of the language type with rich data resources can be transferred to other language types, so that the obtained multilingual multimodal pre-training model can achieve multilingual multimodal understanding ability.

[0054] Furthermore, this approach does not require the translation of large amounts of data, which can reduce resource consumption and improve the accuracy of the trained multilingual multimodal pre-trained model. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the present disclosure.

[0056] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0057] Figure 1 A flowchart of a training method for a multi-language multi-modal pre-training model provided in an embodiment of the present disclosure;

[0058] Figure 2 A schematic diagram of pre-training based on a unified multi-language multi-modal model framework provided in an embodiment of the present disclosure;

[0059] Figure 3 A schematic diagram of a unified multi-language multi-modal model framework provided for an embodiment of the present disclosure;

[0060] Figure 4 A schematic diagram of a flow chart of another method for training a multi-language multi-modal pre-training model provided in an embodiment of the present disclosure;

[0061] Figure 5 A schematic diagram of fine-tuning downstream tasks on the multi-language multi-modal pre-training model provided in an embodiment of the present disclosure;

[0062] Figure 6 A schematic diagram of a visual question answering task provided by an embodiment of the present disclosure;

[0063] Figure 7 A schematic diagram of a visual reasoning task provided by an embodiment of the present disclosure;

[0064] Figure 8 A structural block diagram of a training device for a multi-language multi-modal pre-training model provided in an embodiment of the present disclosure;

[0065] Fig. 9 A schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0066] In order to more clearly understand the above-mentioned objectives, features and advantages of the present disclosure, the scheme of the present disclosure will be further described below. It should be noted that the embodiments of the present disclosure and the features in the embodiments can be combined with each other without conflict.

[0067] In the following description, many specific details are set forth to facilitate a full understanding of the present disclosure, but the present disclosure may also be implemented in other ways different from those described herein; it is obvious that the embodiments in the specification are only part of the embodiments of the present disclosure, rather than all of the embodiments.

[0068] Multimodality refers to information in multiple modes, including visual information, language information, etc. Multimodal understanding tasks refer to understanding visual information (vision) and language information (language) at the same time to solve related problems, such as: cross-modal retrieval tasks (image-text retrieval), visual question answering tasks (visual question answering), visual reasoning tasks (visual reasoning), etc. Pre-trained multimodal models are usually used to improve the effect of downstream multimodal understanding tasks.

[0069] Image-text pairs in some languages ​​with rich data resources are sufficient for multimodal pre-training. However, it is difficult to collect sufficient training data in some languages ​​with less data resources, making it impossible to apply multimodal pre-training technology to various languages.

[0070] Since multilingual multimodal pre-training (i.e., multimodal pre-training in multiple languages) lacks direct training data, i.e., image and text pairing data in various languages, the image and text pairing data in various languages ​​can be replaced mainly in the following two ways:

[0071] (1) Work represented by Learning Universal Representations via Multitask Multilingual Multimodal Pre-training (M3P) uses paired data of images and English texts and non-parallel multilingual data as substitutes. M3P combines multilingual pre-training and multimodal pre-training through multi-task pre-training objectives to map objects that appear in different ways or texts expressed in different languages ​​into a common semantic space. This method uses English as a fulcrum and uses multi-task training shared parameters to try to transfer English multimodal capabilities to other languages.

[0072] Among them, non-parallel multilingual data refers to text data in different languages ​​that are unrelated.

[0073] (2) Work represented by Universal Cross-lingual Cross-modal Vision-and-Language Pre-training (UC2) uses a translation system to translate English image-text pairing data into other languages, thereby constructing image-text pairing data in various languages ​​for training.

[0074] Among them, in order to solve the scarcity problem of multi-language subtitles of image datasets, the existing English-only dataset is first expanded with other languages ​​through machine translation (MT), and then image-text pairing data in various languages ​​is constructed.

[0075] Both of the above methods belong to multi-language and multi-modal multi-task training. Through parameter sharing, it is hoped that the model can transfer the multi-modal understanding ability of high-resource languages ​​(for example, English / Chinese) to other low-resource languages. It is generally believed that the experimental results of the trained multi-language multi-modal pre-training model need to be much better than the results of the translation test (translate-test) in order to meet the accuracy requirements when applied in practice. However, according to the latest research on the transfer learning benchmark across modalities, tasks, and languages ​​(IGLUE), after a detailed experimental analysis of the multi-language multi-modal pre-training models trained by the above two methods, it was found that the experimental results were far worse than the accuracy of the "translate-test", so the above two methods are difficult to apply in practice.

[0076] In order to solve the above problems, the embodiments of the present disclosure provide a training method, device and electronic device for a multilingual multimodal pre-training model. Since the image-text pairing data includes pictures and texts with related meanings, and the parallel pairing data includes texts in two different languages ​​with similar semantics, it can be seen that the image-text pairing data and the parallel pairing data are both pairing data for two representations of the same substance. Therefore, the image-text pairing data and the parallel pairing data are pre-trained based on a unified multilingual multimodal model framework to obtain a multilingual multimodal pre-training model. In this way, if the paired sample set includes texts corresponding to a language type with rich data resources, the multimodal understanding ability of the language type with rich data resources can be transferred to other language types, so that the obtained multilingual multimodal pre-training model can achieve multilingual multimodal understanding ability. This method does not require the translation of a large amount of data, can reduce resource consumption, and improve the accuracy of the trained multilingual multimodal pre-training model.

[0077] In the embodiment of the present disclosure, the training method of the multi-language multi-modal pre-training model can be applied to the training device and electronic device of the multi-language multi-modal pre-training model, and the training device of the multi-language multi-modal pre-training model can be a functional module or functional entity in the electronic device for implementing the training method of the multi-language multi-modal pre-training model. Wherein, the electronic device includes but is not limited to: a server, a personal computer, a laptop, a tablet computer, a smart phone, etc. The computer device includes a user device and a network device. Wherein, the user device includes but is not limited to a computer, a smart phone, a tablet computer, etc.; the network device includes but is not limited to a single network server, a server group composed of multiple network servers, or a cloud composed of a large number of computers or network servers in cloud computing, wherein cloud computing is a kind of distributed computing, a super virtual computer composed of a group of loosely coupled computer sets. Wherein, the computer device can be operated alone to implement the present disclosure, and can also be connected to the network and implement the present disclosure through interactive operations with other computer devices in the network. Wherein, the network where the computer device is located includes but is not limited to the Internet, a wide area network, a metropolitan area network, a local area network, a virtual private network (Virtual Private Network, VPN) network, etc.

[0078] like Figure 1 FIG. 1 is a flow chart of a method for training a multi-language multi-modal pre-training model provided by an embodiment of the present disclosure, the method comprising:

[0079] 101. Obtain a paired sample set.

[0080] The paired sample set includes: image-text paired data of an image and a target text, and parallel paired data of a first language text and a second language text.

[0081] The target text is of any language type, and the first language text and the second language text are of different language types.

[0082] The above parallel paired data refers to the first language text and the second language text having the same meaning but different language types. The above target text and the first language text may be texts with similar semantics, or the target text and the first language text may be texts with unrelated semantics.

[0083] Exemplarily, the picture-text pairing data may be the pairing data of a picture and English text, and the parallel pairing data may be the pairing data of English text and Chinese text.

[0084] 102. Pre-train the paired sample set based on a unified multilingual multimodal model framework to obtain a multilingual multimodal pre-trained model.

[0085] In some embodiments, the paired sample set is pre-trained based on a unified multilingual multimodal model framework to obtain a multilingual multimodal pre-trained model, including: pre-training image-text paired data and parallel paired data alternately based on a unified multilingual multimodal model framework to obtain a multilingual multimodal pre-trained model.

[0086] The above-mentioned pre-training of the image-text pairing data and the parallel pairing data alternately based on the unified multilingual multimodal model framework includes: first pre-training the image-text pairing data based on the unified multilingual multimodal model framework, then pre-training the parallel pairing data based on the unified multilingual multimodal model framework, and then pre-training the image-text pairing data based on the unified multilingual multimodal model framework again, and thus cyclically pre-training based on the unified multilingual multimodal model framework; or, first pre-training the parallel pairing data based on the unified multilingual multimodal model framework, then pre-training the image-text pairing data based on the unified multilingual multimodal model framework, and then pre-training the parallel pairing data based on the unified multilingual multimodal model framework again, and thus cyclically pre-training based on the unified multilingual multimodal model framework. This alternating training method can achieve better training results.

[0087] It should be noted that the paired sample set in the embodiment of the present disclosure may include a large amount of image-text paired data and a large amount of parallel paired data. During the pre-training process, the image-text paired data and parallel paired data input each time may be different from the previous one.

[0088] For example, Figure 2 FIG. 1 is a schematic diagram of pre-training based on a unified multi-language multi-modal model framework provided by an embodiment of the present disclosure. Figure 2 It can be seen that the input of the unified multilingual multimodal model framework includes: picture-text pairing data and parallel pairing data. Among them, each round of pre-training can input one of the picture-text pairing data and parallel pairing data.

[0089] like Figure 3 As shown, it is a schematic diagram of a unified multilingual multimodal model framework provided in an embodiment of the present disclosure, and the unified multilingual multimodal model framework includes: a text encoder 301, an image / text encoder 302, a feature fusion model 303, and the feature fusion model (Transformer model) 303 includes a self-attention layer 3031, a cross-attention layer 3032, and a feed forward layer (feed forward) 3033, and a linear layer 304.

[0090] The text encoder 301 may be a cross-language text encoder, and the image / text encoder 302 may be an encoder having the functions of a text encoder and an image encoder. The feature fusion model 303 is a Transformer model, and may be an N-layer Transformer model.

[0091] The self-attention layer 3031 in the above Transformer model receives the text encoding information obtained after encoding by the text encoder 301, and uses the self-attention mechanism to learn the internal relationship of the target sentence (i.e., the text encoding information), and then outputs it to the cross-attention layer 3032. The cross-attention layer 3032 receives the image / text encoding information obtained after encoding by the image / text encoder 302. The cross-attention layer 3032 learns the relationship between the image / text encoding information and the text encoding information through the cross-attention mechanism (encoder-decoder rattattention), and inputs the learning result to the forward propagation layer 3033 for decoding, so as to obtain the decoded output, and inputs the decoded output to the linear layer 304 for processing to obtain the final output.

[0092] Among them, the matching loss parameters and / or CMLM loss parameters can be calculated by using the processing results after processing by the linear layer 304 and the matching loss function and / or the conditioned masked language modeling (CMLM) loss function, the text encoding information obtained after encoding by the text encoder 301 and the image / text encoding information obtained after encoding by the image / text encoder 302 and the contrastive learning loss function, and the contrastive learning loss parameters can be calculated.

[0093] In some embodiments, Figure 4 As shown, it is a flowchart of another training method of a multilingual multimodal pre-training model provided by an embodiment of the present disclosure. In the process of pre-training based on the unified multilingual multimodal model framework through the paired sample set to obtain the multilingual multimodal pre-training model, the following steps 401 to 406 need to be performed once or multiple times:

[0094] 401. Encode target paired data to obtain first encoding information and second encoding information.

[0095] The target pairing data includes image-text pairing data or parallel pairing data.

[0096] In some embodiments, when the target paired data is image-text paired data, the first encoding information is encoding information of the image, and the second encoding information is encoding information of the target text.

[0097] Combined with the above Figure 3 As shown, when the target paired data is image-text paired data, the image can be input into the image / text encoder 302 to be encoded to obtain first encoded information, and the target text can be input into the text encoder 303 to be encoded to obtain second encoded information.

[0098] In some embodiments, when the target paired data is parallel paired data, the first encoding information is encoding information of a first language text, and the second encoding information is encoding information of a second language text.

[0099] Combined with the above Figure 3 As shown, when the target paired data is image-text paired data, the first language text can be input into the image / text encoder 302 to be encoded to obtain first encoded information, and the second language text can be input into the text encoder 303 to be encoded to obtain second encoded information.

[0100] 402. Input the first encoding information and the second encoding information into an N-layer feature fusion model.

[0101] Wherein, N is an integer greater than or equal to 1.

[0102] The above-mentioned inputting the first coded information and the second coded information into the N-layer feature fusion model may be inputting the first coded information into Figure 3 The cross attention layer 3032 in the feature fusion model 303 shown in FIG. 3 inputs the second encoded information into Figure 3 The self-attention layer 3031 in the feature fusion model 303 is shown.

[0103] 403. Input the first output result of the feature fusion model into the linear layer for processing to obtain a first processing result, and determine a first loss parameter based on the first processing result and the first loss function.

[0104] Among them, the first loss function includes: a matching loss function and / or a CMLM loss function.

[0105] 404. Based on the first loss parameter, update the model parameters of the unified multilingual multimodal model framework.

[0106] Optionally, the following steps 5 and 6 may also be included:

[0107] 405. Determine a second loss parameter based on the first encoding information, the second encoding information, and the second loss function.

[0108] 406. Based on the second loss parameter, update the model parameters of the unified multilingual multimodal model framework.

[0109] Among them, the second loss function is the contrastive learning loss function.

[0110] The training objectives of pre-training the paired sample set based on the unified multilingual multimodal model framework include at least one of the following training objectives:

[0111] a) Contrastive learning on target paired data;

[0112] The training goal can be achieved by at least one of the following methods, including but not limited to:

[0113] One way is to calculate the contrastive learning loss parameter by the contrastive learning loss function, and update the model parameters of the unified multilingual multimodal model framework based on the contrastive learning loss parameter.

[0114] Another way is to calculate the matching loss parameter through the matching loss function, and update the model parameters of the unified multilingual multimodal model framework based on the matching loss parameter.

[0115] b) Restore the masked text in the target paired data.

[0116] Among them, the target paired data includes masked text, the CMLM loss parameters are calculated based on the CMLM loss function, and the model parameters of the unified multilingual multimodal model framework are updated based on the CMLM loss parameters, so that the masked text can be restored based on pictures or text in another language.

[0117] Since the image-text pairing data includes pictures and texts with related meanings, and the parallel pairing data includes texts in two different languages ​​with similar semantics, it can be seen that both the image-text pairing data and the parallel pairing data are pairing data for two representations of the same substance, and the multilingual multimodal pre-training process in the disclosed embodiment is to bring the two representations of the same substance closer together and optimize the maximum mutual information between the two representations of the same substance. Based on this, the disclosed embodiment proposes to perform pre-training based on image-text pairing data and parallel pairing data through a unified multilingual multimodal model framework, so that the connection between the same vocabulary / concepts in different languages ​​can be brought closer through the input of parallel pairing data, which can effectively help transfer the multimodal understanding ability of high-resource languages ​​(for example: English / Chinese) to other low-resource languages.

[0118] If two random variables A and B are given, I(A,B) represents the mutual information of the two random variables A and B, that is, it represents the correlation between them. Here, the definition A=a, B=b is two different representations of the same essence, and these two representations can be image-text paired data or parallel paired data. The loss parameters corresponding to the above three loss functions (contrastive learning loss function, CMLM loss function and matching loss function) used in the present invention can all be classified as maximizing the lower bound of the mutual information I(A,B), that is, minimizing the information noise contrast estimation loss (info Noise Contrastive Estimation loss, InfoNCE).

[0119] The InfoNCE function is shown in the following formula (1):

[0120]

[0121] Among them, f θ is a function based on the parameter θ, f θ ∈R, R is the set of real numbers, is a set containing a positive sample b and negative samples, E p(A,) represents expectation, exp(*) is the exponential power function e * .

[0122] The above contrastive learning loss function is a symmetric form of the above InfoNCE loss. The above contrastive learning loss function is shown in the following formula (2):

[0123]

[0124] in, are the sizes of A and B sets respectively, and are equal to the batch size of training data. p(A,B) represents expectation, exp(*) is the exponential power function e * .

[0125] When the unified multilingual multimodal model framework input is image-text pairing data, function When the model input is parallel paired data,

[0126] in, and are the [CLS] output vectors of the image / text encoder and the text encoder at their respective special symbol [CLS] positions, g v and g ware two functions that map the [CLS] output vector to lower-dimensional feature representations, respectively, and τ is a learnable parameter, wherein the symbol [CLS] is used to receive and summarize useful information from all positions, and the contextual representation of [CLS] will be used as the representation of sentences in downstream tasks. In the disclosed embodiment, the first encoded information obtained by the image / text encoder or the second encoded information obtained by the text encoder is composed of the [CLS] output vector and the output vectors of the corresponding positions of the remaining tokens.

[0127] Similarly, in the [CLS] output vector of the fusion model (i.e. ) can also be regarded as the symmetric form of the InfoNCE loss function. The above matching loss function can be expressed as the following formula (3):

[0128]

[0129] In the matching loss function, one negative sample can be sampled in the current training batch for each positive sample pair (a, b), and then all sample pairs are judged whether the two samples in these sample pairs match (whether they are positive sample pairs). where g m is a multi-layer perceptron network, is a parameterized vector.

[0130] Similarly, the CMLM loss function can also be expressed as maximizing context information and The masked token W in i The mutual information between represents the partially covered text input, b is the corresponding image or translated text (i.e., text in another language), and the CMLM loss function can be expressed as follows:

[0131]

[0132] in, is the fusion model in w i The output vector at position ψ(w) is a lookup function used to map a token w to a parameterized vector, and v represents the complete vocabulary.

[0133] Through the representation of the above formula (2), formula (3) and formula (4), it can be known that the three loss functions in the embodiment of the present disclosure are all for optimizing the maximum mutual information of two representations of the same substance. Based on this, the embodiment of the present disclosure uses a unified multilingual multimodal model framework and performs pre-training based on image-text pairing data and parallel pairing data. In this way, the connection between the same vocabulary / concepts in different languages ​​can be shortened through the input of parallel pairing data, and the maximum mutual information optimization can be performed through the above three loss functions, which can effectively help transfer the multimodal understanding ability of high-resource languages ​​(for example: English / Chinese) to other low-resource languages.

[0134] like Figure 5 As shown in FIG. 1 , it is a schematic diagram of fine-tuning the multi-language multi-modal pre-training model provided in an embodiment of the present disclosure for downstream tasks. Figure 2 After obtaining the multilingual multimodal pre-trained model, the multilingual multimodal pre-trained model can also be fine-tuned for downstream tasks based on the multimodal downstream task corpus, and the fine-tuned multilingual multimodal pre-trained model can be used to implement downstream tasks, which can be multimodal understanding tasks. Among them, the multimodal downstream task corpus includes task corpora (i.e., text) in multiple languages ​​configured for downstream tasks, and the above downstream tasks can include: image and text retrieval tasks (Retrieval), visual document tasks (VQA) and visual reasoning tasks (NLVR^2), etc. Among them, image and text retrieval tasks: examine the model's ability to retrieve related images through text, and its ability to retrieve related text through images; visual question and answer tasks: this task examines the model's ability to answer questions based on images; visual reasoning tasks: this task examines whether the model can judge whether the text correctly describes the image.

[0135] For example, Figure 6 , which is a schematic diagram of a visual question answering task provided by an embodiment of the present disclosure, in which two pictures of animals are given, and the text "Which animal is bigger?" and the corresponding answer "rabbit" are given. By recognizing the picture, the question and answer described in the text can be obtained.

[0136] For example, Figure 7 As shown, it is a schematic diagram of a visual reasoning task provided by an embodiment of the present disclosure. In the task, the text "There are two dogs in the picture on the left" is given, and two pictures on the left and right are given. By combining the text with the two pictures on the left and right, it can be determined whether the text correctly describes the number of dogs in the picture on the left.

[0137] As shown in Table 1 below, on the public multilingual multimodal task list IGLUE, in the zero-shot evaluation method, the multilingual multimodal pre-training method of the disclosed embodiment significantly surpasses the current most advanced models, such as M3P and UC2. In addition, considering all tasks, the multilingual multimodal pre-training method of the disclosed embodiment surpasses the best translate-test effect for the first time, which illustrates the practicality of the multilingual multimodal pre-training method of the disclosed embodiment.

[0138] Among them, zero-shot: fine-tune the multilingual multimodal pre-trained model using English data, and then directly test it on multiple other languages. translate-test: translate data in other languages ​​into English, and then test it using the English model. It is generally believed that zero-shot multilingual multimodal pre-trained models need to exceed "translate-test" before they can be used in practice.

[0139] Table 1

[0140]

[0141] Among them, xGQA stands for Cross-lingual Grounded Question Answering, XVNLI stands for Cross-lingual Visual Natural Language Inference, MaRVL stands for Multicultural Reasoning over Vision and Language, xFlickr&CO stands for multilingual image-text retrieval on Flickr30K and COCO test split, WIT stands for Wikipedia-based Image Text dataset, UNITER stands for Universal Image-TExt Representation Learning, mUNITER stands for UNITER initialized with mBERT, and xUNITER stands for UNITER initialized with XLM-R. XLM-R), XLM-R stands for Crosslingual language model-RoBERTa, RoBERTa stands for Robustly optimized BERT approach, BERT stands for Bidirectional Encoder Representations from Transformers, mBERT stands for multilingual BERT, Visual BERT stands for Visual BERT, and VL-BERT stands for Visual-Linguistic BERT.

[0142] The training method of the multilingual multimodal pre-training model provided by the embodiment of the present disclosure first obtains a paired sample set, wherein the paired sample set includes: image-text paired data of an image and a target text, and parallel paired data of a first language text and a second language text; then the paired sample set is pre-trained based on a unified multilingual multimodal model framework to obtain a multilingual multimodal pre-training model. The target text is of any language type, and the language types of the first language text and the second language text are different. Through this scheme, since the picture-text pairing data includes pictures and texts with related meanings, and the parallel pairing data includes texts in two different languages ​​with similar semantics, it can be seen that both the picture-text pairing data and the parallel pairing data are pairing data for two representations of the same substance. Therefore, the picture-text pairing data and the parallel pairing data are pre-trained based on a unified multilingual multimodal model framework to obtain a multilingual multimodal pre-training model. In this way, if the paired sample set includes texts corresponding to language types with rich data resources, the multimodal understanding ability of language types with rich data resources can be transferred to other language types, so that the obtained multilingual multimodal pre-training model can achieve multilingual multimodal understanding ability. This method does not require the translation of a large amount of data, can reduce resource consumption, and improve the accuracy of the trained multilingual multimodal pre-training model.

[0143] like Figure 8 As shown, a structural block diagram of a training device for a multi-language multi-modal pre-training model provided by an embodiment of the present disclosure, the device includes:

[0144] The acquisition module 801 is used to acquire a paired sample set, wherein the paired sample set includes: image-text paired data of an image and a target text, and parallel paired data of a first language text and a second language text;

[0145] A pre-training module 802 is used to pre-train the paired sample set based on a unified multi-language multi-modal model framework to obtain a multi-language multi-modal pre-training model;

[0146] The target text is in any language type, and the first language text and the second language text are in different language types.

[0147] As an optional implementation of the embodiment of the present disclosure, as an optional implementation of the embodiment of the present disclosure, the pre-training module 802 is specifically used for:

[0148] The image-text paired data and the parallel paired data are pre-trained alternately based on a unified multilingual multimodal model framework to obtain the multilingual multimodal pre-trained model.

[0149] As an optional implementation of the embodiment of the present disclosure, the training objectives of pre-training the paired sample set based on the unified multilingual multimodal model framework include:

[0150] Contrastive learning on target paired data;

[0151] and / or,

[0152] The target paired data includes masked text, and the masked text in the target paired data is restored;

[0153] Wherein, the target pairing data includes the image-text pairing data or the parallel pairing data.

[0154] As an optional implementation of the embodiment of the present disclosure, the pre-training module 802 includes:

[0155] Encoder 8021, used to encode target pairing data to obtain first encoding information and second encoding information, wherein the target pairing data includes the image-text pairing data or the parallel pairing data;

[0156] N-layer feature fusion model 8022, used to receive the first encoded information and the second encoded information as input, and output a first input result, where N is an integer greater than or equal to 1;

[0157] The linear layer 8023 is used to receive the first output result, and process the first output result to obtain a first processing result;

[0158] A loss calculation module 8024, configured to determine a first loss parameter based on the first processing result and a first loss function;

[0159] The updating module 8025 is used to update the model parameters of the unified multilingual multimodal model framework based on the first loss parameter.

[0160] As an optional implementation of the embodiment of the present disclosure, the first loss function includes: a matching loss function and / or a CMLM loss function.

[0161] As an optional implementation of the embodiment of the present disclosure, the loss calculation module 8024 is further used to: determine a second loss parameter based on the first encoding information, the second encoding information and the second loss function;

[0162] The updating module 8025 is further used to update the model parameters of the unified multilingual multimodal model framework based on the second loss parameter.

[0163] As an optional implementation of the embodiment of the present disclosure, the second loss function is a contrastive learning loss function.

[0164] As an optional implementation of the embodiment of the present disclosure, the target pairing data is the image-text pairing data, the first encoding information is the encoding information of the image, and the second encoding information is the encoding information of the target text;

[0165] The target paired data is the parallel paired data, the first encoding information is the encoding information of the first language text, and the second encoding information is the encoding information of the second language text.

[0166] like Fig. 9 As shown, it is a schematic diagram of the hardware structure of an electronic device provided by an embodiment of the present disclosure, and the electronic device includes: a processor 901, a memory 902, and a computer program stored in the memory 902 and executable on the processor 901, and when the computer program is executed by the processor 901, each process of the training method of the multi-language multi-modal pre-training model in the above method embodiment is implemented. And the same technical effect can be achieved, so it will not be repeated here to avoid repetition.

[0167] An embodiment of the present invention provides a computer-readable storage medium, characterized in that a computer program is stored on the computer-readable storage medium. When the computer program is executed by a processor, each process of the training method of the multi-language multi-modal pre-training model in the above method embodiment is implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0168] The computer readable storage medium may be a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0169] An embodiment of the present invention provides a computer program product, which stores a computer program. When the computer program is executed by a processor, the various processes of the training method of the multi-language multi-modal pre-training model in the above method embodiment are implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0170] Those skilled in the art will appreciate that the embodiments of the present disclosure may be provided as methods, systems, or computer program products. Therefore, the present disclosure may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Furthermore, the present disclosure may take the form of a computer program product implemented on one or more computer-usable storage media containing computer-usable program code.

[0171] In the present disclosure, the processor may be a central processing unit (CPU), a graphics processing unit (GPU), other general-purpose processors, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0172] In the present disclosure, memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. Memory is an example of a computer-readable medium.

[0173] In the present disclosure, computer-readable media include permanent and non-permanent, removable and non-removable storage media. Storage media can be implemented by any method or technology to store information, and the information can be computer-readable instructions, data structures, modules of programs or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, read-only compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. According to the definition in this article, computer-readable media does not include temporary computer-readable media (transitory media), such as modulated data signals and carrier waves.

[0174] It should be noted that, in this article, relational terms such as "first" and "second" are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0175] The above are only specific embodiments of the present disclosure, so that those skilled in the art can understand or implement the present disclosure. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein can be implemented in other embodiments without departing from the spirit or scope of the present disclosure. Therefore, the present disclosure will not be limited to these embodiments herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A training method for a multi-language multi-modal pre-training model, characterized in that: include: Acquire a paired sample set, wherein the paired sample set includes: image-text paired data of an image and a target text, and parallel paired data of a first language text and a second language text; The paired sample set is pre-trained in a preset manner based on a unified multilingual multimodal model framework to obtain a multilingual multimodal pre-trained model; the preset manner includes: encoding target paired data to obtain first encoded information and second encoded information, the target paired data includes the image-text paired data or the parallel paired data; inputting the first encoded information and the second encoded information into an N-layer feature fusion model, where N is an integer greater than or equal to 1; inputting a first output result of the N-layer feature fusion model into a linear layer for processing to obtain a first processing result, and determining a first loss parameter based on the first processing result and a first loss function; and updating the model parameters of the unified multilingual multimodal model framework based on the first loss parameter; The target text is in any language type, and the first language text and the second language text are in different language types.

2. The method according to claim 1, characterized in that The paired sample set is pre-trained based on a unified multi-language multi-modal model framework to obtain a multi-language multi-modal pre-trained model, including: The image-text paired data and the parallel paired data are pre-trained alternately based on a unified multilingual multimodal model framework to obtain the multilingual multimodal pre-trained model.

3. The method according to claim 1, characterized in that The training objectives of pre-training the paired sample set based on the unified multilingual multimodal model framework include: Contrastive learning on target paired data; and / or, The target paired data includes masked text, and the masked text in the target paired data is restored; Wherein, the target pairing data includes the image-text pairing data or the parallel pairing data.

4. The method according to any one of claims 1 to 3, characterized in that: The first loss function includes: a matching loss function and / or a conditional mask language model CMLM loss function.

5. The method according to any one of claims 1 to 3, characterized in that: The method further comprises: Determine a second loss parameter based on the first encoding information, the second encoding information and a second loss function; Based on the second loss parameter, the model parameters of the unified multilingual multimodal model framework are updated.

6. The method according to claim 5, characterized in that The second loss function is a contrastive learning loss function.

7. The method according to any one of claims 1 to 3, characterized in that: The target paired data is the image-text paired data, the first encoding information is the encoding information of the image, and the second encoding information is the encoding information of the target text; The target paired data is the parallel paired data, the first encoding information is the encoding information of the first language text, and the second encoding information is the encoding information of the second language text.

8. A training device for a multi-language multi-modal pre-training model, characterized in that: include: An acquisition module, used to acquire a paired sample set, wherein the paired sample set includes: image-text paired data of an image and a target text, and parallel paired data of a first language text and a second language text; A pre-training module, used for pre-training the paired sample set based on a unified multi-language multi-modal model framework to obtain a multi-language multi-modal pre-training model; The pre-training module includes an encoder, an N-layer feature fusion model, a linear layer, a loss calculation module, and an update module; The encoder is used to encode the target pairing data to obtain first encoding information and second encoding information, wherein the target pairing data includes the image-text pairing data or the parallel pairing data; The N-layer feature fusion model is used to receive the first encoded information and the second encoded information as input, and output a first input result, where N is an integer greater than or equal to 1; The linear layer is used to receive the first output result and process the first output result to obtain a first processing result; The loss calculation module is used to determine a first loss parameter based on the first processing result and a first loss function; The updating module is used to update the model parameters of the unified multilingual multimodal model framework based on the first loss parameter; The target text is in any language type, and the first language text and the second language text are in different language types.

9. An electronic device, characterized in that: include: A processor, a memory, and a computer program stored in the memory and executable on the processor, wherein when the computer program is executed by the processor, the training method for the multilingual multimodal pre-training model as described in any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium, characterized in that: include: The computer-readable storage medium stores a computer program, and when the computer program is executed by the processor, the training method of the multilingual multimodal pre-training model according to any one of claims 1 to 7 is implemented.

Images