A method and device for training a multi-modal pre-training model

Abstract

The application provides a kind of multi-modal pre-training model training method and device, constructs the multi-modal pre-training model containing multimodal graph-text information processing network;Weak alignment image-text data set is constructed;Wherein, weak alignment image-text data set contains text data set, image-label data set and image-reference description data set;Multi-modal pre-training model is trained using weak alignment image-text data set.The multimodal graph-text information processing network of the application can directly process multi-modal graph-text information, without external model auxiliary extraction image feature, and has strong inference ability.At the same time, weak alignment image-text data set is used to train multi-modal pre-training model, reduce the dependence on artificial labeling image-text alignment data, avoid the problem of large data overhead that occurs when using alignment large-scale image-text data set to train multi-modal pre-training model.

Description

Technical Field

[0001] This invention relates to the field of unsupervised machine learning technology, and in particular to a training method and apparatus for a multimodal pre-trained model. Background Technology

[0002] With the continuous development of transfer learning technology, a series of pre-trained multimodal models have emerged, such as UNITER, VinVL, CLIP, and DALL·E. These multimodal pre-trained models are typically pre-trained on relatively simple tasks using large-scale datasets. After pre-training, the parameters of the multimodal pre-trained model are fine-tuned according to specific downstream tasks (such as visual question answering tasks, image and text retrieval tasks, etc.) and used in the execution of the corresponding downstream tasks to improve the performance of the downstream tasks.

[0003] Most existing multimodal pre-trained models use aligned, large-scale image-text datasets for pre-training to achieve cross-modal understanding capabilities. However, aligned, large-scale image-text data often requires manual annotation or data cleaning, resulting in excessive data overhead. A few models use unaligned image-text datasets for pre-training, which have relatively poor cross-modal capabilities, and the pre-training process requires external models to help extract image feature vectors, leading to relatively low inference efficiency.

[0004] Therefore, there is an urgent need to provide a method that can fully utilize unaligned image and text datasets to train multimodal pre-trained models with high cross-modal understanding capabilities. Summary of the Invention

[0005] This invention provides a training method and apparatus for a multimodal pre-trained model. It employs a weakly aligned image-text dataset to train the multimodal pre-trained model, reducing reliance on manually labeled image-text aligned data and avoiding the high data overhead associated with using large-scale aligned image-text datasets. The trained multimodal pre-trained model can directly process multimodal image and text information without external model assistance in extracting image features, avoiding the relatively low inference efficiency associated with using unaligned image-text datasets. Furthermore, by jointly training the multimodal pre-trained model with tasks such as masked label prediction, referential description matching, and masked word segmentation prediction, it avoids the relatively poor cross-modal capability that occurs when using unaligned image-text datasets.

[0006] In a first aspect, the present invention provides a training method for a multimodal pre-trained model, the method comprising:

[0007] Construct a multimodal pre-trained model that includes a multimodal image and text information processing network;

[0008] Construct a weakly aligned image-text dataset; wherein the weakly aligned image-text dataset includes a text dataset, an image-label dataset, and an image-reference description dataset;

[0009] The multimodal pre-trained model was trained using the weakly aligned image-text dataset.

[0010] The samples in the image-label dataset consist of images and their corresponding label text sequences; the label text sequence corresponding to an image is formed by concatenating the label words of all entities in the image.

[0011] The samples in the image-reference description dataset consist of images and their corresponding reference descriptions.

[0012] According to the training method of the multimodal pre-trained model provided by the present invention, the process of generating the image-label dataset includes:

[0013] Obtain the image dataset;

[0014] For each image in the image dataset, all entities present in the image are obtained using an object detector;

[0015] Based on all entities present in the image, generate a text sequence of tag words corresponding to the image;

[0016] The image-label dataset is generated using each image in the image dataset and its corresponding label word text sequence.

[0017] According to the training method of the multimodal pre-trained model provided by the present invention, the generation process of the image-reference description dataset includes:

[0018] Remove overlapping entities from all the entities, and construct a first entity set using the remaining entities;

[0019] Construct a second entity set using any entity in the first entity set and any entity with the same tag word as the first entity;

[0020] Based on the tags of any entity and the size and position of each entity in the second entity set, generate a descriptive description of any entity.

[0021] Use the reference description of any of the entities as the reference description corresponding to the image.

[0022] The image-reference description dataset is generated using each image in the image dataset and its corresponding reference description.

[0023] According to the training method of the multimodal pre-trained model provided by the present invention, the step of training the multimodal pre-trained model using the weakly aligned image-text dataset includes:

[0024] Each sample in the image-label dataset undergoes a first preprocessing step to obtain a first dataset;

[0025] A second preprocessing step is performed on each sample in the image-reference description dataset to obtain a second dataset;

[0026] A third preprocessing step is performed on each sample in the text dataset to obtain a third dataset;

[0027] Using a multimodal graph and text information processing network, a multimodal representation vector is generated for each sample in the first dataset, a multimodal representation vector for each sample in the second dataset, and a multimodal representation vector for each sample in the third dataset.

[0028] Using the multimodal representation vectors corresponding to each sample in the first dataset, the multimodal representation vectors corresponding to each sample in the second dataset, and the multimodal representation vectors corresponding to each sample in the third dataset, the multimodal pre-trained model is used to perform the task of predicting masked label words, which refers to the joint training of the description matching task and the task of predicting masked word segmentation.

[0029] The first preprocessing includes at least:

[0030] The image in the sample is cut into N blocks to obtain the corresponding image block sequence;

[0031] By randomly masking a portion of the labeled words in the text sequence of labeled words in the sample, a text sequence of labeled words with masking annotations is obtained;

[0032] The second preprocessing includes at least:

[0033] The image in the sample is cut into N blocks to obtain the corresponding image block sequence;

[0034] Determine the word segmentation sequence corresponding to the referential descriptions in the sample;

[0035] The third preprocessing includes at least:

[0036] Determine the text segmentation sequence corresponding to the text in the sample;

[0037] Randomly masking some of the words in the text segmentation sequence yields a text segmentation sequence with masking annotations.

[0038] According to the training method of the multimodal pre-trained model provided by the present invention, the multimodal image-text information processing network includes: a text embedding layer, a visual encoder, and a multimodal encoder; the step of generating a multimodal representation vector corresponding to each sample in the first dataset using the multimodal image-text information processing network includes:

[0039] The text embedding layer is used to convert the text sequence of masked labeled words for each sample in the first dataset into text word vectors.

[0040] The visual encoder is used to determine the image feature vector corresponding to the image patch sequence of each sample in the first dataset;

[0041] The multimodal encoder is used to fuse the text word vector converted from the text sequence of labeled words with masking annotations of each sample in the first dataset and the image feature vector corresponding to the image patch sequence of each sample in the first dataset to obtain the multimodal representation vector corresponding to each sample in the first dataset.

[0042] The step of generating a multimodal representation vector for each sample in the second dataset using a multimodal graph-text information processing network includes:

[0043] The text embedding layer is used to convert the segmented word sequence of the referential description of each sample in the second dataset into text word vectors;

[0044] The visual encoder is used to determine the image feature vector corresponding to the image patch sequence of each sample in the second dataset;

[0045] The multimodal encoder is used to fuse the text word vector converted from the word segmentation sequence of the reference description of each sample in the second dataset and the image feature vector corresponding to the image block sequence of each sample in the second dataset to obtain the multimodal representation vector corresponding to each sample in the second dataset.

[0046] The process of generating a multimodal representation vector for each sample in the third dataset using a multimodal graph-text information processing network includes:

[0047] The text embedding layer is used to convert the text segmentation sequence with masking annotations of each sample in the third dataset into text word vectors;

[0048] The text word vectors converted from the text segmentation sequence with masking annotations of each sample in the third dataset are used as the multimodal representation vectors corresponding to each sample in the third dataset.

[0049] According to the training method of the multimodal pre-trained model provided by the present invention, the step of using the multimodal representation vector corresponding to each sample in the first dataset, the multimodal representation vector corresponding to each sample in the second dataset, and the multimodal representation vector corresponding to each sample in the third dataset to perform the masked label prediction task, which refers to the joint training of the description matching task and the masked word segmentation prediction task, includes:

[0050] Predict the masked label word for each sample in the first dataset using the multimodal representation vector corresponding to each sample in the first dataset;

[0051] Using the multimodal representation vector corresponding to each sample in the second dataset, predict the location of the referent entity corresponding to each sample in the second dataset in the image corresponding to each sample in the second dataset;

[0052] The multimodal representation vector corresponding to each sample in the third dataset predicts the masked word segmentation of each sample in the third dataset;

[0053] Calculate the prediction loss of the masked label words in the first dataset;

[0054] Calculate the loss for predicting the location of the referential entity in the second dataset;

[0055] Calculate the occluded word segmentation prediction loss for the third dataset;

[0056] The parameters of the multimodal pre-trained model are optimized by using the sum of the occluded tag word prediction loss of the first dataset, the referential entity location prediction loss of the second dataset, and the occluded word segmentation prediction loss of the third dataset as the training loss.

[0057] Repeat the above steps until the multimodal pre-trained model converges.

[0058] According to the training method of the multimodal pre-trained model provided by the present invention, the prediction loss of the masked label words of the first dataset The calculation formula is as follows:

[0059]

[0060] Among them, (I, L) tag ) represents image I and its corresponding tag text sequence L tag The sample composed of For image-label datasets, T m For the tag word text sequence L tag The text word vectors corresponding to the masked label words, T \m For the tag word text sequence Ltag The text word vectors corresponding to the unmasked labels in image I, V is the image feature vector corresponding to the image patch sequence of image I, and logP(T) m |T \m V) is the text sequence of tag words L tag The cross-entropy between the joint probability distribution of candidate words corresponding to the masked tag words and the ground value;

[0061] The loss for predicting the location of entities referred to by the segmented word sequences in the second dataset. The calculation formula is as follows:

[0062]

[0063]

[0064]

[0065] Among them, (I, L) exp () represents the sample consisting of image I and its corresponding descriptive description. For the image-reference description dataset, M is... The number of images contained in For R I and The soft dice loss between them For R I and The binary cross-entropy loss between them, R I Let be the true mask of image I, and represent the probability functions corresponding to the N image patches of image I. Let be the predicted mask for image I, and represent the probability functions corresponding to the N image patches of image I. For R I The probability function corresponding to the i-th image patch. for The probability function corresponding to the i-th image patch is given. The probability function takes the value 0 or 1. When the value is 0, it means that there is no referent entity. When the value is 1, it means that there is a referent entity.

[0066] The occlusion segmentation prediction loss of the third dataset The calculation formula is as follows:

[0067]

[0068] in, For a text dataset, T m* T is the text word vector corresponding to the masked segment in the text segmentation sequence corresponding to text L. \m*For text L, the unmasked word vectors in the text segmentation sequence are logP(T) m* |T \m* ) represents the cross-entropy between the joint probability distribution of candidate words corresponding to masked words in the text segmentation sequence corresponding to text L and the ground truth value.

[0069] Secondly, the present invention provides a training apparatus for a multimodal pre-trained model, the apparatus comprising:

[0070] The first building module is used to build a multimodal pre-trained model that includes a multimodal image and text information processing network;

[0071] The second building module is used to construct a weakly aligned image-text dataset; wherein, the weakly aligned image-text dataset includes a text dataset, an image-label dataset, and an image-reference description dataset;

[0072] The training module is used to train the multimodal pre-trained model using the weakly aligned image-text dataset;

[0073] The samples in the image-label dataset consist of images and their corresponding label text sequences; the label text sequence corresponding to an image is formed by concatenating the label words of all entities in the image.

[0074] The samples in the image-reference description dataset consist of images and their corresponding reference descriptions.

[0075] Thirdly, the present invention provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the training method of the multimodal pre-trained model as described in the first aspect.

[0076] Fourthly, the present invention provides a non-transitory computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the training method of the multimodal pre-trained model as described in the first aspect.

[0077] This invention provides a training method and apparatus for a multimodal pre-trained model, comprising: constructing a multimodal pre-trained model including a multimodal image-text information processing network; the multimodal image-text information processing network has the ability to directly process multimodal image-text information, without the need for external model assistance to extract image features, and has faster inference speed and stronger practicality; constructing a weakly aligned image-text dataset; wherein the weakly aligned image-text dataset includes a text dataset, an image-label dataset, and an image-reference description dataset; training the multimodal pre-trained model using the weakly aligned image-text dataset; wherein the samples in the image-label dataset consist of images and their corresponding label word text sequences; the label word text sequences corresponding to images are concatenated from the label words of all entities in the image; the samples in the image-reference description dataset consist of images and their corresponding reference descriptions. The weakly aligned image-text data used during training can reduce the reliance on manually labeled cross-modal alignment data, significantly reducing the data overhead for training multimodal pre-trained models. At the same time, weakly aligned image-text data can help multimodal pre-trained models learn cross-modal relationships at different granularities, thereby improving the model's cross-modal understanding ability. Attached Figure Description

[0078] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0079] Figure 1 This is a flowchart illustrating a training method for a multimodal pre-trained model provided by the present invention.

[0080] Figure 2 This is a schematic diagram of the structure of a training device for a multimodal pre-trained model provided by the present invention;

[0081] Figure 3 This is a schematic diagram of the structure of an electronic device that implements the training method for multimodal pre-trained models provided by the present invention. Detailed Implementation

[0082] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0083] The following is combined Figures 1-3 This invention describes a training method and apparatus for a multimodal pre-trained model.

[0084] Firstly, the present invention provides a training method for a multimodal pre-trained model, such as... Figure 1 As shown, the method includes:

[0085] S11. Construct a multimodal pre-trained model that includes a multimodal image and text information processing network;

[0086] This invention aims to train a multimodal pre-trained model without using aligned image-text datasets, reducing reliance on manually labeled image-text alignment data and avoiding the high data overhead associated with using large-scale aligned image-text datasets. It also aims to enable the multimodal pre-trained model to inherently process multimodal image-text information, avoiding the relatively low inference efficiency of existing methods that rely on external models to assist in processing image-text information. Therefore, a multimodal pre-trained model incorporating a multimodal image-text information processing network is first constructed. This network includes a text embedding layer, a visual encoder, and a multimodal encoder, all connected to each other. The text embedding layer converts segmented text sequences into corresponding word vectors. The visual encoder extracts image feature vectors. The multimodal encoder fuses the word vectors and image feature vectors to generate multimodal representation vectors for multimodal pre-training tasks. The visual encoder can use a pre-trained visual Transformer such as ViT or Swin-Transformer and can be initialized with the corresponding model parameters; the multimodal encoder adopts the BERT architecture and uses the BERT-base model for parameter initialization; the text embedding layer is initialized with the corresponding parameters of BERT-base.

[0087] S12. Construct a weakly aligned image-text dataset; wherein, the weakly aligned image-text dataset includes a text dataset, an image-label dataset, and an image-reference description dataset;

[0088] To help multimodal pre-trained models learn cross-modal relationships at different granularities and improve their cross-modal understanding capabilities, this invention uses a weakly aligned image-text dataset for training the multimodal pre-trained model.

[0089] S13. Train the multimodal pre-trained model using the weakly aligned image-text dataset;

[0090] The samples in the image-label dataset consist of images and their corresponding label text sequences; the label text sequence corresponding to an image is formed by concatenating the label words of all entities in the image.

[0091] The samples in the image-reference description dataset consist of images and their corresponding reference descriptions.

[0092] This invention provides a training method for a multimodal pre-trained model. The method constructs a multimodal image-text information processing network within the pre-trained model, enabling it to directly process multimodal image-text information. This eliminates the need for external models to extract image features, resulting in faster inference speeds and greater practicality. The weakly aligned image-text data used in training the pre-trained model reduces reliance on manually labeled cross-modal alignment data, significantly reducing the data overhead. Simultaneously, the weakly aligned image-text data helps the pre-trained model learn cross-modal relationships at different granularities, thereby enhancing the model's cross-modal understanding capabilities.

[0093] The multimodal pre-trained model of this invention is applicable to various real-world application scenarios, such as visual question answering, image and text retrieval, and other multimodal task scenarios.

[0094] When applied to visual question answering, a visual question answering dataset is constructed, in which each sample consists of an image, a question, and the actual answer to the question. Using the visual question answering dataset, a visual question answering model is trained on the basis of a multimodal pre-trained model. By inputting the target image and the corresponding question into the visual question answering model, the answer to the question can be obtained.

[0095] When applied to image-text retrieval, an image-text retrieval dataset is constructed. This dataset consists of images, actual descriptive text for the images (positive examples), and multiple descriptive texts that are incorrect or irrelevant to the images (negative examples). Using this dataset, an image-text retrieval model is trained based on a multimodal pre-trained model. By inputting the target image and a set of text into the image-text retrieval model, the corresponding text for the target image can be obtained. Of course, the principle is the same for image-text retrieval of a single text and multiple images, which will not be elaborated upon here.

[0096] Based on the above embodiments, as an optional embodiment, the process of generating the image-label dataset includes:

[0097] Obtain an image dataset; for each image in the image dataset, obtain all entities present in the image using an object detector; based on all entities present in the image, generate a text sequence of label words corresponding to the image; use each image in the image dataset and its corresponding text sequence of label words to generate the image-label dataset.

[0098] Specifically, this invention requires generating a weakly aligned image-text dataset based on image and text datasets, for example, for...

[0099] For image I in the image dataset, an object detector is used to obtain all entities present in the image, where the label of the j-th entity corresponds to the tag word o. j The corresponding position b in the image j Let K be the total number of all entities. Here, the position b of the *l*th entity in the image is... j It can be represented in several ways, such as as a rectangular bounding box b. j The bounding box is represented in the form [x1, y1, x2, y2], where x1 and y1 are the coordinates of the top-left corner, and x2 and y2 are the coordinates of the bottom-right corner. Concatenating the label words of all entities in image I yields the label text sequence L corresponding to image I. tag = [o1, o2, ..., o K ], and obtain image label data pairs (I, L) tag The image label data pairs of all images in the image dataset constitute a set. gather This is an image-label dataset.

[0100] It should be noted that entity tags are type descriptions of the entity, such as person, sheep, shadow, etc. The tag text sequence is an ordered sequence of tag words, for example: L tag =[people, hair, shirt,..., shadow, boot].

[0101] This invention intelligently processes image datasets to obtain weakly aligned image-label datasets without manual annotation, providing a data foundation for training multimodal pre-trained models.

[0102] Based on the above embodiments, as an optional embodiment, the process of generating the image-reference description dataset includes:

[0103] Remove overlapping entities from all the entities, and construct a first entity set using the remaining entities;

[0104] Construct a second entity set using any entity in the first entity set and any entity with the same tag word as the first entity;

[0105] Based on the tags of any entity and the size and position of each entity in the second entity set, generate a descriptive description of any entity.

[0106] Use the reference description of any of the entities as the reference description corresponding to the image.

[0107] The image-reference description dataset is generated using each image in the image dataset and its corresponding reference description.

[0108] Specifically, for image I in the image dataset, overlapping entities are first removed to avoid different parts of the same entity being mistaken for different entities;

[0109] Then, randomly select an entity, and denote its label and bounding box as (o k b k ).

[0110] Next, select other images in image I that contain the tagged word "o". k The entity is denoted as

[0111] Then, based on the tag word "o" k And based on the relative size and position of these entities, the corresponding text descriptions are heuristically generated, thus obtaining the text descriptions in image I (o k b k The corresponding denotation L exp The resulting image-referencing descriptive data pair is (I, L). exp );

[0112] Finally, all the image reference description data pairs constitute a set. Right now For image-referential description datasets.

[0113] For example, for the example image, overlapping entities are removed, and an entity "sheep" is randomly selected and its position L1 is recorded. Based on the randomly selected entity "sheep", other entities with the same label "sheep" are found in the example image (positions L2, L3, L4, etc.). The "sheep" at position L1 is similar in size to the "sheep" at other positions and is located to the left of the "sheep" at other positions. Therefore, the heuristic generation of the referential description is L. exp =“smaller white sheep on the right”. The example image and its descriptive description are L. exp =“smaller white sheep on the right” yields the image reference description data pair.

[0114] Similar to the image-label weak alignment dataset, this invention performs intelligent processing on the image dataset to obtain the image-reference description weak alignment dataset in a way that does not require manual annotation, providing a data foundation for training multimodal pre-trained models.

[0115] Based on the above embodiments, as an optional embodiment, training the multimodal pre-trained model using the weakly aligned image-text dataset includes:

[0116] Each sample in the image-label dataset undergoes a first preprocessing step to obtain a first dataset;

[0117] A second preprocessing step is performed on each sample in the image-reference description dataset to obtain a second dataset;

[0118] A third preprocessing step is performed on each sample in the text dataset to obtain a third dataset;

[0119] It is understood that this invention performs joint training of the multimodal pre-trained model on multiple tasks, including the masked label prediction task, the reference description matching task, and the masked word segmentation prediction task. Therefore, it is necessary to preprocess the image-label dataset to obtain a sample dataset suitable for the masked label prediction task, preprocess the image-reference description dataset to obtain a sample dataset suitable for the reference description matching task, and preprocess the text dataset to obtain a sample dataset suitable for the masked word segmentation prediction task.

[0120] Using a multimodal graph and text information processing network, a multimodal representation vector is generated for each sample in the first dataset, a multimodal representation vector for each sample in the second dataset, and a multimodal representation vector for each sample in the third dataset.

[0121] That is, the present invention obtains the cross-modal expression of each input sample through a multimodal graph and text information processing network.

[0122] Using the multimodal representation vectors corresponding to each sample in the first dataset, the multimodal representation vectors corresponding to each sample in the second dataset, and the multimodal representation vectors corresponding to each sample in the third dataset, the multimodal pre-trained model is used to perform the task of predicting masked label words, which refers to the joint training of the description matching task and the task of predicting masked word segmentation.

[0123] The first preprocessing includes at least:

[0124] The image in the sample is cut into N blocks to obtain the corresponding image block sequence;

[0125] By randomly masking a portion of the labeled words in the text sequence of labeled words in the sample, a text sequence of labeled words with masking annotations is obtained;

[0126] The second preprocessing includes at least:

[0127] The image in the sample is cut into N blocks to obtain the corresponding image block sequence;

[0128] Determine the word segmentation sequence corresponding to the referential descriptions in the sample;

[0129] The third preprocessing includes at least:

[0130] Determine the text segmentation sequence corresponding to the text in the sample;

[0131] Randomly masking some of the words in the text segmentation sequence yields a text segmentation sequence with masking annotations.

[0132] The image in the sample is divided into N blocks. For example, for image I, it is divided into N = 14 * 14 = 196 blocks.

[0133] Randomly mask tags, for example, randomly mask L tag =Partial tags in [people, hair, shirt, ..., shadow, boot] yield [people, [MASK], shirt, ..., [MASK], boot]. Random masking segmentation follows the same processing method, so it will not be elaborated further.

[0134] This invention directly processes multimodal graph and text information through a multimodal graph and text information processing network to obtain multimodal representation vectors, which can be used to train a multimodal pre-trained model for various tasks. Simultaneously, by training the multimodal pre-trained model on multiple tasks, it further helps the model learn cross-modal connections at different granularities.

[0135] Based on the above embodiments, as an optional embodiment, the multimodal image and text information processing network includes: a text embedding layer, a visual encoder, and a multimodal encoder; the step of using the multimodal image and text information processing network to generate a multimodal representation vector corresponding to each sample in the first dataset includes:

[0136] The text embedding layer is used to convert the text sequence of masked labeled words for each sample in the first dataset into text word vectors.

[0137] The visual encoder is used to determine the image feature vector corresponding to the image patch sequence of each sample in the first dataset;

[0138] The multimodal encoder is used to fuse the text word vector converted from the text sequence of labeled words with masking annotations of each sample in the first dataset and the image feature vector corresponding to the image patch sequence of each sample in the first dataset to obtain the multimodal representation vector corresponding to each sample in the first dataset.

[0139] The step of generating a multimodal representation vector for each sample in the second dataset using a multimodal graph-text information processing network includes:

[0140] The text embedding layer is used to convert the segmented word sequence of the referential description of each sample in the second dataset into text word vectors;

[0141] The visual encoder is used to determine the image feature vector corresponding to the image patch sequence of each sample in the second dataset;

[0142] The multimodal encoder is used to fuse the text word vector converted from the word segmentation sequence of the reference description of each sample in the second dataset and the image feature vector corresponding to the image block sequence of each sample in the second dataset to obtain the multimodal representation vector corresponding to each sample in the second dataset.

[0143] The process of generating a multimodal representation vector for each sample in the third dataset using a multimodal graph-text information processing network includes:

[0144] The text embedding layer is used to convert the text segmentation sequence with masking annotations of each sample in the third dataset into text word vectors;

[0145] The text word vectors converted from the text segmentation sequence with masking annotations of each sample in the third dataset are used as the multimodal representation vectors corresponding to each sample in the third dataset.

[0146] For example, for an image I divided into N image patches, the image feature vector V = [v1, v2, ..., v3] is extracted by a visual encoder. N ].

[0147] For the word segmentation sequence corresponding to text L, it is transformed into text word vectors T = [t1, t2, ..., t3] through a text embedding layer. SD ], SD is the length of the word segmentation sequence.

[0148] This invention utilizes a multimodal image-text information processing network composed of a text embedding layer, a visual encoder, and a multimodal encoder to extract text features and image features, and to fuse and represent text features and image features, thereby assisting in the training of multimodal pre-trained models for multiple tasks.

[0149] Based on the above embodiments, as an optional embodiment, the step of using the multimodal representation vectors corresponding to each sample in the first dataset, the multimodal representation vectors corresponding to each sample in the second dataset, and the multimodal representation vectors corresponding to each sample in the third dataset to perform the masked label prediction task on the multimodal pre-trained model refers to the joint training of the description matching task and the masked word segmentation prediction task, including:

[0150] Predict the masked label word for each sample in the first dataset using the multimodal representation vector corresponding to each sample in the first dataset;

[0151] Using the multimodal representation vector corresponding to each sample in the second dataset, predict the location of the referent entity corresponding to each sample in the second dataset in the image corresponding to each sample in the second dataset;

[0152] The multimodal representation vector corresponding to each sample in the third dataset predicts the masked word segmentation of each sample in the third dataset;

[0153] Calculate the prediction loss of the masked label words in the first dataset;

[0154] Calculate the loss for predicting the location of the referential entity in the second dataset;

[0155] Calculate the occluded word segmentation prediction loss for the third dataset;

[0156] The parameters of the multimodal pre-trained model are optimized by using the sum of the occluded tag word prediction loss of the first dataset, the referential entity location prediction loss of the second dataset, and the occluded word segmentation prediction loss of the third dataset as the training loss.

[0157] Repeat the above steps until the multimodal pre-trained model converges.

[0158] Specifically, using text datasets and weakly aligned multimodal datasets and The multimodal pre-trained model is trained using a designed pre-training task, as fully described below:

[0159] Masked tag word prediction task:

[0160] For image label datasets Perform a task to predict masked tags.

[0161] for Each data pair (I, L) tag Randomly cover L tagThe system identifies some of the masked words and uses other unmasked words and image information to predict masked words.

[0162] For example: randomly covering L tag The partial tags in [people, hair, shirt, ..., shadow, boot] are used to obtain [people, [MASK], shirt, ..., [MASK], boot]. [people, [MASK], shirt, ..., [MASK], boot] is then converted into text word vectors through a text embedding layer. The unmasked parts (i.e., people, shirt, ..., boot, etc.) correspond to text word vectors T. \m The word vector corresponding to the masked portion (i.e., all [MASK] positions) is T. m Image I is processed by an image encoder to extract its image feature vector. A multimodal encoder is then used to fuse the text word vectors corresponding to [people, [MASK], shirt, ..., [MASK], boot] with the image feature vector of image I to obtain a cross-modal representation of the occluded portion. This cross-modal representation is then used to predict the occlusion label words (shirt and shadow). This part of the loss is... express.

[0163] Refers to the description of the matching task:

[0164] For image-reference description datasets Perform a referential description matching task. For Each data pair (I, L) exp According to the referential description L exp To predict the specific location of the referent (the entity corresponding to the multiple descriptions) on image I.

[0165] Specifically, L exp (shaped like an L) exp =“smaller white sheep on the right”) obtains text word vectors through a text embedding layer. Image I extracts image feature vectors through an image encoder. Based on the bounding box of the referent (“sheep”), a corresponding image patch mask R∈{0,1} is generated. N A value of 0 indicates that the corresponding image patch does not contain the entity, while a value of 1 indicates that the corresponding image patch contains the entity. Based on the output of the multimodal encoder, the multimodal pre-trained model predicts a probability value between 0 and 1 for each image patch, representing the probability that the corresponding image patch contains the entity. In other words, the multimodal pre-trained model predicts a prediction mask accordingly. The descriptive matching task requires minimizing the model's prediction error; this part of the loss is determined by... express.

[0166] Masked word segmentation prediction task for text datasets Given text L, we perform a word segmentation masking prediction task. We randomly mask a portion of the word segments in the word segmentation sequence of text L, and then use the unmasked segments to predict the masked segments. This part does not use an image as input, and therefore does not require an image encoder for training. The loss for this part is calculated by... express.

[0167] Ultimately, the total training loss was

[0168]

[0169] This invention trains a multimodal pre-trained model on various tasks using a weakly aligned multimodal dataset, thereby improving the cross-modal understanding capability of the multimodal pre-trained model.

[0170] Based on the above embodiments, as an optional embodiment, the prediction loss of the masked label words in the first dataset... The calculation formula is as follows:

[0171]

[0172] Among them, (j, L) tag ) represents image I and its corresponding tag text sequence L tag The sample composed of For image-label datasets, T m For the tag word text sequence L tag The text word vectors corresponding to the masked label words, T \m For the tag word text sequence L tag The unmasked label word is the text word vector, V is the image feature vector corresponding to the image patch sequence of image I, and logP(T) m |T \m V) is the text sequence of tag words L tag The cross-entropy between the joint probability distribution of candidate words corresponding to the masked tag words and the ground value;

[0173] It is important to understand that for masked tags, the prediction result is the probability of each word in the vocabulary represented by the candidate word probability distribution. The dot product of the candidate word probability distributions of multiple masked tags in the tag word text sequence is the joint probability distribution of the candidate words. The true value is determined by the true result corresponding to the masked tags in the tag word text sequence.

[0174] The loss for predicting the location of entities referred to by the segmented word sequences in the second dataset. The calculation formula is as follows:

[0175]

[0176]

[0177]

[0178] Among them, (I, L) exp () represents the sample consisting of image I and its corresponding descriptive description. Let M be an image-reference description dataset. The number of images contained in For R I and The soft dice loss between them For R I and The binary cross-entropy loss between them, R I Let be the true mask of image I, and represent the probability functions corresponding to the N image patches of image I. Let be the predicted mask for image I, and represent the probability functions corresponding to the N image patches of image I. For R I The probability function corresponding to the i-th image patch. for The probability function corresponding to the i-th image patch is given. The probability function takes the value 0 or 1. When the value is 0, it means that there is no referent entity. When the value is 1, it means that there is a referent entity.

[0179] The occlusion segmentation prediction loss of the third dataset The calculation formula is as follows:

[0180]

[0181] in, For a text dataset, T m* T is the text word vector corresponding to the masked segment in the text segmentation sequence corresponding to text L. \m* For text L, the unmasked word vectors in the text segmentation sequence are logP(T) m* |T \m* ) represents the cross-entropy between the joint probability distribution of candidate words corresponding to masked words in the text segmentation sequence corresponding to text L and the ground truth value.

[0182] logP(T m* |T \m* ) and logP(T m |T \m Similar to V), it will not be elaborated here.

[0183] This invention provides a loss calculation formula for the masked tag word prediction task, which refers to the description matching task and the masked word segmentation prediction task, and facilitates parameter optimization of multimodal pre-trained models.

[0184] Secondly, the training apparatus for the multimodal pre-trained model provided by the present invention will be described. The training apparatus for the multimodal pre-trained model described below and the training method for the multimodal pre-trained model described above can be referred to in correspondence with each other. Figure 2 A schematic diagram of the training device for a multimodal pre-trained model is shown in the example. Figure 2 As shown, the device includes:

[0185] The first building module 21 is used to build a multimodal pre-trained model containing a multimodal image and text information processing network;

[0186] The second construction module 22 is used to construct a weakly aligned image-text dataset; wherein, the weakly aligned image-text dataset includes a text dataset, an image-label dataset, and an image-reference description dataset;

[0187] Training module 23 is used to train the multimodal pre-trained model using the weakly aligned image-text dataset;

[0188] The samples in the image-label dataset consist of images and their corresponding label text sequences; the label text sequence corresponding to an image is formed by concatenating the label words of all entities in the image.

[0189] The samples in the image-reference description dataset consist of images and their corresponding reference descriptions.

[0190] This invention provides a training device for a multimodal pre-trained model. The pre-trained model incorporates a multimodal image-text information processing network capable of directly processing multimodal image-text information, eliminating the need for external models to extract image features, resulting in faster inference speeds and greater practicality. The weakly aligned image-text data used in training the pre-trained model reduces reliance on manually labeled cross-modal alignment data, significantly reducing data overhead. Simultaneously, the weakly aligned image-text data helps the pre-trained model learn cross-modal relationships at different granularities, thereby enhancing the model's cross-modal understanding capabilities.

[0191] Based on the above embodiments, as an optional embodiment, the apparatus further includes an image-label dataset generation module, which includes:

[0192] Image dataset acquisition unit, used to acquire image datasets;

[0193] An entity detection unit is used to obtain all entities present in each image in the image dataset using an object detector;

[0194] The tag word text sequence generation unit is used to generate a tag word text sequence corresponding to the image based on all entities present in the image;

[0195] The image-label dataset generation unit is used to generate the image-label dataset using each image in the image dataset and its corresponding label word text sequence.

[0196] Based on the above embodiments, as an optional embodiment, the apparatus further includes: an image-reference description dataset generation module, the image-reference description dataset generation module comprising:

[0197] The first entity set construction unit is used to remove overlapping entities from all the entities and construct the first entity set using the remaining entities;

[0198] The second entity set construction unit is used to construct a second entity set using any entity in the first entity set and entities that have the same tag words as the first entity set.

[0199] The denotation generation unit is used to generate a denotation of any entity based on the tag words of any entity and the size and position of each entity in the second entity set.

[0200] A definition unit is used to use the referential description of any entity as the referential description corresponding to the image;

[0201] The image-reference description dataset generation unit is used to generate the image-reference description dataset using each image in the image dataset and its corresponding reference description.

[0202] Based on the above embodiments, as an optional embodiment, the training module includes:

[0203] The first preprocessing unit is used to perform a first preprocessing on each sample in the image-label dataset to obtain a first dataset;

[0204] The second preprocessing unit is used to perform a second preprocessing on each sample in the image-reference description dataset to obtain a second dataset;

[0205] The third preprocessing unit is used to perform third preprocessing on each sample in the text dataset to obtain a third dataset;

[0206] The multimodal representation vector generation unit is used to generate multimodal representation vectors corresponding to each sample in the first dataset, each sample in the second dataset, and each sample in the third dataset using a multimodal graph and text information processing network.

[0207] The multi-task training unit is used to perform the masked label word prediction task on the multimodal pre-trained model by using the multimodal representation vectors corresponding to each sample in the first dataset, the multimodal representation vectors corresponding to each sample in the second dataset, and the multimodal representation vectors corresponding to each sample in the third dataset, which refers to the joint training of the description matching task and the masked word segmentation prediction task.

[0208] The first preprocessing includes at least:

[0209] The image in the sample is cut into N blocks to obtain the corresponding image block sequence;

[0210] By randomly masking a portion of the labeled words in the text sequence of labeled words in the sample, a text sequence of labeled words with masking annotations is obtained;

[0211] The second preprocessing includes at least:

[0212] The image in the sample is cut into N blocks to obtain the corresponding image block sequence;

[0213] Determine the word segmentation sequence corresponding to the referential descriptions in the sample;

[0214] The third preprocessing includes at least:

[0215] Determine the text segmentation sequence corresponding to the text in the sample;

[0216] Randomly masking some of the words in the text segmentation sequence yields a text segmentation sequence with masking annotations.

[0217] Based on the above embodiments, as an optional embodiment, the multimodal representation vector generation unit includes: a first multimodal representation vector generation submodule, a first multimodal representation vector generation submodule, and a first multimodal representation vector generation submodule;

[0218] The first submodule for generating the multimodal representation vector includes:

[0219] The first text word vector generation subunit is used to convert the text sequence of masked labeled words of each sample in the first dataset into text word vectors using the text embedding layer.

[0220] The first image feature vector subunit is used to determine the image feature vector corresponding to the image block sequence of each sample in the first dataset using the visual encoder.

[0221] The first fusion subunit is used to fuse the text word vector converted from the masked label text sequence of each sample in the first dataset and the image feature vector corresponding to the image patch sequence of each sample in the first dataset using the multimodal encoder to obtain the multimodal representation vector corresponding to each sample in the first dataset.

[0222] The second submodule for generating the multimodal representation vector includes:

[0223] The step of generating a multimodal representation vector for each sample in the second dataset using a multimodal graph-text information processing network includes:

[0224] The second text word vector generation subunit is used to convert the word segmentation sequence of the reference description of each sample in the second dataset into text word vectors using the text embedding layer.

[0225] The second image feature vector subunit is used to determine the image feature vector corresponding to the image block sequence of each sample in the second dataset using the visual encoder.

[0226] The second fusion subunit is used to fuse the text word vector converted from the word segmentation sequence of the reference description of each sample in the second dataset and the image feature vector corresponding to the image block sequence of each sample in the second dataset using the multimodal encoder to obtain the multimodal representation vector corresponding to each sample in the second dataset.

[0227] The third submodule for generating the multimodal representation vector includes:

[0228] The third text word vector generation subunit is used to convert the text segmentation sequence with masking annotations of each sample in the third dataset into text word vectors using the text embedding layer.

[0229] Define a sub-unit for using the text word vector converted from the text segmentation sequence with masking annotations of each sample in the third dataset as the multimodal representation vector corresponding to each sample in the third dataset.

[0230] Based on the above embodiments, as an optional embodiment, the multi-task training unit includes:

[0231] The first prediction submodule is used to predict the masked label word of each sample in the first dataset using the multimodal representation vector corresponding to each sample in the first dataset.

[0232] The second prediction submodule is used to predict the position of the referent entity corresponding to each sample in the second dataset in the image corresponding to each sample in the second dataset using the multimodal representation vector corresponding to each sample in the second dataset;

[0233] The third prediction submodule is used to predict the masked word segmentation of each sample in the third dataset using the multimodal representation vector corresponding to each sample in the third dataset.

[0234] The first calculation submodule is used to calculate the prediction loss of the masked label words in the first dataset;

[0235] The second calculation submodule is used to calculate the referential entity location prediction loss of the second dataset;

[0236] The third calculation submodule is used to calculate the occluded word segmentation prediction loss of the third dataset;

[0237] The optimization submodule is used to optimize the parameters of the multimodal pre-trained model by using the sum of the occluded label prediction loss of the first dataset, the referential entity location prediction loss of the second dataset, and the occluded word segmentation prediction loss of the third dataset as the training loss.

[0238] The repeat submodule is used to repeat the above operations until the multimodal pre-trained model converges.

[0239] Based on the above embodiments, as an optional embodiment, the prediction loss of the masked label words in the first dataset... The calculation formula is as follows:

[0240]

[0241] Among them, (I, L) tag ) represents image I and its corresponding tag text sequence L tag The sample composed of For image-label datasets, T m For the tag word text sequence L tag The text word vectors corresponding to the masked label words, T \m For the tag word text sequence L tag The text word vectors corresponding to the unmasked labels in image I, V is the image feature vector corresponding to the image patch sequence of image I, and logP(T) m |T \m V) is the text sequence of tag words L tag The cross-entropy between the joint probability distribution of candidate words corresponding to the masked tag words and the ground value;

[0242] The loss for predicting the location of entities referred to by the segmented word sequences in the second dataset. The calculation formula is as follows:

[0243]

[0244]

[0245]

[0246] Among them, (I, L) exp () represents the sample consisting of image I and its corresponding descriptive description. Let M be an image-reference description dataset. The number of images contained in For R I and The soft dice loss between them For R I and The binary cross-entropy loss between them, R I Let be the true mask of image I, and represent the probability functions corresponding to the N image patches of image I. Let be the predicted mask for image I, and represent the probability functions corresponding to the N image patches of image I. For R I The probability function corresponding to the i-th image patch. for The probability function corresponding to the i-th image patch is given. The probability function takes the value 0 or 1. When the value is 0, it means that there is no referent entity. When the value is 1, it means that there is a referent entity.

[0247] The occlusion segmentation prediction loss of the third dataset The calculation formula is as follows:

[0248]

[0249] in, For a text dataset, T m* T is the text word vector corresponding to the masked segment in the text segmentation sequence corresponding to text L. \m* For text L, the unmasked word vectors in the text segmentation sequence are logP(T) m* |T \m* ) represents the cross-entropy between the joint probability distribution of candidate words corresponding to masked words in the text segmentation sequence corresponding to text L and the ground truth value.

[0250] Figure 3 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 3As shown, the electronic device may include: a processor 310, a communication interface 320, a memory 330, and a communication bus 340, wherein the processor 310, the communication interface 320, and the memory 330 communicate with each other through the communication bus 340. The processor 310 can call logical instructions in the memory 330 to execute a training method for a multimodal pre-trained model. This method includes: constructing a multimodal pre-trained model containing a multimodal image-text information processing network; constructing a weakly aligned image-text dataset; wherein the weakly aligned image-text dataset includes a text dataset, an image-label dataset, and an image-reference description dataset; training the multimodal pre-trained model using the weakly aligned image-text dataset; wherein the samples in the image-label dataset consist of images and their corresponding label word text sequences; the label word text sequences corresponding to images are concatenated from the label words of all entities in the image; the samples in the image-reference description dataset consist of images and their corresponding reference descriptions. Furthermore, the logical instructions in the aforementioned memory 330 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0251] On the other hand, the present invention also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the training method of the multimodal pre-trained model provided by the above methods. The method includes: constructing a multimodal pre-trained model containing a multimodal image-text information processing network; constructing a weakly aligned image-text dataset; wherein the weakly aligned image-text dataset includes a text dataset, an image-label dataset, and an image-reference description dataset; training the multimodal pre-trained model using the weakly aligned image-text dataset; wherein the samples in the image-label dataset consist of images and their corresponding label word text sequences; the label word text sequences corresponding to images are formed by concatenating the label words of all entities in the image; the samples in the image-reference description dataset consist of images and their corresponding reference descriptions.

[0252] In another aspect, the present invention also provides a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements a training method for the multimodal pre-trained model provided by the above methods. This method includes: constructing a multimodal pre-trained model comprising a multimodal image-text information processing network; constructing a weakly aligned image-text dataset; wherein the weakly aligned image-text dataset comprises a text dataset, an image-label dataset, and an image-reference description dataset; training the multimodal pre-trained model using the weakly aligned image-text dataset; wherein the samples in the image-label dataset consist of images and their corresponding label word text sequences; the label word text sequences corresponding to images are formed by concatenating the label words of all entities in the image; and the samples in the image-reference description dataset consist of images and their corresponding reference descriptions.

[0253] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0254] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0255] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A training method for a multimodal pre-trained model, characterized in that, The method includes: Construct a multimodal pre-trained model that includes a multimodal image and text information processing network; Construct a weakly aligned image-text dataset; wherein the weakly aligned image-text dataset includes a text dataset, an image-label dataset, and an image-reference description dataset; The multimodal pre-trained model was trained using the weakly aligned image-text dataset. The samples in the image-label dataset consist of images and their corresponding label text sequences; the label text sequence corresponding to an image is formed by concatenating the label words of all entities in the image. The samples in the image-reference description dataset consist of images and their corresponding reference descriptions; Training the multimodal pre-trained model using the weakly aligned image-text dataset includes: Each sample in the image-label dataset undergoes a first preprocessing step to obtain a first dataset; A second preprocessing step is performed on each sample in the image-reference description dataset to obtain a second dataset; A third preprocessing step is performed on each sample in the text dataset to obtain a third dataset; Using a multimodal graph and text information processing network, a multimodal representation vector is generated for each sample in the first dataset, a multimodal representation vector for each sample in the second dataset, and a multimodal representation vector for each sample in the third dataset. Using the multimodal representation vectors corresponding to each sample in the first dataset, the multimodal representation vectors corresponding to each sample in the second dataset, and the multimodal representation vectors corresponding to each sample in the third dataset, the multimodal pre-trained model is used to perform the task of predicting masked label words, which refers to the joint training of the description matching task and the task of predicting masked word segmentation. The first preprocessing includes at least: The image in the sample is cut into N blocks to obtain the corresponding image block sequence; By randomly masking a portion of the labeled words in the text sequence of labeled words in the sample, a text sequence of labeled words with masking annotations is obtained; The second preprocessing includes at least: The image in the sample is cut into N blocks to obtain the corresponding image block sequence; Determine the word segmentation sequence corresponding to the referential descriptions in the sample; The third preprocessing includes at least: Determine the text segmentation sequence corresponding to the text in the sample; Randomly masking some of the words in the text segmentation sequence yields a text segmentation sequence with masking annotations.

2. The training method for the multimodal pre-trained model according to claim 1, characterized in that, The process of generating the image-label dataset includes: Obtain the image dataset; For each image in the image dataset, all entities present in the image are obtained using an object detector; Based on all entities present in the image, generate a text sequence of tag words corresponding to the image; The image-label dataset is generated using each image in the image dataset and its corresponding label word text sequence.

3. The training method for the multimodal pre-trained model according to claim 2, characterized in that, The generation process of the image-referential description dataset includes: Remove overlapping entities from all the entities, and construct a first entity set using the remaining entities; Construct a second entity set using any entity in the first entity set and any entity with the same tag word as the first entity; Based on the tags of any entity and the size and position of each entity in the second entity set, generate a descriptive description of any entity. Use the reference description of any of the entities as the reference description corresponding to the image. The image-reference description dataset is generated using each image in the image dataset and its corresponding reference description.

4. The training method for the multimodal pre-trained model according to claim 1, characterized in that, The multimodal image-text information processing network includes: a text embedding layer, a visual encoder, and a multimodal encoder; the step of generating a multimodal representation vector corresponding to each sample in the first dataset using the multimodal image-text information processing network includes: The text embedding layer is used to convert the text sequence of masked labeled words for each sample in the first dataset into text word vectors. The visual encoder is used to determine the image feature vector corresponding to the image patch sequence of each sample in the first dataset; The multimodal encoder is used to fuse the text word vector converted from the text sequence of labeled words with masking annotations of each sample in the first dataset and the image feature vector corresponding to the image patch sequence of each sample in the first dataset to obtain the multimodal representation vector corresponding to each sample in the first dataset. The step of generating a multimodal representation vector for each sample in the second dataset using a multimodal graph-text information processing network includes: The text embedding layer is used to convert the segmented word sequence of the referential description of each sample in the second dataset into text word vectors; The visual encoder is used to determine the image feature vector corresponding to the image patch sequence of each sample in the second dataset; The multimodal encoder is used to fuse the text word vector converted from the word segmentation sequence of the reference description of each sample in the second dataset and the image feature vector corresponding to the image block sequence of each sample in the second dataset to obtain the multimodal representation vector corresponding to each sample in the second dataset. The process of generating a multimodal representation vector for each sample in the third dataset using a multimodal graph-text information processing network includes: The text embedding layer is used to convert the text segmentation sequence with masking annotations of each sample in the third dataset into text word vectors; The text word vectors converted from the text segmentation sequence with masking annotations of each sample in the third dataset are used as the multimodal representation vectors corresponding to each sample in the third dataset.

5. The training method for the multimodal pre-trained model according to claim 1, characterized in that, The step of using the multimodal representation vectors corresponding to each sample in the first dataset, the second dataset, and the third dataset to perform the masked label prediction task on the multimodal pre-trained model refers to the joint training of the description matching task and the masked word segmentation prediction task, including: Predict the masked label word for each sample in the first dataset using the multimodal representation vector corresponding to each sample in the first dataset; Using the multimodal representation vector corresponding to each sample in the second dataset, predict the location of the referent entity corresponding to each sample in the second dataset in the image corresponding to each sample in the second dataset; The multimodal representation vector corresponding to each sample in the third dataset predicts the masked word segmentation of each sample in the third dataset; Calculate the prediction loss of the masked label words in the first dataset; Calculate the loss for predicting the location of the referential entity in the second dataset; Calculate the occluded word segmentation prediction loss for the third dataset; The parameters of the multimodal pre-trained model are optimized by using the sum of the occluded tag word prediction loss of the first dataset, the referential entity location prediction loss of the second dataset, and the occluded word segmentation prediction loss of the third dataset as the training loss. Repeat the above steps until the multimodal pre-trained model converges.

6. The training method for the multimodal pre-trained model according to claim 5, characterized in that, Masked label prediction loss for the first dataset The calculation formula is as follows: ； in, Representing an image and its corresponding tag text sequence The sample composed of For image-label datasets, For tag word text sequence The text word vectors corresponding to the masked label words. For tag word text sequence The text word vectors corresponding to the unmasked tags in the text. For image The image feature vector corresponding to the image patch sequence. For tag word text sequence The cross-entropy between the joint probability distribution of candidate words corresponding to the masked tag words and the ground value; The loss for predicting the location of entities referred to by the segmented word sequences in the second dataset. The calculation formula is as follows: ； in, Representing an image The sample consisting of its corresponding denotation and its corresponding reference description. For image-referential description datasets, for The number of images contained in for and The soft dice loss between them for and Binary cross-entropy loss between them For image A true mask, representing an image The probability functions corresponding to the N image patches, For image The predicted mask, representing the image The probability functions corresponding to the N image patches, for The Middle The probability function corresponding to each image patch for The Middle The probability function corresponding to each image patch takes the value 0 or 1. When the value is 0, it means that there is no referent entity, and when the value is 1, it means that there is a referent entity. The occlusion segmentation prediction loss of the third dataset The calculation formula is as follows: ； in, For text datasets, For text The corresponding text word vectors of the masked segments in the text segmentation sequence. For text The text word vectors corresponding to the unmasked segments in the corresponding text segmentation sequence For text The cross-entropy between the joint probability distribution of candidate words corresponding to the masked words in the corresponding text segmentation sequence and the ground truth value.

7. A training device for a multimodal pre-trained model, characterized in that, The device includes: The first building module is used to build a multimodal pre-trained model that includes a multimodal image and text information processing network; The second building module is used to construct a weakly aligned image-text dataset; wherein, the weakly aligned image-text dataset includes a text dataset, an image-label dataset, and an image-reference description dataset; The training module is used to train the multimodal pre-trained model using the weakly aligned image-text dataset; The samples in the image-label dataset consist of images and their corresponding label text sequences; the label text sequence corresponding to an image is formed by concatenating the label words of all entities in the image. The samples in the image-reference description dataset consist of images and their corresponding reference descriptions; The training module is also used to perform a first preprocessing on each sample in the image-label dataset to obtain a first dataset; A second preprocessing step is performed on each sample in the image-reference description dataset to obtain a second dataset; A third preprocessing step is performed on each sample in the text dataset to obtain a third dataset; Using a multimodal graph and text information processing network, a multimodal representation vector is generated for each sample in the first dataset, a multimodal representation vector for each sample in the second dataset, and a multimodal representation vector for each sample in the third dataset. Using the multimodal representation vectors corresponding to each sample in the first dataset, the multimodal representation vectors corresponding to each sample in the second dataset, and the multimodal representation vectors corresponding to each sample in the third dataset, the multimodal pre-trained model is used to perform the task of predicting masked label words, which refers to the joint training of the description matching task and the task of predicting masked word segmentation. The first preprocessing includes at least: The image in the sample is cut into N blocks to obtain the corresponding image block sequence; By randomly masking a portion of the labeled words in the text sequence of labeled words in the sample, a text sequence of labeled words with masking annotations is obtained; The second preprocessing includes at least: The image in the sample is cut into N blocks to obtain the corresponding image block sequence; Determine the word segmentation sequence corresponding to the referential descriptions in the sample; The third preprocessing includes at least: Determine the text segmentation sequence corresponding to the text in the sample; Randomly masking some of the words in the text segmentation sequence yields a text segmentation sequence with masking annotations.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the training method for the multimodal pre-trained model as described in any one of claims 1 to 6.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the training method of the multimodal pre-trained model as described in any one of claims 1 to 6.

Images