Image-text data multi-modal training method based on image-text semantic relation alignment

Abstract

The invention discloses an image-text data multi-modal training method based on image-text semantic relation alignment. Inputting the training set not including the label into a pre-training model for feature extraction to obtain a multi-modal feature, further processing by using the multi-modal feature as input to obtain a classified predicted value, and obtaining a clustering label in combination with a clustering algorithm; the loss of image-text semantic relation alignment is obtained through classification prediction values and clustering label processing and in combination with multi-modal features, and training is carried out on a classifier with the purpose of minimizing the loss of image-text semantic relation alignment; and repeating the steps until the classifier converges. According to the innovation of the method, a deep clustering and comparative learning model and a classifier are established, and image-text semantic relationship alignment processing training is utilized, so that the ability of image-text relationship classification and image-text semantic relationship understanding is enhanced, and the processing speed and accuracy are improved.

Description

Technical Field

[0001] The invention relates to a computer graphic data processing method, and to a graphic data multimodal training method based on graphic and text semantic relationship alignment. Background Art

[0002] The study of the relationship between text and images on the Internet is very important because text and images are the two most basic elements on the website. Only a reasonable and effective combination can create wonderful content. Understanding the relationship between text and images on the Internet can enhance the user experience of the website. In addition, the relationship between text and images can also assist downstream tasks, such as multimodal named entity recognition, multimodal sentiment analysis, and multimodal disaster classification. In order to thoroughly understand the relationship between text and images, the advertising text and image dataset proposes two tasks and four types of text and image semantics. The first task is called the text task. According to whether there is semantic overlap between text and images, it is divided into two semantics: "text is represented" and "text is not represented"; the second task is called the image task. According to whether the image provides additional information for the text, it is divided into two semantics: "image provides additional information" and "image does not provide additional information". As a result, four types of image-text relationship tasks (image-texttask) can be obtained. However, the network text and images are complex, and manual annotation of the relationship between text and images is a labor-intensive task.

[0003] Universal feature representation has achieved great success in the fields of machine vision and natural language processing. By combining the visual pre-training model with the language training model, or the multimodal training model, fine-tuning some image and text tasks in the multimodal field has achieved good results. However, the above two models cannot achieve complete semantic alignment. Summary of the invention

[0004] The purpose of the present invention is to solve the above problems and provide a multimodal training method based on image-text semantic alignment. This method is different from the general pre-training method. By adding two pre-training tasks, deep multimodal clustering and multimodal comparative learning, the pre-trained multimodal model's ability to understand the semantic relationship between images and text is improved.

[0005] The technical solution adopted by the present invention to solve the technical problem is as follows: S1. Multimodal feature extraction Input the training set X without labels into the pre-training model of deep learning technology characteristics to extract features and obtain multimodal features; S2. Pseudo-label generation The multimodal features obtained in step S1 are used as input for further processing to obtain the predicted value of the classification, and the clustering algorithm is combined to obtain the clustering label; S3, using the classification prediction value and cluster label processing and combining multimodal features to obtain the loss of image-text semantic relationship alignment; In the present invention, label information is classification information.

[0006] S4, training the classifier with the goal of minimizing the loss of image-text semantic relationship alignment; S5. Repeat the above steps S1-S4 until the classifier converges.

[0007] The training set X in S1 is represented by X={x1, x2,…, x n}, where x i represents the i-th image-text pair sample, each image-text pair sample x i Contains images x i v and text x i t Two parts.

[0008] The pre-trained model in S1 is divided into two types: single-tower architecture and dual-tower architecture. The feature extraction methods of the two architectures are specifically processed as follows: A) If the pre-trained model adopts a dual-tower architecture, specifically: The pre-trained visual model Encoder_V is used as the image feature encoder, and the pre-trained language model Encoder_T is used as the text feature encoder. The image-text pair samples are input into the image feature encoder and the text feature encoder respectively to extract the image features e. i v and text features i t , and then concatenate the image features e i v and text features i t The feature vector is used as the multimodal image-text feature e i m , specifically expressed as follows: e i v =Encoder_V (x i v ) e i t =Encoder_T(x i t ) e im =e i v ⊕e i t Among them, ⊕ represents the splicing operation, x i v and x i t Respectively represent the pictures x in the image-text pair samples i v and text; B) If the pre-trained model adopts a single-tower architecture, specifically: The pre-trained multimodal model Encoder_M is used as the multimodal encoder. The image-text pair samples and the starting characters are input into the multimodal encoder to obtain the multimodal image-text features e. i m 、Image features i v and text features i t , specifically expressed as follows: e i m , e i v , e i t = Encoder_M([cls]; x i v , x i t ) The starting character is the character [cls], which is used as an input starting character when input into the multimodal encoder together with the image-text pair sample.

[0009] In step S2, the multimodal image and text features in the multimodal features are used for processing, and in step S3, the image features and text features in the multimodal features are used for processing.

[0010] The step S2 is specifically as follows: The multimodal features corresponding to all the image-text pairs in the training set are used as input and input into the clustering algorithm to obtain the classification results of the training set, that is, to obtain each image-text pair sample x i Corresponding to a cluster label, the target problem expressed by the following formula is specifically established and solved: min c ((∑ i=1 min yi ||e i m -Cy i ||2 2 ) / n) Where C represents the partitioning function, i.e., a many-to-one function that maps the serial numbers of the image-text pair samples {1, 2, ..., n} to the serial numbers of the classification types {1, 2, ..., n}, i = 1, ..., n, n represents the total number of image-text pair samples; || || 2 2 represents the square of the two norm, min c Indicates finding the minimum value of the partition function C, min yi Indicates y i Find the minimum value, y i Represents the i-th image-text pair sample x i The corresponding cluster label, y i ∈{1,2,…,k}, k represents the number of types; In step S2, the multimodal features are further input into the fully connected layer to obtain the predicted value of the classification, and the classifier is composed of the pre-trained model plus the fully connected layer.

[0011] The step S3 is specifically as follows: S31, combining the predicted value of the classification and the cluster label processing to obtain the first loss, specifically, the cluster label generated by clustering is correlated with the predicted value to obtain the first loss L1: L1=(∑ i=1 H(y i ', y i )) / n y i '= h(e i m ) Among them, H() is the cross entropy loss function, h() is the classification head function, and y i ' represents the i-th image-text pair sample x i The predicted value, y i Represents the i-th image-text pair sample x i The cluster labels of , i=1,…,n; S32, obtaining a second loss based on the alignment of the image-text semantic relationship; First, the image features e in the dataset i v and text features i t Unify to the same dimension R according to the following formula d’ , get the image vector z v ' and text vector z t ': z v =W v e v , z t =W t e t z v '=z v / |z v |, z t '=z t / |z t | W v ∈R d’×dv , W t ∈R d’×dt Among them, W v represents the text learnable matrix, W t represents the text learnable matrix, R represents the real number field, and dv represents the image feature e v The dimension size of , dt represents the text feature e t The dimension size, z t represents the text vector, z v represents the image vector, d' represents the vector z t and z v The dimension size of Then, stack all the image vectors z v 'Form the visual stacking matrix Zv, Zv∈R d’×l , stack all text vectors z t 'Form the text stacking matrix Zt, Zt∈R d’×l , and then use the scaling method to calculate the cosine similarity as the prediction matrix Z: Z=(Zt T Zv) / τ Where T represents the matrix transpose, τ represents the temperature parameter, which is used to control the prediction range; Finally, the second loss L2 is obtained using the prediction matrix Z according to the following formula: L2=(H(σ(Z), y t )+ H(σ(Z T ), y v )) / 2l Among them, σ() represents the SoftMax calculation along the row, y t and v Represent the true labels of text and image respectively, l represents the number of image-text pair samples contained in the training; S33. Combining the first loss and the second loss to obtain a total loss.

[0012] In step S33, the first loss and the second loss are added together to obtain the total loss.

[0013] The application scenario of the present invention may generally be social media graphic and text data classification, but is not limited thereto.

[0014] The innovation of the present invention lies in the establishment of a model and classifier for deep clustering and contrastive learning, and the use of image-text semantic relationship alignment processing training, thereby enhancing image-text relationship classification and improving accuracy.

[0015] The beneficial effects of the present invention are: The present invention utilizes a pre-trained model to extract multimodal features, and adopts deep clustering and contrastive learning pre-training tasks to further enhance the performance of the image-text relationship classifier.

[0016] The present invention designs two pre-training tasks, deep multimodal clustering and multimodal contrastive learning, for the alignment of image-text semantic relationships, further improving the ability of the pre-trained multimodal model to understand the image-text semantic relationship.

[0017] The present invention adopts a novel method, which can be used with a single tower pre-trained multimodal model or a dual tower pre-trained multimodal model, and experiments show that the two pre-training tasks of deep multimodal clustering and multimodal contrastive learning have an improvement effect on classification tasks. In addition, the model generated by the present invention can be applied to specific tasks such as multimodal named entity recognition, multimodal sentiment classification, and multimodal disaster classification. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 It is a working schematic diagram of the single tower structure and the double tower structure of the present invention; Figure 2 It is a schematic diagram of the deep clustering task of the present invention; Figure 3 It is a schematic diagram of the multimodal contrastive learning task of the present invention. DETAILED DESCRIPTION

[0019] The present invention will be further described below in conjunction with the accompanying drawings and embodiments.

[0020] Embodiments of the present invention are as follows: This embodiment is aimed at the application scenario of social media semantic analysis. It pre-trains the model for the text and pictures in Twitter social media, and then detects and processes the data for the text and picture relationship classification task after training. For example, search advertising recommendation, that is, search advertising recommendation, obtains user preference information through the text and picture information sent by the user.

[0021] S1. Multimodal feature extraction Input the given training set X without labels into the pre-training model of deep learning technology characteristics to extract features and obtain multimodal features; The training set X is represented by X={x1, x2,…, x n}, where x i represents the i-th image-text pair sample, each image-text pair sample x iContains images with related relationships x i v and text x i t In the specific implementation, the image-text pairs that can be used come from the image-text relationship dataset of Twitter.

[0022] The pre-trained models are divided into two types: single-tower architecture and dual-tower architecture. The feature extraction methods of the two architectures are as follows: Figure 1 The specific processing is as follows: A) If the pre-trained model uses a dual-tower architecture, specifically: The pre-trained visual model Encoder_V is used as the image feature encoder, and the pre-trained language model Encoder_T is used as the text feature encoder. The image-text pair samples are input into the image feature encoder and the text feature encoder respectively to extract the image features e. i v and text features i t , and then splice the above picture features e i v and text features i t The feature vector is used as the multimodal image-text feature e i m , specifically expressed as follows: e i v =Encoder_V (x i v ) e i t =Encoder_T(x i t ) e i m =e i v ⊕e i t Among them, ⊕ represents the splicing operation, x i v and x i t Respectively represent the pictures x in the image-text pair samples i v and text; The above picture features obtained by i v and text features i t and multimodal image-text features i mTogether they serve as multimodal features.

[0023] In a specific implementation, the pre-trained visual model Encoder_V adopts the pre-trained visual model ResNet or ViT, and the pre-trained language model Encoder_T adopts the pre-trained language model including Bert.

[0024] B) If the pre-trained model uses a single-tower architecture, specifically: The pre-trained multimodal model Encoder_M is used as the multimodal encoder. The image-text pair samples and the starting character [cls] are input into the multimodal encoder respectively to obtain the multimodal image-text features e. i m 、Image features i v and text features i t , together as multimodal features, are specifically expressed as follows: e i m , e i v , e i t = Encoder_M([cls]; x i v , x i t ) The starting character is the character [cls], which is used as the input start character when input into the multimodal encoder together with the image-text pair sample, followed by the image-text pair sample.

[0025] In the specific implementation, the pre-trained multimodal model Encoder_M adopts models such as Flan-T5.

[0026] S2. Pseudo-label generation The multimodal image-text feature e in the multimodal feature obtained in step S1 is used i m The input is further processed to obtain the predicted value of the classification, and combined with the clustering algorithm to obtain the cluster label; In the specific implementation, the data set in the present invention is a Twitter image-text relationship data set, which is classified into four categories: 1. The text is represented and the image adds additional information 2. The text is not represented and the image adds additional information 3. The text is not represented and the image does not add additional information 4. The text is marked and the text does not add additional information.

[0027] The number of types k is preset, and the multimodal features corresponding to all the image-text pairs in the training set are used as input to the clustering algorithm to obtain the classification results of the training set, that is, to obtain each image-text pair sample x iCorresponding to a cluster label, the target problem expressed by the following formula is specifically established to solve the problem with the goal of minimization: min c ((∑ i=1 min yi ||e i m -Cy i ||2 2 ) / n) Where C represents the partitioning function, i.e., a many-to-one function that maps the serial numbers of the image-text pair samples {1, 2, ..., n} to the serial numbers of the classification types {1, 2, ..., n}, i = 1, ..., n, n represents the total number of image-text pair samples; || || 2 2 represents the square of the two norm, min c Indicates finding the minimum value of the partition function C, min yi Indicates y i Find the minimum value, y i Represents the i-th image-text pair sample x i The corresponding cluster label, y i ∈{1,2,…,k}, k represents the number of types; The above steps take the generated multimodal features as input, use the clustering algorithm to divide the samples into k clusters, and save the identifier (ID) y of the cluster where the sample is located i as pseudo labels.

[0028] like Figure 2 As shown, the specific implementation uses the k-means algorithm as the clustering algorithm, k represents the number of cluster types, and according to the given data set, k=4 here.

[0029] At the end of step S2, the multimodal features are further input into the fully connected layer to obtain the classification prediction value, and the classifier is composed of the pre-trained model plus the fully connected layer.

[0030] S3. Use the classification prediction value and cluster label processing and combine the image features in the multimodal features i v and text features i t And combined with the known label information to obtain the loss of image-text semantic relationship alignment; S31, combining the predicted value of the classification and the cluster label processing to obtain the first loss, specifically, the cluster label generated by clustering is correlated with the predicted value to obtain the first loss L1 generated by the current task: L1=(∑ i=1 H(y i ', y i )) / n yi '= h(e i m ) Among them, H() is the cross entropy loss function, h() is the classification head function, and y i ' represents the i-th image-text pair sample x i The predicted value, y i Represents the i-th image-text pair sample x i The clustering label of the current task is shown in Appendix 2, i=1,…,n; like Figure 2 As shown in the figure, the classification head is the last linear layer of the model, that is, the fully connected layer, which is used to save the probabilities of each category. The subscript of the maximum probability is the classification result.

[0031] S32, image-text similarity comparison: obtain the second loss based on image-text semantic relationship alignment; The dataset is divided into multiple batches in advance, that is, all the image-text pair samples in the dataset are divided into different batches. It is known that a batch contains l image-text pairs, and it is predicted that l×l possible pairing forms actually occur in this batch.

[0032] First, if Figure 3 As shown, the image features e of all image-text pair samples in the same batch in the dataset are i v and text features i t Unify to the same dimension R according to the following formula d’ , get the image vector z v ' and text vector z t ': z v =W v e v , z t =W t e t z v '=z v / |z v |, z t '=z t / |z t | e v ={e1 v ,…, e l v},e t ={e1 t ,…, e l t} W v ∈R d’×dv , Wt ∈R d’×dt Among them, W v represents the text learnable matrix, W t Represents a text learnable matrix, which is a two-dimensional matrix. Each number in the matrix is ​​a real number. The shape of the matrix is ​​d'×dv or d'×dt. R represents the real number field, and dv represents the image feature e v The dimension size of , dt represents the text feature e t The dimension size, z t represents the text vector, z v represents the image vector, d' represents the vector z t and z v The dimension size of Then, stack all the image vectors z v 'Form the visual stacking matrix Zv, Zv∈R d’×l , stack all text vectors z t 'Form the text stacking matrix Zt, Zt∈R d’×l , l represents the number of image-text pairs input in training, and then the cosine similarity is calculated as the prediction matrix Z using the scaling method: Z=(Zt T Zv) / τ Among them, T represents matrix transpose, τ represents temperature parameter, which is used to control the prediction range; parameter τ is pre-set, and this variable is used to control the generation of prediction matrix Z.

[0033] Finally, the prediction matrix Z is used to obtain the second loss L2 generated by the current task according to the following formula: L2=(L t + L v ) / 2l=( H(y t ', y t )+ H(y v ', y v )) / 2l=( H(σ(Z), y t )+ H(σ(Z T ),y v )) / 2l Among them, L t and L v Represent the losses of text tasks and image tasks respectively, σ() represents the SoftMax calculation along the row, and y t and v Represent the previously known text and image true labels, y t ' and y v'represent the predicted labels of text and picture respectively, and l represents the number of picture-text pair samples contained in the training.

[0034] S33, combining the first loss and the second loss to obtain a total loss, specifically, adding the first loss and the second loss to obtain the total loss.

[0035] S4, training the classifier with the goal of minimizing the loss of image-text semantic relationship alignment; S5, iterative optimization: repeat the above steps S1-S4 until the pre-trained model and classifier converge.

[0036] In the specific implementation, the classifier is constructed by adding a pre-trained model to a fully connected layer, and the loss functions L1 and L2 are added to obtain the comprehensive loss function L of the classifier. The parameters of the classifier are further updated using optimizers such as Adam to achieve training.

[0037] After training, the trained classifier was used to process the image and text data, and the classification result of F1 was 0.578.

[0038] The experiments and results of this embodiment are as follows: In order to verify the effectiveness of the pre-training method, the following four sets of experiments were set up.

[0039] In Experiments 1 and 2, Twitter ITR datasets were used to compare the pre-training method proposed by the present invention with that without the pre-training method, and the following results were obtained through linear probing or fine-tuning (as shown in Tables 1 and 2).

[0040] Table 1 Linear detection F1 score results on Twitter ITR dataset (%)

[0041] The above F1 score is an indicator of machine learning.

[0042] As can be seen from the above table, the F1 index of the original model can be improved by pre-training plus linear detection.

[0043] Table 2 Fine-tuning F1 score results on Twitter ITR dataset (%)

[0044] It can be seen from the above table that the F1 index of the original model can also be improved by pre-training and fine-tuning, and the method of the present invention performs well in various tasks.

[0045] In Experiment 3, the Twitter ITR dataset is used to compare the latest models, such as the multimodal GPT series related models, and the text and image classification results are obtained by using prompt words, as shown in Table 3.

[0046] Table 3 Comparison of F1 scores (%) with the state-of-the-art results on the Twitter ITR dataset Model text image Graphics Vempala Model 58.0 81.0 44.0 Lu Model 58.6 87.7 52.4 Hessel Model - - 53.4 Xu Model - - 53.4 Villegas Model - - 55.9 Sun Model 63.6 87.6 54.0 BLIP-2 Model 39.3 51.5 21.1 Mini-GPT4 model 41.1 52.7 22.9 Llava-Adapter V2 Model 42.3 54.2 24.1 Method of the present invention -Pre-trained ViLT model 64.0 87.6 55.3 -Pre-trained BERTweet+ViT model (this invention) 64.1 89.5 57.8 As mentioned above, ViLT is a model of a pre-trained multimodal single-tower architecture, and Bertweet+ViT is a model of a pre-trained multimodal dual-tower architecture. (The present invention forms a framework for pre-training, fine-tuning or linear detection. Bertweet+ViT is the best model obtained through the framework of the present invention and is a representative of the architecture model.) It can be seen from the table that, by using the pre-training method of the present invention, both the single-tower architecture and the dual-tower architecture model outperform other models in the evaluation indicator F1 score.

Claims

1. A multimodal training method for image-text data based on image-text semantic relationship alignment, characterized in that: The methods include the following: S1. Multimodal feature extraction Input the training set X without labels into the pre-training model of deep learning technology characteristics to extract features and obtain multimodal features; S2. Pseudo-label generation The multimodal features obtained in step S1 are used as input for further processing to obtain the predicted value of the classification, and the clustering algorithm is combined to obtain the clustering label; S3, using the classification prediction value and cluster label processing and combining multimodal features to obtain the loss of image-text semantic relationship alignment; S4, training the classifier with the goal of minimizing the loss of image-text semantic relationship alignment; S5. Repeat the above steps S1-S4 until the classifier converges.

2. The method for multimodal training of image-text data based on image-text semantic relationship alignment according to claim 1, characterized in that: The training set X in S1 is represented by X={x1, x2,…, x n }, where x i represents the i-th image-text pair sample, each image-text pair sample x i Contains images x i v and text x i t Two parts.

3. The method for multimodal training of image-text data based on image-text semantic relationship alignment according to claim 1, characterized in that: The pre-trained model in S1 is divided into two types: single-tower architecture and dual-tower architecture. The feature extraction methods of the two architectures are specifically processed as follows: A) If the pre-trained model adopts a dual-tower architecture, specifically: The pre-trained visual model Encoder_V is used as the image feature encoder, and the pre-trained language model Encoder_T is used as the text feature encoder. The image-text pair samples are input into the image feature encoder and the text feature encoder respectively to extract the image features e. i v and text features i t , and then concatenate the image features e i v and text features i t The feature vector is used as the multimodal image-text feature e i m , specifically expressed as follows: e i v =Encoder_V (x i v ) e i t =Encoder_T(x i t ) And i m =and i v ⊕e i t Among them, ⊕ represents the splicing operation, x i v and x i t Respectively represent the pictures x in the image-text pair samples i v and text; B) If the pre-trained model adopts a single-tower architecture, specifically: The pre-trained multimodal model Encoder_M is used as the multimodal encoder. The image-text pair samples and the starting characters are input into the multimodal encoder to obtain the multimodal image-text features e. i m 、Image features i v and text features i t , specifically expressed as follows: e i m , e i v , e i t = Encoder_M([cls]; x i v , x i t ) The starting character is the character [cls], which is used as an input starting character when input into the multimodal encoder together with the image-text pair sample.

4. The method for multimodal training of image-text data based on image-text semantic relationship alignment according to claim 1, characterized in that: In step S2, the multimodal image and text features in the multimodal features are used for processing, and in step S3, the image features and text features in the multimodal features are used for processing.

5. The method for multimodal training of image-text data based on image-text semantic relationship alignment according to claim 1, characterized in that: The step S2 is specifically as follows: The multimodal features corresponding to all the image-text pairs in the training set are used as input and input into the clustering algorithm to obtain the classification results of the training set, that is, to obtain each image-text pair sample x i Corresponding to a cluster label, the target problem expressed by the following formula is specifically established and solved: my c ((∑ i=1 my yi ||e i m -Cy i ||2 2 ) / n) Where C represents the partitioning function, i.e., a many-to-one function that maps the serial numbers of the image-text pair samples {1, 2, ..., n} to the serial numbers of the classification types {1, 2, ..., n}, i = 1, ..., n, n represents the total number of image-text pair samples; || || 2 2 represents the square of the two norm, min c Indicates finding the minimum value of the partition function C, min yi Indicates y i Find the minimum value, y i Represents the i-th image-text pair sample x i The corresponding cluster label, y i ∈{1,2,…,k}, k represents the number of types; In step S2, the multimodal features are further input into the fully connected layer to obtain the predicted value of the classification, and the classifier is composed of the pre-trained model plus the fully connected layer.

6. The method for multimodal training of image-text data based on image-text semantic relationship alignment according to claim 1, characterized in that: The step S3 is specifically as follows: S31, combining the predicted value of the classification and the cluster label processing to obtain the first loss, specifically, the cluster label generated by clustering is correlated with the predicted value to obtain the first loss L1: L1=(∑ i=1 H(and i ', and i )) / n y i ’= h(e i m ) Among them, H() is the cross entropy loss function, h() is the classification head function, and y i ' represents the i-th image-text pair sample x i The predicted value, y i Represents the i-th image-text pair sample x i The cluster labels of , i=1,…,n; S32, obtaining a second loss based on the alignment of the image-text semantic relationship; First, the image features e in the dataset i v and text features i t Unify to the same dimension R according to the following formula d’ , get the image vector z v ' and text vector z t ': With v =In v and v ,With t =In t and t With v '=from v / |from v |,from t '=from t / |from t | IN v ∈R d’×dv ,IN t ∈R d’×dt Among them, W v represents the text learnable matrix, W t represents the text learnable matrix, R represents the real number field, and dv represents the image feature e v The dimension size of , dt represents the text feature e t The dimension size, z t represents the text vector, z v represents the image vector, d' represents the vector z t and z v The dimension size of Then, stack all the image vectors z v 'Form the visual stacking matrix Zv, Zv∈R d’×l , stack all text vectors z t 'Form the text stacking matrix Zt, Zt∈R d’×l , and then use the scaling method to calculate the cosine similarity as the prediction matrix Z: Z=(Zt T Zv) / τ Where T represents the matrix transpose, τ represents the temperature parameter, which is used to control the prediction range; Finally, the second loss L2 is obtained using the prediction matrix Z according to the following formula: L2=( H(σ(Z), y t )+ H(σ(Z T ), y v )) / 2l Among them, σ() represents the SoftMax calculation along the row, y t and v Represent the true labels of text and image respectively, l represents the number of image-text pair samples contained in the training; S33. Combining the first loss and the second loss to obtain a total loss.

7. The method for multimodal training of image-text data based on image-text semantic relationship alignment according to claim 6, characterized in that: In step S33, the first loss and the second loss are added together to obtain the total loss.

Images