An entity alignment method based on multi-modal collaborative representation learning

Abstract

The application discloses an entity alignment method based on multi-modal collaborative representation learning. On enhanced data, the initial semantic information of text and images is extracted based on a BERT model and a deep residual network, and the text and image features are projected into the same semantic space. The triplet loss loss is combined to make the text and image positive samples more similar and the text and image negative samples more different in the space. After training, the feature extraction and similarity calculation are performed on the unlabeled text and image data, the high-confidence entity alignment result is added to the seed data set, the model is iteratively updated, and the alignment of all text and images in the multi-modal data set is completed. The method uses a multi-modal representation learning method based on a pre-training model to optimize entity representation, does not need to manually construct entity features, simultaneously adopts an iterative data updating and network training process, greatly reduces the requirement for the amount of manually labeled data in the initial seed data set, saves a large amount of manpower and cost, and can obtain more accurate feature representation and alignment result.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of knowledge graph in natural language processing, and particularly relates to an entity alignment method based on multi-modal collaborative representation learning. BACKGROUND

[0002] Entity alignment (EA) is also known as entity matching, which is a key technology in knowledge fusion process, aiming to determine whether two entities point to the same object in the real world. Due to the rich sources of knowledge content and the different understanding of people, the expressions referring to the same entity will be different, which leads to the problem of heterogeneity and redundancy of knowledge in the same or different knowledge graphs, and thus the entity alignment technology is needed to merge, which is of great significance to the technical exploration of data fusion and knowledge sharing.

[0003] The research on entity alignment can be mainly divided into two categories. One is the traditional entity alignment method, in which researchers mainly use the way of manually constructing features to represent the character features, attribute features and relationship features of entities, and then determine whether the entities can be aligned through the calculation of feature similarity. The similarity measurement methods used include Term Frequency-Inverse Document Frequency (TFIDF), active learning and machine learning classifier, and NGram matching / edit distance / digit matching, etc. In addition, synonym sets and semantic verification and filtering machines can be used to improve the performance of the entity alignment algorithm based on similarity calculation. However, due to the differences in different fields and the different attributes of entities, it is difficult for this method to give a unified similarity calculation function, and the discrete attribute information ignores the implicit semantic information, resulting in limited alignment effect.

[0004] The other is the entity alignment method based on knowledge representation learning, which mainly represents different entities as low-dimensional vectors in the same space through knowledge representation learning technology, and then performs entity alignment by calculating the distance or similarity between entities in the vector space. Obviously, the focus of this method is on the knowledge representation learning method of entities. Representation learning (also known as feature learning or embedding representation, etc.) aims to use machine learning technology to represent objects as low-dimensional dense vectors, and the distance between two vectors reflects the semantic relationship between the corresponding two objects. Using representation learning for knowledge representation, i.e. knowledge representation learning, can realize the vector representation of entities and the relationship between entities in the knowledge graph.

[0005] With the rapid development of deep learning technology, the entity alignment method based on knowledge representation learning has become the mainstream method at present. According to the different ways of knowledge representation learning, this kind of method can be divided into two frameworks. One framework of entity alignment firstly uses translation model, GNN and other knowledge representation learning technologies to embed the knowledge graph. At this time, the embedding space of different knowledge graphs is different. Then, according to the aligned entity pair, the embedding space of different knowledge graphs is mapped into the same vector space, and then the entity similarity is calculated to determine whether the entity alignment can be performed; another framework of entity alignment directly maps different knowledge graphs into the same vector space by fusing the elements in different knowledge graphs in the data preparation stage, and finally obtains the entity alignment result according to the distance or similarity between entities in the vector space.

[0006] At present, the entity alignment method based on knowledge representation learning mostly focuses on the single knowledge graph and single modal entity alignment problem. However, with the expansion of the application range of knowledge graph technology, multi-source and multi-modal entity alignment has become a problem to be solved by the academic and industrial circles. The problem of the existing work is that it only focuses on single modal data (such as text), resulting in that the entity feature information in other modal data (such as image) is not effectively utilized. The difficulty of multi-modal entity alignment lies in how to eliminate the heterogeneity between different modal data, and thus multi-modal knowledge representation learning becomes the key problem of this research. In existing research, translation model or neural network is mainly used for related knowledge representation learning, and the calculation cost of these models is large, and they are more dependent on the labeled data set. SUMMARY

[0007] In view of the problem of large calculation overhead required by multi-modal knowledge representation learning in the multi-modal entity alignment process, the present disclosure provides an entity alignment method based on multi-modal collaborative representation learning, which uses BERT pre-training language model and ResNet-200 deep residual network model to extract the initial semantic information of text and image respectively, avoids the cumbersome manual construction of features in traditional entity alignment methods, and does not need to train the knowledge representation of entities from scratch, but only needs to fine-tune a small amount of labeled data for specific tasks, so the utilization rate of labeled data is high.

[0008] In view of the problem of less labeled data in multi-modal entity alignment research, the present disclosure uses data enhancement to perform horizontal flipping, vertical flipping, random cropping, random erasing and other operations on picture data, so that the data amount is multiplied, and the robustness of the model is also enhanced, ensuring that the effective feature representation of the picture data is learned. In addition, the present disclosure uses an iterative mechanism to add the newly aligned entity pair with high confidence to the already aligned seed set data, helping the alignment learning of the next round, so as to achieve the purpose of expanding the labeled data set.

[0009] The entity alignment method based on multimodal collaborative representation learning disclosed herein includes the following steps:

[0010] Step 1: Manually label the data in both text and image modalities, align semantically identical text and images to obtain the initial seed dataset;

[0011] Step 2: Using the Chinese pre-trained language model BERT, the MaskedLM language learning model is used to optimize the network on all the text data in this work to learn the network parameters suitable for multimodal entities, thereby obtaining the multimodal entity BERT model. The output of the multimodal entity BERT model is used as the semantic information of the text.

[0012] Step 3: Based on the initial seed dataset, data augmentation is used to enhance the images in the dataset. On this basis, a triplet of <text-positive image sample-negative image sample> is constructed, and the semantic features of the text are extracted using the optimized BERT model. The features of the positive and negative image samples are extracted using the ResNet-200 deep residual network. The semantic features of the text and the features of the positive and negative image samples are projected into a new low-dimensional semantic feature space through a single-layer perceptron. The similarity between the semantic features and the image features is measured based on the cosine value of the vector. On this basis, a triplet loss function is constructed to make the text more similar to the positive image samples in the low-dimensional semantic space, and more dissimilar to the negative image samples.

[0013] Step 4: Based on the model trained in Step 3, perform feature extraction and cosine similarity calculation on the unlabeled text and image data, add high-confidence <text-image> pairs to the seed dataset, and retrain the model based on Step 3 to achieve iterative expansion of data and iterative update of the model until all data is iteratively labeled and the model is trained on this basis.

[0014] Compared with the prior art, the beneficial effects of this disclosure are: 1. It avoids the tedious process of manually constructing feature engineering. It uses the BERT pre-trained language model and the ResNet-200 deep residual network model to obtain the original feature representation of text and image data. Furthermore, through spatial transformation and training optimization, the feature vectors of text and images are projected into the same space, thus enabling the extraction of knowledge representation of multimodal data in the same low-dimensional space, thereby enabling further similarity judgment.

[0015] 2. The triplet loss function is introduced as the loss function to constrain positive and negative samples. At the same time, the distance between positive and negative samples is controlled by setting a margin threshold, so that similar samples are closer in the low-dimensional representation space of multimodal data, and different samples are farther apart.

[0016] 3. Data augmentation methods are used on the original labeled dataset. In combination with the need for image retrieval through text in real tasks, the image data is subjected to operations such as horizontal flipping, vertical flipping, random cropping, and random erasing, which multiplies the training data. This transformation can ensure that effective feature representations are learned during training, and achieve the effect of finding the same entity for alignment before and after the transformation.

[0017] 4. An iterative mechanism was introduced to add high-confidence alignment results to the aligned dataset to guide the next round of training, while also achieving the goal of increasing the labeled dataset. Attached Figure Description

[0018] The above and other objects, features and advantages of this disclosure will become more apparent from the more detailed description of exemplary embodiments of this disclosure taken in conjunction with the accompanying drawings, in which the same reference numerals generally represent the same components.

[0019] Figure 1 This is a framework diagram of the entity alignment method based on multimodal collaborative representation learning according to this disclosure. Detailed Implementation

[0020] Preferred embodiments of the present disclosure will now be described in more detail with reference to the accompanying drawings. While preferred embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that the present disclosure will be thorough and complete, and will fully convey the scope of the present disclosure to those skilled in the art.

[0021] This disclosure presents an entity alignment method based on multimodal collaborative representation learning. On augmented data, initial semantic information of text and images is extracted using a BERT pre-trained language model and a ResNet-200 deep residual network model. Text and image features are projected onto the same semantic space through spatial transformation. Triplet loss is used to make text more similar to positive image samples and more dissimilar to negative image samples in this space. After training, feature extraction and cosine similarity calculation are performed on unlabeled text and image data. High-confidence entity alignment results are added to a seed dataset for iterative model updates. The seed dataset and network model are iteratively updated until alignment of all text and images in the multimodal dataset is completed.

[0022] The algorithm framework according to exemplary embodiments of this disclosure is attached. Figure 1 As shown, it specifically includes:

[0023] Step 1: Obtain the seed dataset through manual annotation.

[0024] Step 2: Utilizing the pre-trained Chinese language model BERT on a multimodal text dataset, network optimization is performed using the Masked LM language learning model. Specifically:

[0025] The parameters of the BERT model were tuned using Masked LM on all multimodal text datasets. During training, the following strategy was adopted: 15% of the words in the sentence were randomly selected for masking. Among the selected masked words, 80% were replaced with [Mask], 10% were not replaced, and the remaining 10% were replaced with a random word.

[0026] Multimodal text dataset T1~T p For all multimodal text data; the maximum number of epochs in the BERT model is Epoches, and the number of batches per epoch is batch_per_epoch; the BERT base model is Bert_base_chinese, and the maximum length of each sentence is max_len;

[0027] For all epochs in the BERT model, perform the following training procedure to obtain the optimized BERT model parameters:

[0028] For each batch in the epoch, execute S201 to S205:

[0029] S201, The multimodal text data set The dataset is divided into training, testing, and validation sets according to a ratio of 80%-10%-10%.

[0030] S202, pad the input sentences in the training set with zeros or truncate them to a length of max_len, and obtain the index I1 of the sentence segmentation;

[0031] S203, randomly select 15% of the words in the sentence to mask. Among the selected masked words, 80% are replaced with [Mask], 10% are not replaced, and the remaining 10% are replaced with a random word.

[0032] S204: The masked sentence is input into the BERT base model Bert_base_chinese to obtain the feature vector, followed by θ0 to predict the word segmentation index I2 corresponding to each position of the sentence;

[0033] S205 uses the Adam optimizer to minimize the difference between I1 and I2, defined as the first loss function L(θ,θ0); when the first loss function on the validation set no longer decreases within a certain epoch, an early stopping strategy is adopted.

[0034] The first loss function L(θ,θ0) is defined as follows:

[0035]

[0036] Where θ is the parameter of the Encoder part in the BERT model, the input is processed by θ to obtain the feature vector, θ0 is the parameter following θ in the Masked LM task, |V| is the size of the dictionary composed of the masked words; m i The word that is masked; p(m=m) i |θ,θ0) means that, given the learned parameters θ and θ0, the predicted word m is the masked word m. i The probability of;

[0037] In the training process, during the first two epochs of the BERT model, θ is fixed, and the learning rate is lr = 5e -4 Adjust θ0, and in subsequent epochs, use lr = 1e -5 The learning rate is adjusted simultaneously by θ and θ0 until the stopping condition is met. Here, e is in scientific notation. -5 =1*10 -5 .

[0038] Step 3 specifically involves: For the images in the seed dataset, data augmentation is applied to enhance them. Augmentation methods include horizontal flipping, vertical flipping, random cropping, and random erasing. The augmented images are then paired with the text information aligned to the original images to form new <text-image> pairs, which are added to the seed dataset to create the augmented seed dataset.

[0039] Augmented seed dataset D1~D N For all aligned <text-image> pairs; the set of text information in the seed data set is Ts1~ N The text information consists of all aligned text-image pairs; the image information set in the seed dataset is... Is1~ N For all aligned <text-image> pairs, extract the image information; based on this, construct a <text-image positive sample-image negative sample> triplet dataset, where <text-image positive sample> is the seed dataset. Aligned <text-image> pairs are provided, while <image negative samples> are obtained from the seed dataset. Randomly select from any other image that is not aligned with the text in the current text-image pair to obtain the final set of text-image positive sample-image negative sample triplet data. DT1~DT M This represents all the <text-image positive sample-image negative sample> triples. Here, M is an integer multiple of N: if one image negative sample is selected for each <text-image positive sample> pair, then M = N; if two image negative samples are selected for each <text-image positive sample> pair, then M = 2N; if three image negative samples are selected for each <text-image positive sample> pair, then M = 3N, and so on. The actual configuration can be adjusted freely according to requirements.

[0040] Let the maximum value of an epoch be Epoches1, and the number of batches per epoch be batch_per_epoch1. The optimized BERT model, Bert_fine_tune, has a maximum sentence length of max_len; the image feature extraction model is ResNet-200, and the input size of each image is input*input; the feature space dimension after text and image alignment is d; for all epochs in this model, the learned model parameters are obtained by performing the following training process, achieving the alignment of features obtained from text and images in the same low-dimensional space:

[0041] For each batch in the epoch, execute SS1 to SS11:

[0042] SS1. Pad the text t in <text-image positive sample-image negative sample> with zeros or truncate it to a length of max_len;

[0043] SS2. Input the text into Bert_fine_tune to obtain the text feature vector.

[0044] SS3. Eigenvectors Followed by a single-layer perceptron network θ t The text features are projected into a feature space of dimension d to obtain the feature vector.

[0045] SS4. Resize the positive image samples (image_p) in <text-image positive samples-image negative samples> to the input size of the ResNet-200 network (input*input).

[0046] SS5. Input the resized positive samples of the image into ResNet-200, whose network parameters are θ. R Obtain image feature vectors

[0047] SS6. Eigenvectors Followed by a single-layer perceptron network θ I The positive sample features of the image are projected into a feature space of dimension d to obtain the feature vector.

[0048] SS7. Resize the negative image sample image_n in <text-image positive sample-image negative sample> to the input size input*input of the ResNet-200 network;

[0049] SS8. Input the negative samples of the resized image into ResNet-200, whose network parameters are θ. r Obtain image feature vectors

[0050] SS9. Eigenvectors Followed by a single-layer perceptron network θ I The positive sample features of the image are projected into a feature space of dimension d to obtain the feature vector.

[0051] SS10. Construct the second loss function L T (θ,θ t ,θ R ,θ I The Adam optimizer is used to minimize the second loss function.

[0052] SS11. When the loss on the validation set no longer decreases within a certain number of epochs, adopt the early stopping strategy.

[0053] Specifically, the second loss function is defined as follows:

[0054]

[0055] in,

[0056]

[0057]

[0058]

[0059]

[0060]

[0061] This involves two single-layer perceptron networks, one of which is a single-layer perceptron network θ for text. t One is a single-layer perceptron network for images, where positive and negative image samples share the same single-layer perceptron network θ. I .

[0062] Step 4 specifically involves: based on the model trained in Step 3, extracting features of all text and all images in the low-dimensional space from the multimodal data, traversing all <text-image> pairs, and calculating their similarity E. w Based on this similarity, text-image alignment is performed, and high-confidence text-image pairs are added to the seed dataset. Step 3 is repeated to generate a new set of triplet datasets, and the training θ is iteratively updated. t ,θ I Network parameters.

[0063] The seed set consisting of the iteratively updated aligned multimodal <text-image> pairs For all aligned <text-image> pairs in the iteratively updated seed set; multimodal text data set. T1~T p For all multimodal text data; multimodal text image set I1~I Q For all multimodal image data; confidence threshold τ; optimized network parameters θ, θ t ,θ R ,θ I .

[0064] Execute SSS1 to SSS11:

[0065] SSS1. A collection of multimodal text data. The text T in i Pad with zeros or truncate to a length of max_len;

[0066] SSS2. Input text θ and θ t Obtain text feature vectors

[0067] SSS3. A collection of multimodal image data. Image I j resize is the input size of the ResNet-200 network, input * input;

[0068] SSS4. Input the resized image into θ R and θ I Obtain image feature vectors

[0069] SSS5. Traversing a Multimodal Text Data Set The text T in i ,i=1,...,P and multimodal image data set Image I j For each j = 1, ..., Q, obtain the corresponding set of text feature vectors. and image feature vector set

[0070] SSS6. Calculate according to the formula in step 3. P texts and The similarity set is obtained by calculating the pairwise cosine similarity of Q images.

[0071] SSS7. Extracting Similarity Sets The text-image pairs corresponding to values ​​with a similarity greater than the confidence threshold τ constitute a set of text-image pairs. Where K represents all text-image pairs with a similarity greater than the confidence threshold τ;

[0072] SSS8. will Merge into the seed set formed by the aligned multimodal <text-image> pairs updated in the previous iteration. If new seed data is added, step 3 is executed to iterate and train the model first, and then SSS1-SSS8 are executed to update the iterative data; otherwise, the process ends, and the final text-image alignment result and network parameters of the multimodal data alignment model are obtained.

[0073] As can be seen, this disclosure uses a multimodal representation learning method based on pre-trained model tuning for entity representation, which does not require manual construction of entity features. At the same time, the iterative data update and network training process adopted also greatly reduces the amount of manually labeled data required in the initial seed dataset, saving a lot of manpower and costs.

[0074] By introducing the triplet loss function, we can better model the details of the samples by comparing and measuring the differences between two samples in the input, thereby learning more accurate feature representations.

[0075] Performing operations such as horizontal flipping, vertical flipping, random cropping, and random erasing on image data can multiply the amount of training data, making up for the problem of insufficient labeled data. At the same time, introducing an iterative mechanism into the model adds the entity alignment results to the already aligned entity pairs, thereby achieving the purpose of automatically expanding the labeled data.

[0076] The above technical solutions are merely exemplary embodiments of the present invention. For those skilled in the art, based on the application methods and principles disclosed in the present invention, it is easy to make various types of improvements or modifications, and not limited to the methods described in the specific embodiments of the present invention. Therefore, the methods described above are merely preferred and not restrictive.

Claims

1. An entity alignment method based on multimodal collaborative representation learning, comprising the following steps: S1, through manual annotation and alignment, establish the initial <text-image> pair seed dataset; S2, Obtain a BERT model suitable for multimodal text data; S3 includes: S31, based on image enhancement and expansion, expand the <text-image> pairs in the seed set; S32, expand the <text-image> pairs in the seed set into <text-image positive sample-image negative sample> triples; S33, use the BERT model from step S2 to extract the semantic features of the text, and use a deep residual network to extract the features of positive and negative image samples respectively; project these features into a low-dimensional semantic feature space through a single-layer perceptual network respectively; construct a loss function with the goal of making the text more similar to positive image samples in the low-dimensional semantic feature space and more different from negative image samples; and train the single-layer perceptual network for feature projection using the data in the seed set. S4. Perform feature extraction and similarity calculation on unlabeled text and images, add high-confidence <text-image> pairs to the seed dataset, retrain the model based on step S3, and iteratively expand and update the data until all data is iteratively labeled. The specific method of step S33 includes: set up BERT The model is Bert_fine_tune The maximum length of each sentence is max_len The image feature extraction model adopts ResNet-200 The input size for each image is [size to be specified]. input*input Dimensions of the feature space after text and image alignment d ; Then targeting epoch Each of them batch ,implement SS 1~ SS 11: SS 1. Extract the text from <text-image positive sample-image negative sample>. t Padding with zeros or truncating to a length of max_len ; SS 2. Enter the text into Bert_ fine_tune In the process, text feature vectors are obtained. ; SS 3. Feature vector Follow A single-layer perceptron network projects text features onto a dimension of 1. d The feature vector is obtained from the feature space. ; SS 4. Resize the image_p in <text-image positive sample-image negative sample> to [size missing]. ResNet- 200 Network input size input*input ; SS 5. Input the resized positive sample image into... ResNet-200 In this context, its network parameters are: Obtain image feature vectors ; SS 6. Eigenvectors Follow A single-layer perceptron projects positive sample features of an image onto a dimension of 1. d The feature vector is obtained from the feature space. ; SS 7. Resize the negative image sample `image_n` in `<text-image positive sample-image negative sample>` to... ResNet- 200 Network input size input*input ; SS 8. Input the resized negative sample of the image into... ResNet-200 In this context, its network parameters are: Obtain image feature vectors ; SS 9. Eigenvectors Follow A single-layer perceptron projects positive sample features of an image onto a dimension of 1. d The feature vector is obtained from the feature space. ; SS 10. Construct the second loss function ,use Adam The optimizer minimizes the second loss function; SS 11. When the validation set is loss In a certain epoch Stop training when the internal rate stops decreasing. The definition of the second loss function includes: In the formula, 。 2. The alignment method according to claim 1, characterized in that, The method of step S2 includes: using BERT In the model MaskedLM Language learning task tools utilize multimodal text datasets to... BERT The model parameters are trained and tuned to obtain a BERT model suitable for multimodal text data, specifically including: Suppose a multimodal text dataset , T 1 ~T p For all multimodal text data; BERT In the model epoch The maximum value is Epoches Each epoch of batch Number of batch_per_epoch ; BERT The basic model is Bert_base_chinese The maximum length of each sentence is max_len ; against BERT All of them in the model epoch The optimized result was obtained by performing the following training procedure. BERT Model parameters: against epoch Each of them batch ,implement S 201~ S 205: S 201, a collection of multimodal text data. The dataset is divided into training set, test set, and validation set according to a ratio of 80%-10%-10%. S 202, pad the input sentences in the training set with zeros or truncate them to a length of [length missing]. max _ len Get the index of sentence segmentation. I 1; S 203. Randomly select 15% of the words in the sentence for... Mask In choosing as Mask Of the words used, 80% employ [ Mask 10% will be replaced, 10% will not be replaced, and the remaining 10% will be replaced with a random word; S 204, Get Mask The following sentence is entered into BERT Basic Model Bert_base_chinese In the process, the feature vector is obtained, followed by... Predict the index of the word segment at each position in the sentence. I 2; S 205, Use Adam optimizer minimization I 1 and I The difference between 2 is defined as the first loss function. When the first loss function on the validation set is within a certain range... epoch Stop training when the internal rate stops decreasing. First loss function The definition is as follows: in, yes BERT In the model Encoder Some parameters, input through Obtain the feature vector. Is MaskedLM During the mission, The parameters that follow; It was by mask The size of the dictionary comprised of the words; m i Indicates being mask The word; This indicates that, given the learned parameters and In this case, the predicted words m It was just that mask words m i The probability of; In the training process, BERT The first two models epoch In the middle, fixed Adjust at the first learning rate In the subsequent epoch In the middle, adjust simultaneously with the second learning rate and Continue until the stopping condition is met.

3. The alignment method according to claim 2, characterized in that, The first learning rate is taken The second learning rate is taken , e is the symbol for scientific notation.

4. The alignment method according to claim 1, characterized in that, The specific method of step S31 includes: The images in the initial seed dataset are enhanced and expanded to form new <text-image> pairs with the text information aligned with the original images, and then added to the seed dataset. The methods for enhancing and expanding the images include horizontal flipping, vertical flipping, random cropping, and random erasing.

5. The alignment method according to claim 1 or 4, characterized in that, In step S32, the specific method for expanding the <text-image> pairs in the seed set into <text-image positive sample-image negative sample> triples includes: Suppose we have a seed dataset that has been augmented and expanded. : <Text-Image Positive Samples> is the seed data set. Aligned text-image pairs; <Image negative samples> are obtained from the seed data set. Randomly selected from any other image that does not align with the text in the current text-image pair; Finally, we obtain a triplet data set of <text-image positive sample-image negative sample> for each <text>.

6. The alignment method according to claim 1, characterized in that, Step S4 specifically includes: SSS 1. A collection of multimodal text data. text in Padding with zeros or truncating to a length of max_ len ; SSS 2. Enter the text and Obtain text feature vectors ; SSS 3. Assemble multimodal image data sets Images in Adjust the size to ResNet-200 Network input size input*input ; SSS 4. Input the resized image and Obtain image feature vectors ; SSS 5. Traverse the multimodal text data set text in , i=1,...,P and multimodal image data sets Images in , j=1,...,Q Obtain the corresponding set of text feature vectors. and image feature vector set ; SSS 6. Calculation middle text and middle The cosine similarity of each pair of images is used to obtain the similarity set. ; SSS 7. Extract the similarity set Similarity greater than the confidence threshold The text-image pairs corresponding to the numerical values ​​constitute ,in For all similarities greater than the confidence threshold <text-image> pairs; SSS 8. Merge into the seed set formed by the aligned multimodal <text-image> pairs updated in the previous iteration. If new seed data is added, step S3 is executed first to iteratively train the model, and then... SSS 1- SSS 8. Update the iterative data; otherwise, end the iteration and obtain the final text-image alignment result and the network parameters of the multimodal data alignment model.

Images