A multimodal concept knowledge alignment method for unpaired image-text matching

Abstract

The application provides a multimodal concept knowledge alignment method for unpaired image text matching, comprising: collecting image text groups from a public data set, and obtaining pre-trained common sense based on prototype region representation corresponding to a word related image region in the image text group; based on bidirectional region word cycle consistent learning and unpaired image and text, refining the pre-trained common sense through a self-supervised learning manner to obtain fine-tuned domain knowledge; based on the fine-tuned domain knowledge, matching the unpaired image and text, and representing the parsed words in the text through the prototype region representation to determine the region word similarity scores corresponding to each word and each image region representation; based on a bidirectional similarity pooling module, aggregating the region word similarity scores corresponding to each word and each image region representation in the image text group into each image text global similarity score, thereby significantly improving the performance in zero sample and cross data set image text matching.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of knowledge alignment, and particularly relates to a multi-modal concept knowledge alignment method for unpaired image-text matching. BACKGROUND

[0002] Image-text matching is one of the most representative technologies in the field of visual and language understanding, and has a wide range of applications in online shopping, video surveillance, human-computer interaction, automatic driving and other fields.

[0003] The existing knowledge-based image-text matching method is always designed to be combined with paired data in the model training process, and cannot well handle unpaired image-text matching. Another method is multi-modal knowledge graph, most of which is to link semantically related images to corresponding words and then extend from existing language knowledge graphs.

[0004] However, the image-word alignment still has the following problems: 1) images usually contain redundant backgrounds or irrelevant content to words, and these disordered contents will affect the cross-modal alignment effect; 2) for each word, its related objects (or attributes) usually have different appearances in different images, resulting in this one-to-many alignment is not robust, and it is easy to cause confusion in practice. Therefore, how to obtain a better multi-modal knowledge alignment method is very challenging. SUMMARY

[0005] The present application provides a multi-modal concept knowledge alignment method for unpaired image-text matching, to solve the defects that images usually contain redundant backgrounds or irrelevant content to words, and these disordered contents will affect the cross-modal alignment effect, and for each word, its related objects usually have different appearances in different images, resulting in this one-to-many alignment is not robust, and it is easy to cause confusion in practice.

[0006] The present application provides a multi-modal concept knowledge alignment method for unpaired image-text matching, comprising:

[0007] Collecting image-text groups from a public data set, and obtaining pre-trained common sense based on the prototype region representation corresponding to the word-related image region in the image-text group; the image-text group comprises paired words and their related image regions;

[0008] Based on bidirectional region-word cycle consistency learning and unpaired images and texts, the pre-trained common sense is refined through a self-supervised learning manner to obtain fine-tuned domain knowledge;

[0009] Based on the fine-tuned domain knowledge, the unmatched image and text are matched, and the parsed words in the text are represented by the prototype region representation, and the region word similarity scores corresponding to each word and each image region representation in the unmatched image and text are determined;

[0010] Based on the bidirectional similarity pooling module, the region word similarity scores corresponding to each word and each image region representation are aggregated into global image text similarity scores.

[0011] According to the multi-modal concept knowledge alignment method for unpaired image-text matching provided by the application, the fine-tuned domain knowledge is obtained, which comprises:

[0012] Obtaining a word-image prototype region pair;

[0013] The word-image prototype region pair is input into the bidirectional region word cyclic consistency learning model, and the fine-tuned domain knowledge is obtained and output by the bidirectional region word cyclic consistency learning model;

[0014] The bidirectional region word cyclic consistency learning model is obtained based on self-supervised training of the sample word-image prototype region pair; the sample word-image prototype region pair comprises a sample word representation and a related image prototype region representation of the sample word representation.

[0015] According to the multi-modal concept knowledge alignment method for unpaired image-text matching provided by the application, the training step of the bidirectional region word cyclic consistency learning model comprises:

[0016] An initial bidirectional region word cyclic consistency learning model is determined, and the sample word-image prototype region pair is obtained;

[0017] The sample word-image prototype region pair is input into the initial bidirectional region word cyclic consistency learning model, and each word representation in the sample word-image prototype region pair is reconstructed into a reconstructed image region representation by the initial bidirectional region word cyclic consistency learning model, and each image prototype region representation in the sample word-image prototype region pair is reconstructed into a reconstructed word representation;

[0018] Based on the word representation and the reconstructed image region representation, and the image prototype region representation and the reconstructed word representation, the initial bidirectional region word cyclic consistency learning model is iteratively parameterized to obtain the bidirectional region word cyclic consistency learning model.

[0019] The application provides a multi-modal concept knowledge alignment method for unpaired image-text matching.

[0020] The first reconstruction loss is determined based on the difference between the word representation and the reconstructed image region representation.

[0021] The second reconstruction loss is determined based on the difference between the image prototype region representation and the reconstructed word representation.

[0022] The initial bidirectional region-word cycle-consistent learning model is iterated based on the first reconstruction loss and the second reconstruction loss, and the bidirectional region-word cycle-consistent learning model is obtained.

[0023] The application provides a multi-modal concept knowledge alignment method for unpaired image-text matching.

[0024] The sample word-image region pair is obtained.

[0025] The relevant image region representation of each sample word in the sample word-image region pair is averaged to obtain the sample word-image prototype region pair.

[0026] The application provides a multi-modal concept knowledge alignment method for unpaired image-text matching.

[0027] The image-text global similarity scores are sorted in descending order to obtain a first similarity vector corresponding to the first pre-set number of images in the descending order sorting result.

[0028] The first pre-set number of images are rearranged under the guidance of the fine-tuned domain knowledge to obtain a second similarity vector corresponding to the rearranged images.

[0029] The images in the image-text group are re-ranked based on the first similarity vector and the second similarity vector to obtain an image-text matching result.

[0030] The application provides a multi-modal concept knowledge alignment method for unpaired image-text matching.

[0031] In the case that the aggregation direction is image to text, maximum pooling is performed on each column in the first similarity matrix, and mean pooling is performed on the maximum-pooled first similarity matrix to obtain first region-word similarity scores;

[0032] In the case that the aggregation direction is text to image, maximum pooling is performed on each row in the second similarity matrix, and mean pooling is performed on the maximum-pooled second similarity matrix to obtain second region-word similarity scores;

[0033] The first region-word similarity scores and the second region-word similarity scores are used to determine the global image-text similarity scores.

[0034] The application also provides a multi-modal concept knowledge alignment device for unpaired image-text matching.

[0035] An acquisition unit is configured to collect an image-text group from a public data set and acquire pre-trained common sense based on a prototype region representation corresponding to a word and a related image region in the image-text group; the image-text group comprises a pair of words and related image regions;

[0036] A bidirectional region-word cyclic consistency learning unit is configured to refine the pre-trained common sense through a self-supervised learning manner based on bidirectional region-word cyclic consistency learning and unpaired images and texts, and acquire fine-tuned domain knowledge;

[0037] A matching unit is configured to match the unpaired images and texts based on the fine-tuned domain knowledge, represent a resolved word in the text through the prototype region representation, and determine region-word similarity scores corresponding to each word and each image region representation in the unpaired images and texts;

[0038] An aggregation unit is configured to aggregate the region-word similarity scores corresponding to each word and each image region representation into global image-text similarity scores based on a bidirectional similarity pooling module.

[0039] The application also provides an electronic device comprising a memory, a processor, and a computer program stored in the memory and capable of running on the processor; when the processor executes the program, the method for multi-modal concept knowledge alignment for unpaired image-text matching is realized.

[0040] The application further provides a non-transitory computer-readable storage medium, which stores a computer program, and the computer program is executed by a processor to implement the multi-modal concept knowledge alignment method for unpaired image-text matching according to any one of the above.

[0041] The application further provides a computer program product, which comprises a computer program, and the computer program is executed by a processor to implement the multi-modal concept knowledge alignment method for unpaired image-text matching according to any one of the above.

[0042] The multi-modal concept knowledge alignment method for unpaired image-text matching provided by the application is based on bidirectional region word cycle consistency learning, and the pre-trained common sense is refined through a self-supervised learning manner based on unpaired images and texts to obtain fine-tuned domain knowledge; based on the fine-tuned domain knowledge, the unpaired images and texts are matched, and the parsed words in the text are represented through a prototype region representation to determine the region word similarity scores of each word in the unpaired images and texts and each image region representation; based on a bidirectional similarity pooling module, the region word similarity scores are aggregated into global similarity scores of each image-text, and the global similarity scores of each image-text can be directly used for matching of the unpaired image-text, and can be well combined with an existing image-text matching model, thereby significantly improving the performance in zero-shot and cross-dataset image-text matching. BRIEF DESCRIPTION OF DRAWINGS

[0043] In order to more clearly illustrate the technical solutions of the present application or the prior art, the following will briefly introduce the drawings needed in the embodiments or prior art description. Obviously, the drawings in the following description are some embodiments of the present application, and those skilled in the art can obtain other drawings according to these drawings without creative labor.

[0044] Figure 1 is one of the flowcharts of the multi-modal concept knowledge alignment method for unpaired image-text matching provided by the application;

[0045] Figure 2 is another flowchart of the multi-modal concept knowledge alignment method for unpaired image-text matching provided by the application;

[0046] Figure 3 is a third flowchart of the multi-modal concept knowledge alignment method for unpaired image-text matching provided by the application;

[0047] Figure 4 is a structural schematic diagram of the multi-modal concept knowledge alignment device for unpaired image-text matching provided by the application;

[0048] Figure 5 is a structural schematic diagram of an electronic device provided by the present application. DETAILED DESCRIPTION

[0049] To make the objectives, technical solutions and advantages of the present application clearer, the technical solutions in the present application will be described clearly and completely below in conjunction with the drawings in the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative labor fall within the scope of protection of the present application.

[0050] In the related art, the main challenge of image-text matching is how to accurately measure the cross-modal similarity between images and texts. Recently, various multi-modal pre-training models attempt to solve the problem in a supervised learning manner by training on a super large (hundreds of millions or billions) of paired images and texts. Although the accuracy of image-text matching is greatly improved, it is both time-consuming and costly to collect and label such a large amount of data in real-world applications.

[0051] Single-modal visual or single-modal language knowledge has been widely used in visual and language understanding, such as image annotation, visual reasoning, and visual question answering, etc. By combining the two types of knowledge of vision and language together, multi-modal knowledge shows more complementary advantages.

[0052] However, the image-word alignment still has the following problems: 1) the image usually contains redundant background or content irrelevant to the word, which is chaotic and will affect the cross-modal alignment effect; 2) for each word, its related object (or attribute) usually has different appearances in different images, resulting in this one-to-many alignment is not robust, and in practice it is easy to cause confusion. Therefore, how to obtain a better multi-modal knowledge alignment method is very challenging.

[0053] Based on the above problems, the present application provides a multi-modal concept knowledge alignment method for unpaired image-text matching, Figure 1 is one of the flowcharts of the multi-modal concept knowledge alignment method for unpaired image-text matching provided by the present application, as shown in the figure, the method comprises: Figure 1

[0054] Step 110, collect image-text groups from a public data set, and obtain pre-trained common sense based on the prototype region representation corresponding to the word-related image region in the image-text group; the image-text group comprises a pair of words and their related image regions.

[0055] ​Specifically, the image-text pairs can be collected from a public dataset, and a pre-trained common sense can be obtained based on prototype region representations of word-related image regions in the image-text pairs, where the pre-trained common sense can be defined as a set of semantic concepts with paired multi-modal representations {(w k ,v k )} k＝1,…,k , where w k ∈{0,1} K and are one-hot and real-valued representations of region representations of the k-th semantic concept, respectively, and K is the total number of semantic concepts. The prototype region representation herein can reflect the feature information of the word-related image region.

[0056] The public dataset herein can be a Flickr30k dataset, an MSCOCO dataset, or the like, and the embodiments of the present application do not make specific limitations thereto.

[0057] The image-text pairs herein include pairs of words and their related image regions, where the images can be pre-acquired by an image acquisition device, or can be obtained by real-time shooting. The text can be obtained by voice transcription of the acquired audio, or can be obtained by scanning, mobile phone, camera, tablet, and the like image acquisition device, and performing OCR (Optical Character Recognition) on the image, and the embodiments of the present application do not make specific limitations thereto.

[0058] In step 120, the pre-trained common sense is refined by self-supervised learning based on the bidirectional region-word cyclic consistent learning and the unpaired image and text, and a fine-tuned domain knowledge is obtained.

[0059] Specifically, in order to make the obtained pre-trained common sense better adapt to some datasets, the pre-trained common sense is obtained by self-supervised training based on the bidirectional region-word cyclic consistent learning model and the sample word-image prototype region pair, that is, the initial bidirectional region-word cyclic consistent learning model performs bidirectional region-word consistent cyclic learning based on the sample word-image prototype region pair, so as to obtain better fine-tuned domain knowledge for subsequent region-word similarity score. The fine-tuned domain knowledge herein refers to prior knowledge for image and text matching.

[0060] The bidirectional region-word cyclic consistent learning herein refers to the learning of word representation to image region representation, and the learning of image region representation to word representation.

[0061] At step 130, based on the fine-tuned domain knowledge, the unmatched image and text are matched, and the resolved words in the text are represented by the prototype region representation, and the region word similarity scores of each word in the unmatched image and text corresponding to each image region representation are determined.

[0062] Specifically, after obtaining the fine-tuned domain knowledge, the unmatched image and text can be matched, and the resolved words in the text are represented by the prototype region representation, and the region word similarity scores of each word in the unmatched image and text corresponding to each image region representation are determined. The region word similarity scores here reflect the matching of each word and each image region.

[0063] That is, for the words in the text, first use the Natural Language Toolkit (NLTK) to predict the word type and parse the word dependency relationship. Then, use the fine-tuned domain knowledge to represent all the words as a set of prototype region representations. Finally, take each word in the text as an anchor point, and combine the word representation with the related adjective representation (if any) in an average manner. The resulting word representation is represented as {u l} l＝1,…,L or And for the image, use the region-based convolutional network method (Faster-RCNN) to obtain a set of detected image region representations {q i} i＝1,…,I or Therefore, the region word similarity scores in the similarity matrix S corresponding to each text and each image region representation in the image-text group are obtained:

[0064] S=U T Q

[0065] At step 140, based on the bidirectional similarity pooling module, the region word similarity scores corresponding to each word and each image region representation are aggregated into image-text global similarity scores.

[0066] Specifically, considering that the image has multiple regions, the region word similarity scores corresponding to each word and each image region representation can be aggregated into image-text global similarity scores. The image-text global similarity scores here reflect the matching of the image and the text at the global level.

[0067] Here, the bidirectional similarity pooling module can be used to aggregate the region word similarity scores into image-text global similarity scores.

[0068] For example, the bidirectional similarity pooling module can perform maximum pooling on the rows or columns of the similarity matrix S according to the aggregation direction, and then perform average pooling on the maximum pooled similarity matrix to obtain the global image-text similarity score.

[0069] The method provided by the embodiment of the application is based on bidirectional region-word cycle consistency learning, and the pre-trained common sense is refined through self-supervised learning based on unpaired images and texts to obtain fine-tuned domain knowledge; based on the fine-tuned domain knowledge, the unpaired images and texts are matched, and the parsed words in the text are represented by a prototype region representation, and the region-word similarity scores of each word in the unpaired images and texts and each image region representation are determined; based on the bidirectional similarity pooling module, the region-word similarity scores are aggregated into global image-text similarity scores, which can be directly used for matching of unpaired images and texts, and can be well combined with existing image-text matching models, thereby significantly improving the performance in zero-shot and cross-dataset image-text matching.

[0070] Based on the above embodiment, the fine-tuned domain knowledge obtained in step 120 includes:

[0071] In step 121, a word-image prototype region pair is obtained.

[0072] In step 122, the word-image prototype region pair is input into the bidirectional region-word cycle consistency learning model, and the fine-tuned domain knowledge is obtained and output by the bidirectional region-word cycle consistency learning model.

[0073] The bidirectional region-word cycle consistency learning model is obtained through self-supervised training based on the sample word-image prototype region pair, and the sample word-image prototype region pair includes a sample word representation and a related image prototype region representation of the sample word representation.

[0074] Specifically, in order to make the obtained pre-trained common sense better adapt to some datasets, the pre-trained common sense is refined through self-supervised learning based on unpaired images and texts to obtain fine-tuned domain knowledge.

[0075] The bidirectional region-word cycle consistency learning model is obtained through self-supervised training based on the sample word-image prototype region pair, and the sample word-image prototype region pair includes a sample word representation and a related image prototype region representation of the sample word representation.

[0076] After self-supervised training, the learned parameter transformation matrix W can be used to convert all related image prototype region representations into fine-tuned domain knowledge, represented as {(w k ,u k )k＝1,…,K ,in

[0077] For each word, different image regions typically have different visual appearances. To avoid the influence of appearance variations, this is achieved by representing all relevant image regions... Calculate the average for each word w k Related image prototype region representation v k :

[0078]

[0079] Each relevant image prototype region represents r. j This is achieved by inputting bounding boxes and images into a pre-trained object detection model—a region-based convolutional network method (Faster-RCNN). The number of related image prototype regions for different words {J} k} k＝1,…,K The results are very unbalanced, ranging from tens to tens of thousands. For words with low frequency of occurrence, an initial prototype region representation is calculated using a very limited number of regions (e.g., less than 10), and then refined as more relevant regions are encountered.

[0080] In practical applications, word image prototype region pairs can be obtained, which include word representations and related image prototype region representations of the word representations.

[0081] Then, the word image prototype region pairs can be input into the bidirectional region word cycle consistency learning model, which will then obtain and output fine-tuned domain knowledge.

[0082] Based on the above embodiments, the training steps of the bidirectional region word cycle consistency learning model include:

[0083] Step 210: Determine the initial bidirectional region word cycle consistency learning model and obtain the prototype region pairs of the sample word images;

[0084] Step 220: Input the sample word image prototype region pair into the initial bidirectional region word cyclic consistency learning model, and the initial bidirectional region word cyclic consistency learning model reconstructs each text representation in the sample word image prototype region pair into each reconstructed image region representation, and reconstructs each image prototype region representation in the sample word image prototype region pair into each reconstructed word representation;

[0085] Step 230: Based on the word representations and the reconstructed image region representations, as well as the image prototype region representations and the reconstructed word representations, perform parameter iteration on the initial bidirectional region word cyclic consistency learning model to obtain the bidirectional region word cyclic consistency learning model.

[0086] Specifically, the bidirectional region-word cycle consistent learning model can be obtained by the following steps:

[0087] The sample word-image prototype region pairs can be collected in advance, and the initial bidirectional region-word cycle consistent learning model can also be constructed in advance. The initial bidirectional region-word cycle consistent learning model can functionally include two parts of text-image prototype region self-supervised training and converting all related image prototype region representations into fine-tuned domain knowledge. Subsequently, the initial bidirectional region-word cycle consistent learning model can be trained based on the sample word-image prototype region pairs, and the initial bidirectional region-word cycle consistent learning model after training can be used as the bidirectional region-word cycle consistent learning model.

[0088] Firstly, the sample word-image prototype region pairs can be input into the initial bidirectional region-word cycle consistent learning model, and each word representation in the sample word-image prototype region pairs can be reconstructed into each reconstructed image region representation by the initial bidirectional region-word cycle consistent learning model, and each image prototype region representation in the sample word-image prototype region pairs can be reconstructed into each reconstructed word representation.

[0089] The sample word-image prototype region pairs can be a set of detected regions and a set of recognized words obtained by using the above-mentioned region-based convolutional network method (Faster-RCNN). For the word set, the repeated words are deleted, and the remaining words are denoted as {w m} m＝1,…,M Then, all the words can be represented by corresponding prototype region representations (in pre-trained common sense) as {v m} m＝1,…,M or For the region set, the similarity scores of the regions and the prototype region representations are calculated, and then the redundant regions are deleted in a manner similar to non-maximum suppression, and the generated region representations are {r n} n＝1,…,N or

[0090] That is, a bidirectional region-word cycle consistent loss can be designed to learn the parameter transformation matrix It contains a bidirectional process of cross-modal similarity measurement: region to word (R2W) and word to region (W2R). For R2W, the main idea is to first measure the similarity between each word and all regions, and then use the similarity as a weight to combine all regions into a reconstructed word. Similarly, in the reverse W2R, this process is used to combine all words into a reconstructed region, and the corresponding formula of the bidirectional process is:

[0091]

[0092] wherein, is a similarity matrix between transformed word and region representations, and σ(·) is a normalization exponential function (Softmax) operation along the column dimension, comprises reconstructed word representations obtained from the region representations, comprises reconstructed region representations obtained from the word representations.

[0093] Then, the initial bidirectional region-word cycle-consistent learning model can be iterated in parameters based on the respective word representations and the respective reconstructed image region representations, and the respective image prototype region representations and the respective reconstructed word representations, and the initial bidirectional region-word cycle-consistent learning model after the parameter iteration is completed can be taken as the bidirectional region-word cycle-consistent learning model.

[0094] Based on the above embodiment, step 230 comprises:

[0095] Step 231, determining a first reconstruction loss based on the difference between the respective word representations and the respective reconstructed image region representations;

[0096] Step 232, determining a second reconstruction loss based on the difference between the respective image prototype region representations and the respective reconstructed word representations;

[0097] Step 233, iterated in parameters of the initial bidirectional region-word cycle-consistent learning model based on the first reconstruction loss and the second reconstruction loss, to obtain the bidirectional region-word cycle-consistent learning model.

[0098] Specifically, the first reconstruction loss can be determined based on the difference between the respective word representations and the respective reconstructed image region representations.

[0099] It can be understood that the greater the difference between the respective word representations and the respective reconstructed image region representations, the greater the first reconstruction loss; the smaller the difference between the respective word representations and the respective reconstructed image region representations, the smaller the first reconstruction loss.

[0100] The second reconstruction loss can be determined based on the difference between the respective image prototype region representations and the respective reconstructed word representations.

[0101] It can be understood that the greater the difference between the respective image prototype region representations and the respective reconstructed word representations, the greater the second reconstruction loss; the smaller the difference between the respective image prototype region representations and the respective reconstructed word representations, the smaller the second reconstruction loss.

[0102] After obtaining the first reconstruction loss and the second reconstruction loss, a self-supervised loss can be determined based on the first reconstruction loss and the second reconstruction loss, and then the initial bidirectional region-word cycle-consistent learning model can be iterated in parameters based on the self-supervised loss, to obtain the bidirectional region-word cycle-consistent learning model.

[0103] That is, the representation of each original word (or region) is compared with its reconstructed representation to generate a label indicating whether they are the same. By minimizing the cross-entropy between the predicted label and the true label, a self-supervised loss L is obtained for optimizing W:

[0104]

[0105]

[0106]

[0107] L=L R2W +L W2R

[0108] where L W2R represents the first reconstruction loss, L R2W represents the second reconstruction loss, and are two matrices containing the predicted labels in the two directions of R2W and W2R, respectively, in and , the mth column and the nth column are represented as and y m and y n are two true label vectors, whose mth and nth values are 1, and the rest are 0. After the loop-consistent learning, the learned W is used to convert all prototype region representations into fine-tuned domain knowledge, which is represented as {(w k ,u k )} k＝1,…,K , where

[0109] Based on the above embodiment, the step of obtaining the sample word image prototype region pair includes:

[0110] Step 310, obtaining a sample word image region pair; the sample text image region pair includes a sample word and a relevant image region representation of the sample word;

[0111] Step 320, averaging the relevant image region representation of each sample word in the sample word image region pair to obtain the sample word image prototype region pair.

[0112] Specifically, a sample word image region pair can be obtained, where the sample word image region pair includes a sample word and a relevant image region representation of the sample word. To remove the redundant content in the image region, here, the relevant image region representation only focuses on the semantic concept in the local image region.

[0113] To avoid the impact of appearance changes, all relevant image regions are represented. Calculate the average for each word w k Related image prototype region representation v k :

[0114]

[0115] Each relevant image prototype region represents r. j This is achieved by inputting bounding boxes and images into a pre-trained object detection model—a region-based convolutional network method (Faster-RCNN). The number of related image prototype regions for different words {J} k} k＝1,…,K The results are very unbalanced, ranging from tens to tens of thousands. For words with low frequency of occurrence, an initial prototype region representation is calculated using a very limited number of regions (e.g., less than 10), and then refined as more relevant regions are encountered.

[0116] Based on the above embodiments, step 140 further includes:

[0117] Step 410: Sort the global similarity scores of each image text in descending order to obtain the first similarity vector corresponding to the first preset number of images in the descending sort result;

[0118] Step 420: Under the guidance of the fine-tuned domain knowledge, rearrange the first preset number of images to obtain the second similarity vector corresponding to the rearranged images;

[0119] Step 430: Based on the first similarity vector and the second similarity vector, re-rank the images in the image-text group to obtain the image-text matching result.

[0120] Specifically, the global similarity scores of each image and text can be sorted in descending order to obtain the first similarity vector corresponding to the first preset number of images in the descending sort result. The first similarity vector can be used... This indicates that the preset number of items here can be top-k.

[0121] Then, guided by fine-tuned domain knowledge, the first preset number of images are rearranged to obtain the second similarity vector corresponding to the rearranged images. This second similarity vector can be used... This indicates that the preset number of items here can be top-k.

[0122] Finally, the images in the image-text group can be re-ranked based on the first and second similarity vectors to obtain the image-text matching results, as shown in the following formula:

[0123]

[0124] wherein s k denotes the first similarity vector, g k denotes the second similarity vector, and λ denotes a balancing factor, denotes the final image-text matching result, which is used to re-rank the top k images to improve the matching performance.

[0125] It can be understood that, under the guidance of fine-tuned domain knowledge, the top pre-set number of images are rearranged to obtain the second similarity vector corresponding to the rearranged images, and then the images in the image-text group are re-ranked based on the first similarity vector and the second similarity vector to obtain the image-text matching result. The image-text matching result can be directly used for matching of unpaired image-texts, and can be well combined with existing image-text matching models, thereby significantly improving the performance in zero-shot and cross-dataset image-text matching.

[0126] Based on the above embodiment, step 140 comprises:

[0127] Step 141, in the case of the aggregation direction being image to text, performing max-pooling on each column in the first similarity matrix, and performing mean-pooling on the first similarity matrix after max-pooling to obtain a first regional word similarity score;

[0128] Step 142, in the case of the aggregation direction being text to image, performing max-pooling on each row in the second similarity matrix, and performing mean-pooling on the second similarity matrix after max-pooling to obtain a second regional word similarity score;

[0129] Step 143, determining the global similarity score of each image-text based on the first regional word similarity score and the second regional word similarity score.

[0130] Specifically, the regional word similarity scores in the similarity matrix S can be aggregated into the global similarity score of each image-text by a bidirectional similarity pooling module, so that the method better adapts to two sub-tasks of unpaired image-text matching: 1) image annotation: retrieving a text library given an image query, the direction of which is image to text (I2T), and 2) image retrieval: retrieving an image library given a text query, the direction of which is text to image (T2I).

[0131] I2T is to find the matching words from the candidate texts of the given image query region, in order to select the most relevant words, in the aggregation direction of image to text, max-pooling is performed on each column in the first similarity matrix, and mean-pooling is performed on the max-pooled first similarity matrix to obtain the first region-word similarity score.

[0132] That is, first, max-pooling is performed on the i-th column in the similarity matrix S, that is, the region-word similarity score between the i-th region and all words. By jointly considering other regions in the image, then all similarity scores after max-pooling are combined using mean-pooling to obtain the first region-word similarity score.

[0133] T2I is to find the matching regions from the candidate images of the given text query word, in the aggregation direction of text to image, max-pooling is performed on each row in the second similarity matrix, and mean-pooling is performed on the max-pooled second similarity matrix to obtain the second region-word similarity score. So first, max-pooling is performed on the rows of the similarity matrix S, and then mean-pooling is performed to obtain the second region-word similarity score.

[0134] Wherein, the formulas corresponding to I2T and T2I are:

[0135] g I2T = p mean (p max (s T )), g T2I = p mean (p max (S))

[0136] Wherein, p max (·) and p mean (·) are max-pooling and mean-pooling operations on the last dimension of the input matrix or vector, respectively. g I2T and g T2I are the first region-word similarity scores and the second region-word similarity scores required by I2T and T2I, respectively. In the case of multiple images and texts being given, their similarity scores can be similarly obtained for unpaired image-text matching.

[0137] Finally, based on the first region-word similarity scores and the second region-word similarity scores, the global image-text similarity scores of each image and text are determined.

[0138] In addition, the effectiveness of the method proposed in the present application is verified by experimental results. The experiment is carried out on the following two standard public data sets: 1) Flickr30k data set, containing 31783 images collected from Flickr website, each image has 5 manually annotated texts, the experiment uses the public training, validation and test split method, containing 29000, 1014 and 1000 images respectively. 2) MSCOCO data set, consisting of 123287 images, each image is associated with 5 texts, the experiment uses the public training, validation and test split method, containing 113287, 5000 and 5000 images respectively.

[0139] Image-text matching generally includes two sub-tasks: 1) image annotation: given an image, retrieve relevant texts; 2) image retrieval: given a text, retrieve relevant images. Here, given a text, retrieve relevant images can use a CLIP (Contrastive Language-Image Pre-training) model.

[0140] The commonly used evaluation criteria are "R@1", "R@5" and "R@10", that is, the recall rates of top-1, 5 and 10 results, and the experiment also evaluates the overall performance by summing all the recall rates, thereby using the additional criterion Rs:

[0141] R s = R@1 + R@5 + R@10 + R@1 + R@5 + R@10

[0142] Wherein, R@1+R@5+R@10 belongs to Image annotation, R@1+R@5+R@10 belongs to Image retrieval, and the experiment compares several ablation models of the model proposed in the present application to verify their effectiveness for unpaired image-text matching. The first representative model is a pre-trained common sense, which contains paired prototype region representations and semantically related words. On this basis, the experiment further designs three model variants of it, including:

[0143] (1) With or without prototype region representation: use randomly selected region representations to replace each prototype region representation, similar to existing multi-modal knowledge graph.

[0144] (2) With or without adjectives: do not combine adjective representations with related noun representations, so that region and word representations may have different levels of semantic information.

[0145] (3) With or without max-mean pooling: aggregate region-word similarity scores into image-text similarity scores, replace bidirectional max-mean pooling with global mean pooling.

[0146] Another representative model, which refines the field knowledge, uses a bidirectional region-word cycle-consistency loss based on unpaired images and texts to fine-tune the pre-trained general knowledge. Experiments further design four model variants of it, including:

[0147] (1) With / without paired prior knowledge: Instead of using images and texts from the same Flickr30k (or MSCOCO) dataset, images from Flickr30k and texts from MSCOCO are used to eliminate the implicit paired prior knowledge.

[0148] (2) With / without bidirection: Only use the unidirectional (i.e., R2W direction) region-word cycle-consistency loss to fine-tune the knowledge.

[0149] (3) With / without linear transformation: Extend the single-layer linear transformation to two-layer nonlinear transformation with rectified linear unit (ReLU) activation function.

[0150] (4) With / without knowledge: Directly learn the word representation from one-hot without using knowledge to convert them to the corresponding prototype region representation.

[0151] Experiments show that for the pre-trained general knowledge, the performance will be worse when removing the region prototype, adjective and max average pooling. Among them, the prototype region representation contributes the most to the performance, i.e., 6.7%~9.3% and 13.9%~15.1% in R@1 on two datasets, respectively.

[0152] By learning unpaired images and texts, the fine-tuned field knowledge shows further performance improvement than the pre-trained general knowledge, especially on the subtask of image retrieval. When deleting the implicit prior in unpaired images and texts, the performance decreases slightly on the Flickr30k dataset and improves on the MSCOCO dataset, which shows that the implicit prior knowledge does not always have a great impact on the performance. When only using the unidirectional loss, the overall performance becomes worse, i.e., -3.9% and -9.6% in Rs on two datasets, which proves the effectiveness of the above bidirectional method. By learning nonlinear transformation, the performance does not show consistent improvement on two datasets. However, if the knowledge is not used as a cross-modal bridge to represent the word representation as the prototype region representation, the performance will be poor.

[0153] Based on any of the above embodiments, Figure 2 is a flowchart of a second embodiment of a multi-modal concept knowledge alignment method for unpaired image-text matching provided by the present application, as shown in Figure 2 , the method comprises:

[0154] Step 1, obtain the word-related region.

[0155] The second step is to calculate the prototype region representation of the word-related region, thereby obtaining the pre-trained common sense.

[0156] The third step involves refining the pre-trained common sense through self-supervised learning, based on bidirectional regional word cycle consistency learning and unpaired images and text, to obtain fine-tuned domain knowledge.

[0157] The fourth step is to obtain the similarity score of the words in the given image and words.

[0158] The fifth step involves aggregating the similarity scores of each word with the corresponding region word in each image region using the bidirectional similarity pooling module to obtain a global similarity score for each image text.

[0159] Based on any of the above embodiments Figure 3 This is the third flowchart of the multimodal concept knowledge alignment method for unpaired image-text matching provided by the present invention, as shown below. Figure 3 As shown, the method includes:

[0160] The first step is to obtain the image text group. The text in the image text group here could be "A young couple with a kneeling vendor reading a book of manage".

[0161] The second step involves using each noun in the text as an anchor point and combining its representation with the representation of the relevant adjective (if present) in an average manner. The resulting word representation is denoted as {u l} l＝1,…,L or For images, a region-based convolutional network method is used to obtain a set of detected region representations {q}. i} i＝1,…,I or

[0162] For example, the phrase "A young couple with a kneeling vendor reading a book of manage" can be represented as:

[0163] “young—couple—outdoors

[0164] kneeling—vendor—reading—book—manage”

[0165] Therefore, the similarity matrix S is obtained:

[0166] S = U T Q

[0167] The third step is to use the fine-tuned domain knowledge as a cross-model bridge. Based on the fine-tuned domain knowledge, unpaired images and texts are matched, and the parsed word concept knowledge in the text is represented by the prototype region representation. The similarity score between each word in the unpaired image and text image text group and the corresponding region word in each image region representation is determined.

[0168] The fourth step is to find matching words from the candidate text of the given image query region. In order to select the most relevant words, with the aggregation direction being from image to text, max pooling is performed on each column of the first similarity matrix, and mean pooling is performed on the first similarity matrix after max pooling to obtain the word similarity score of the first region.

[0169] T2I finds matching regions in candidate images of a given text query word. With the aggregation direction from text to image, it performs max pooling on each row of the second similarity matrix, and then performs mean pooling on the max-pooled second similarity matrix to obtain the word similarity score for the second region. Therefore, max pooling is first performed on the rows of the similarity matrix S, followed by mean pooling to obtain the word similarity score for the second region.

[0170] The fifth step is to determine the global similarity score of each image text based on the word similarity scores of the first and second regions.

[0171] The following describes the multimodal concept knowledge alignment device for unpaired image text matching provided by the present invention. The multimodal concept knowledge alignment device for unpaired image text matching described below can be referred to in correspondence with the multimodal concept knowledge alignment method for unpaired image text matching described above.

[0172] Based on any of the above embodiments, the present invention provides a multimodal concept knowledge alignment device for unpaired image-text matching. Figure 4 This is a schematic diagram of the structure of the multimodal concept knowledge alignment device for unpaired image-text matching provided by the present invention, as shown below. Figure 4 As shown, the device includes:

[0173] The acquisition unit 410 is used to collect image-text groups from a public dataset and acquire pre-trained common sense based on the prototype region representations corresponding to the image regions related to words in the image-text groups; the image-text groups include pairs of words and their related image regions;

[0174] The bidirectional region word cycle consistency learning unit 420 is used to refine the pre-trained common sense through self-supervised learning based on bidirectional region word cycle consistency learning and unpaired images and text, and to obtain fine-tuned domain knowledge.

[0175] The matching unit 430 is used to match the unpaired image and text based on the fine-tuned domain knowledge, and to represent the parsed words in the text through the prototype region representation, and to determine the region word similarity score between each word in the unpaired image and text and the corresponding region word representation of each image.

[0176] Aggregation unit 440 is used to aggregate the similarity scores of each word and the corresponding region word of each image region representation into a global similarity score of each image text based on the bidirectional similarity pooling module.

[0177] The apparatus provided in this invention, based on bidirectional region-word cyclic consistency learning and unpaired images and text, refines pre-trained common sense through self-supervised learning to obtain fine-tuned domain knowledge. Based on this fine-tuned domain knowledge, it matches unpaired images and text, and uses prototype region representations to represent parsed words in the text, determining the region-word similarity score between each word in the unpaired images and text and the corresponding image region representation. Based on a bidirectional similarity pooling module, the region-word similarity scores are aggregated into global similarity scores for each image and text. These global similarity scores can be directly used for matching unpaired images and text, and can be well integrated with existing image-text matching models, thereby significantly improving performance in zero-shot and cross-dataset image-text matching.

[0178] Based on any of the above embodiments, the acquisition unit 410 is specifically used for:

[0179] Obtain word image prototype region pairs;

[0180] The word image prototype region pair is input into the bidirectional region word cyclic consistency learning model, and the fine-tuned domain knowledge is obtained and output by the bidirectional region word cyclic consistency learning model.

[0181] The bidirectional region word cycle consistency learning model is obtained by self-supervised training based on the sample word image prototype region pair; the sample word image prototype region pair includes the sample word representation and the related image prototype region representation of the sample word representation.

[0182] Based on any of the above embodiments, the training steps of the bidirectional region word cycle consistency learning model include:

[0183] A model unit is determined to determine the initial bidirectional region word cycle consistency learning model and to obtain the prototype region pairs of the sample word images;

[0184] The reconstruction unit is used to input the sample word image prototype region pair into the initial bidirectional region word cyclic consistency learning model, and the initial bidirectional region word cyclic consistency learning model reconstructs each word representation in the sample word image prototype region pair into each reconstructed image region representation, and reconstructs each image prototype region representation in the sample word image prototype region pair into each reconstructed word representation.

[0185] The parameter iteration unit is used to perform parameter iteration on the initial bidirectional region word cyclic consistency learning model based on the word representations and the reconstructed image region representations, as well as the image prototype region representations and the reconstructed word representations, to obtain the bidirectional region word cyclic consistency learning model.

[0186] Based on any of the above embodiments, the parameter iteration unit is specifically used for:

[0187] The first reconstruction loss is determined based on the difference between each word representation and each reconstructed image region representation;

[0188] The second reconstruction loss is determined based on the difference between the image prototype region representation and the reconstructed word representation;

[0189] Based on the first reconstruction loss and the second reconstruction loss, the parameters of the initial bidirectional region word cycle consistency learning model are iterated to obtain the bidirectional region word cycle consistency learning model.

[0190] Based on any of the above embodiments, the step of obtaining the prototype region pair of the sample word image includes:

[0191] Obtain sample word image region pairs; the sample text image region pairs include sample words and related image region representations of the sample words;

[0192] The sample word image region pairs are obtained by averaging the relevant image region representations of each sample word in the sample word image region pairs.

[0193] Based on any of the above embodiments, the aggregation unit 440 is further specifically used for:

[0194] The global similarity scores of the images and texts are sorted in descending order to obtain the first similarity vectors corresponding to the first preset number of images in the descending sort result;

[0195] Guided by the fine-tuned domain knowledge, the first preset number of images are rearranged to obtain the second similarity vector corresponding to the rearranged images;

[0196] Based on the first similarity vector and the second similarity vector, the images in the image-text group are re-ranked to obtain the image-text matching result.

[0197] Based on any of the above embodiments, the aggregation unit 440 is specifically used for:

[0198] When the aggregation direction is from image to text, max pooling is performed on each column of the first similarity matrix, and mean pooling is performed on the first similarity matrix after max pooling to obtain the word similarity score of the first region.

[0199] When the aggregation direction is from text to image, max pooling is performed on each row of the second similarity matrix, and mean pooling is performed on the max pooled second similarity matrix to obtain the word similarity score of the second region.

[0200] Based on the word similarity scores of the first region and the word similarity scores of the second region, the global similarity score of each image text is determined.

[0201] Figure 5 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 5 As shown, the electronic device may include: a processor 510, a communication interface 520, a memory 530, and a communication bus 540, wherein the processor 510, the communication interface 520, and the memory 530 communicate with each other through the communication bus 540. The processor 510 can invoke logical instructions in the memory 530 to execute a multimodal concept knowledge alignment method for unpaired image-text matching. This method includes: collecting image-text sets from a public dataset and obtaining pre-trained common sense based on prototype region representations corresponding to word-related image regions in the image-text sets; the image-text sets include paired words and their associated image regions; refining the pre-trained common sense through self-supervised learning based on bidirectional region-word cyclic consistency learning and unpaired images and text to obtain fine-tuned domain knowledge; matching the unpaired images and text based on the fine-tuned domain knowledge, and representing the parsed words in the text using the prototype region representations to determine the region-word similarity scores between each word in the unpaired images and text and each corresponding image region representation; and aggregating the region-word similarity scores between each word and each corresponding image region representation into a global similarity score for each image-text based on a bidirectional similarity pooling module.

[0202] Furthermore, the logical instructions in the aforementioned memory 530 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, essentially, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0203] On the other hand, the present invention also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the multimodal concept knowledge alignment method for unpaired image-text matching provided by the above methods. The method includes: collecting image-text groups from a public dataset and obtaining pre-trained common sense based on the prototype region representations corresponding to word-related image regions in the image-text groups; the image-text groups include paired words and their related image regions; refining the pre-trained common sense through self-supervised learning based on bidirectional region word cyclic consistency learning and unpaired images and text to obtain fine-tuned domain knowledge; matching the unpaired images and text based on the fine-tuned domain knowledge, and representing the parsed words in the text through the prototype region representations to determine the region word similarity scores between each word in the unpaired images and text and each image region representation; and aggregating the region word similarity scores between each word and each image region representation into a global similarity score for each image-text based on a bidirectional similarity pooling module.

[0204] In another aspect, the present invention also provides a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements a multimodal concept knowledge alignment method for unpaired image-text matching provided by the methods described above. This method includes: collecting image-text sets from a public dataset and obtaining pre-trained common sense based on prototype region representations corresponding to word-related image regions in the image-text sets; the image-text sets include paired words and their associated image regions; refining the pre-trained common sense through self-supervised learning based on bidirectional region-word cyclic consistency learning and unpaired images and text to obtain fine-tuned domain knowledge; matching the unpaired images and text based on the fine-tuned domain knowledge, and representing the parsed words in the text using the prototype region representations to determine the region-word similarity scores between each word in the unpaired images and text and each image region representation; and aggregating the region-word similarity scores between each word and each image region representation into a global similarity score for each image-text based on a bidirectional similarity pooling module.

[0205] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0206] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0207] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A multimodal concept knowledge alignment method for unpaired image-text matching, characterized in that, include: Image-text pairs are collected from public datasets, and pre-trained common sense is obtained based on the prototype region representations corresponding to the image regions related to words in the image-text pairs; the image-text pairs include pairs of words and their related image regions. Based on bidirectional regional word cycle consistency learning, and unpaired images and text, the pre-trained common sense is refined through self-supervised learning to obtain fine-tuned domain knowledge. Based on the fine-tuned domain knowledge, the unpaired images and texts are matched, and the parsed words in the text are represented by the prototype region representation. The similarity scores between each word in the unpaired images and text and the corresponding region word representations of each image are determined. Based on the bidirectional similarity pooling module, the similarity scores between each word and the corresponding region word in each image region are aggregated into a global similarity score for each image text. The acquisition of the fine-tuned domain knowledge includes: Obtain word image prototype region pairs; The word image prototype region pair is input into the bidirectional region word cyclic consistency learning model, and the fine-tuned domain knowledge is obtained and output by the bidirectional region word cyclic consistency learning model. The bidirectional region-based word cycle consistency learning model is obtained through self-supervised training based on sample word image prototype region pairs; the sample word image prototype region pairs include sample word representations and related image prototype region representations of the sample word representations. The training steps of the bidirectional region word cycle consistency learning model include: Determine the initial bidirectional region word cycle consistency learning model and obtain the prototype region pairs of the sample word images; The sample word image prototype region pair is input into the initial bidirectional region word cyclic consistency learning model. The initial bidirectional region word cyclic consistency learning model reconstructs each word representation in the sample word image prototype region pair into each reconstructed image region representation, and reconstructs each image prototype region representation in the sample word image prototype region pair into each reconstructed word representation. Based on the word representations and reconstructed image region representations, as well as the image prototype region representations and reconstructed word representations, the parameters of the initial bidirectional region word cyclic consistency learning model are iterated to obtain the bidirectional region word cyclic consistency learning model.

2. The multimodal concept knowledge alignment method for unpaired image-text matching according to claim 1, characterized in that, The method of iterating the parameters of the initial bidirectional region word cycle consistency learning model based on the word representations and the reconstructed image region representations, as well as the image prototype region representations and the reconstructed word representations, to obtain the bidirectional region word cycle consistency learning model includes: The first reconstruction loss is determined based on the difference between each word representation and each reconstructed image region representation; The second reconstruction loss is determined based on the difference between the image prototype region representation and the reconstructed word representation; Based on the first reconstruction loss and the second reconstruction loss, the parameters of the initial bidirectional region word cycle consistency learning model are iterated to obtain the bidirectional region word cycle consistency learning model.

3. The multimodal concept knowledge alignment method for unpaired image-text matching according to claim 1, characterized in that, The steps for obtaining the prototype region pairs of the sample word images include: Obtain sample word image region pairs; the sample word image region pair includes the sample word and the related image region representation of the sample word; The sample word image region pairs are obtained by averaging the relevant image region representations of each sample word in the sample word image region pairs.

4. The multimodal concept knowledge alignment method for unpaired image-text matching according to any one of claims 1 to 3, characterized in that, The bidirectional similarity pooling module aggregates the similarity scores between each word in the image-text group and the corresponding region word representation in each image region into a global similarity score for each image-text group. This process further includes: The global similarity scores of the images and texts are sorted in descending order to obtain the first similarity vectors corresponding to the first preset number of images in the descending sort result; Guided by the fine-tuned domain knowledge, the first preset number of images are rearranged to obtain the second similarity vector corresponding to the rearranged images; Based on the first similarity vector and the second similarity vector, the images in the image-text group are re-ranked to obtain the image-text matching result.

5. The multimodal concept knowledge alignment method for unpaired image-text matching according to any one of claims 1 to 3, characterized in that, The bidirectional similarity pooling module aggregates the similarity scores between each word in the image-text group and the corresponding region word representation in each image region into a global similarity score for each image-text group, including: When the aggregation direction is from image to text, max pooling is performed on each column of the first similarity matrix, and mean pooling is performed on the first similarity matrix after max pooling to obtain the word similarity score of the first region. When the aggregation direction is from text to image, max pooling is performed on each row of the second similarity matrix, and mean pooling is performed on the max pooled second similarity matrix to obtain the word similarity score of the second region. Based on the word similarity scores of the first region and the word similarity scores of the second region, the global similarity score of each image text is determined.

6. A multimodal concept knowledge alignment device for unpaired image-text matching, characterized in that, include: An acquisition unit is used to collect image-text groups from a public dataset and acquire pre-trained common sense based on the prototype region representations corresponding to the image regions related to words in the image-text groups; the image-text groups include pairs of words and their related image regions; A bidirectional region word cycle consistency learning unit is used to refine the pre-trained common sense through self-supervised learning based on bidirectional region word cycle consistency learning, as well as unpaired images and text, to obtain fine-tuned domain knowledge. The matching unit is used to match the unpaired images and text based on the fine-tuned domain knowledge, and to represent the parsed words in the text through the prototype region representation, and to determine the region word similarity score between each word in the unpaired images and text and the corresponding region word representation of each image. An aggregation unit is used to aggregate the similarity scores of each word and the corresponding region word of each image region representation into a global similarity score for each image text based on the bidirectional similarity pooling module. The bidirectional regional word cycle consistency learning unit is specifically used for: Obtain word image prototype region pairs; The word image prototype region pair is input into the bidirectional region word cyclic consistency learning model, and the fine-tuned domain knowledge is obtained and output by the bidirectional region word cyclic consistency learning model. The bidirectional region-based word cycle consistency learning model is obtained through self-supervised training based on sample word image prototype region pairs; the sample word image prototype region pairs include sample word representations and related image prototype region representations of the sample word representations. It also includes a training unit, which is specifically used for: Determine the initial bidirectional region word cycle consistency learning model and obtain the prototype region pairs of the sample word images; The sample word image prototype region pair is input into the initial bidirectional region word cyclic consistency learning model. The initial bidirectional region word cyclic consistency learning model reconstructs each word representation in the sample word image prototype region pair into each reconstructed image region representation, and reconstructs each image prototype region representation in the sample word image prototype region pair into each reconstructed word representation. Based on the word representations and reconstructed image region representations, as well as the image prototype region representations and reconstructed word representations, the parameters of the initial bidirectional region word cyclic consistency learning model are iterated to obtain the bidirectional region word cyclic consistency learning model.

7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the multimodal concept knowledge alignment method for unpaired image text matching as described in any one of claims 1 to 5.

8. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the multimodal concept knowledge alignment method for unpaired image text matching as described in any one of claims 1 to 5.

Images