A method for image-text alignment based on mutual information

Abstract

The present invention discloses a method for image-text alignment based on mutual information. First, an image-text alignment network model including a feature extraction module and a feature alignment module is constructed. The feature extraction module includes an image feature extraction module and a text feature extraction module, which are respectively used to extract image features and text features. The feature alignment module is used to calculate alignment loss, and the alignment loss includes intra-modal loss and inter-modal loss. Then, the constructed image-text alignment network model is trained, and the image and text to be aligned are input into the trained image-text alignment network model to obtain the image-to-text sorting or the text-to-image sorting, thereby realizing the alignment of the image and the text. The present invention makes full use of the mutual information between image and text data, aligns image-text pairs that are difficult to distinguish, and improves the image-text alignment performance.

Description

Technical Field

[0001] The present application belongs to the technical field of image-text alignment, and in particular relates to an image-text alignment method based on mutual information. Background Art

[0002] With the rapid development of technology, massive amounts of multimedia data, including images, videos, text, and audio, are rapidly emerging. According to Gartner, image and video data already account for over 90% of big data. In 2018, the total amount of data created, captured, replicated, and consumed globally reached 33 zettabytes (ZB). This figure is projected to grow to 59 ZB in 2020 and reach an unimaginable 175 ZB by 2025. This data is heterogeneous in form and multi-source, semantically interconnected, and crucial to society. Therefore, semantic analysis of this multimedia data and understanding its content have become a research hotspot.

[0003] Image-text alignment enables computers to understand multimedia data in different formats at a semantic level. In real life, it's often necessary to find corresponding images through text descriptions, or to find the corresponding text description for a particular image. Image-text alignment aims to establish connections and mappings between images and text that share the same or highly similar semantic information. However, due to the different storage methods used for image and text data, there are also inconsistencies between semantic representation and human cognition, leading to two major challenges: the "heterogeneous gap" and the "semantic gap." The "heterogeneous gap" refers to the fact that different multimedia data formats, such as images and text, contain different representations due to their different carrier media. The "semantic gap" refers to the fact that each type of multimedia data, such as images, faces a contradiction between its representation and human cognition. Therefore, image-text alignment requires a unified approach to measure the semantic information contained in different multimedia data formats. Summary of the Invention

[0004] The purpose of this application is to provide an image-text alignment method based on mutual information to overcome the "heterogeneous gap" and "semantic gap" and further improve the performance of image-text alignment.

[0005] In order to achieve the above objectives, the technical solutions of this application are as follows:

[0006] A mutual information-based image-text alignment method, comprising:

[0007] Constructing an image-text alignment network model including a feature extraction module and a feature alignment module, wherein the feature extraction module includes an image feature extraction module and a text feature extraction module, respectively used to extract image features and text features, and the feature alignment module is used to calculate alignment loss, wherein the alignment loss includes intra-modality loss and inter-modality loss;

[0008] Obtain training samples and train the constructed image-text alignment network model to obtain a trained image-text alignment network model;

[0009] The image and text to be aligned are input into the trained image-text alignment network model to obtain the image-to-text sorting or text-to-image sorting, thereby achieving alignment between the image and the text.

[0010] Furthermore, the image feature extraction module includes an image cropping module and four levels connected in sequence, the first level includes a fully connected layer and two Swin-Transformer Blocks, the second level includes an image stitching module and two Swin-Transformer Blocks, the third level includes an image stitching module and eighteen Swin-Transformer Blocks, and the fourth level includes an image stitching module and two Swin-Transformer Blocks.

[0011] Furthermore, the image cropping module divides the input image into pieces according to a preset image block size, and then arranges the obtained image libraries into a row and inputs them into a subsequent network module.

[0012] Furthermore, the image stitching module is used to restore the output of the previous layer into a complete image.

[0013] Furthermore, the text feature extraction module includes a fully connected layer and twelve Transformer encoders connected in sequence.

[0014] Furthermore, the intra-modal loss is used to calculate the intra-modal loss of different enhancements of the same image after passing through the feature extraction module; the inter-modal loss is used to calculate the inter-modal loss of different enhancements of the same image with the text respectively, and then take the maximum value as the final inter-modal loss.

[0015] The present application proposes an image-text alignment method based on mutual information, which can fully utilize the mutual information between image and text data, align image and text pairs that are difficult to distinguish, and improve the image-text alignment performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 This is a flowchart of the image-text alignment method based on mutual information in this application;

[0017] Figure 2 Schematic diagram of the image-text alignment network model for this application. DETAILED DESCRIPTION

[0018] In order to make the purpose, technical solutions and advantages of this application more clear, the following further describes this application in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0019] With the continuous maturity and development of various neural network models and computing resources, deep learning methods have achieved remarkable results in various fields. Transformers, in particular, have achieved impressive accuracy across various tasks in natural language processing and computer vision. Therefore, they are currently widely used to extract semantic features from images and text. Furthermore, with the development of contrastive learning techniques, mutual information (MI) has become a key research topic in the field of image-text alignment. Mutual information is a measure of information in information theory. It can be viewed as the amount of information one random variable contains about another, or the reduction in uncertainty in a random variable due to the knowledge of another. Simply put, images and text with similar or identical semantics should have higher mutual information, while images and text with different semantics should have lower mutual information. By utilizing mutual information and contrastive learning, we can bridge the "heterogeneity gap" and "semantic gap," using a unified approach to measure the semantic information of multimedia data. Based on this understanding of mutual information and combined with contrastive learning, this application proposes a mutual information-based image-text alignment method.

[0020] In one embodiment, Figure 1 As shown, a method for image-text alignment based on mutual information is provided, including:

[0021] Step S1: construct a picture-text alignment network model including a feature extraction module and a feature alignment module, wherein the feature extraction module includes an image feature extraction module and a text feature extraction module, which are used to extract image features and text features respectively, and the feature alignment module is used to calculate the alignment loss, which includes intra-modality loss and inter-modality loss.

[0022] In this embodiment, the image-text alignment network model is as follows Figure 2 As shown, it includes a feature extraction module and a feature alignment module, and the feature extraction module includes an image feature extraction module and a text feature extraction module.

[0023] In a specific embodiment, the image feature extraction module consists of multiple fully connected layers and Swin-Transformer Blocks. The Swin-Transformer Blocks consist of layer normalization and window attention. The image feature extraction module includes an image cropping module and four stages connected in sequence. The image cropping module segments the input image according to a preset image block size, and then arranges the resulting image library in a row and inputs it into the subsequent network module. Specifically, a complete image is segmented into image patches. For example, a 224x224 image is cut into 7x7 patches, a total of 32x32 patches, and then 1024 patches are arranged in a row and input into the subsequent network module. The first stage consists of a fully connected layer and two Swin-Transformer Blocks; the second and fourth stages consist of an image stitching module and two Swin-Transformer Blocks; and the third stage consists of an image stitching module and 18 Swin-Transformer Blocks. The image stitching module restores the image sequence output by the stages into a complete image.

[0024] In this embodiment, the text feature extraction model consists of a fully connected layer and multiple Transformer Encoders. The text is first segmented by a word segmenter and then converted into the corresponding code through text embedding. Text embedding refers to encoding text in the form of a vector, so that the computer can perform calculations on the text. The text feature extraction model consists of a fully connected layer and 12 Transformer Encoders. All the above structures will be standardized to ensure the consistency of the data dimension. All fully connected layers in the feature extraction model are followed by a Relu activation function.

[0025] The feature alignment module in this embodiment is used to calculate alignment loss, which includes intra-modality loss and inter-modality loss. The IMCMI loss function is an intra-modality loss function, which is used to calculate the intra-modality loss of different enhancements of the same image after passing through the feature extraction module. The SNCE loss function is an inter-modality loss function, which is used to calculate the inter-modality loss of different enhancements of the same image with the text, and then take the maximum value as the final inter-modality loss.

[0026] The IMCMI loss function takes different enhancements of an image as input, extracts image features, and then calculates the loss. SNCE is an inter-modal loss function that calculates the loss function for different enhancements of an image and text separately, and then takes the maximum value.

[0027] The two loss functions described above are based on the idea of maximizing the mutual information of positive pairs and minimizing the mutual information of negative pairs. First, the similarity between the corresponding image-text pairs is calculated using the cosine distance between the feature vectors, and then the similarity score is amplified by the softmax function. For the remaining image-text pairs, the similarity is calculated similarly to the previous one, and then the similarity score is amplified by the softmax function and multiplied by the hyperparameter λ. The two are added together to obtain the final loss function, which is then fed back to the network for forward propagation. For details, see the following formula:

[0028]

[0029]

[0030] Here's i,j is the similarity between the i-th image and the j-th text, τ is the temperature coefficient, and λ is the hyperparameter. K represents the size of a batch in one training. exp represents the power operation with natural logarithm as the base, v i and t i Represent the features of image and text respectively. v2t means using image as query and querying text, and t2v means using text as query and querying image.

[0031] Step S2: Obtain training samples and train the constructed image-text alignment network model to obtain a trained image-text alignment network model.

[0032] In this embodiment, to train the network, a dataset is first obtained and then preprocessed. During preprocessing, the dataset is divided into a training set, a validation set, and a test set. The data in the dataset is then randomly augmented and normalized to make it more conducive to model training.

[0033] In this embodiment, network training uses the pre-trained parameters provided by Swin-Transformer, BERT, and X-VLM to initialize the feature extraction module, and inputs the training samples in the training set into the feature extraction module. For images, random horizontal flipping, random vertical flipping, random rotation, random zooming, and other enhancement methods are performed for enhancement processing. Then, features are extracted for the image and text respectively. Finally, the feature alignment module calculates the alignment loss. Two parts of loss are calculated, namely the intra-modal loss and the inter-modal loss, and then the two are combined to form the final loss. The loss error is fed back to the network, the network parameters are adjusted, and the model performance is optimized until the network training is completed.

[0034] Step S3: Input the image and text to be aligned into the trained image-text alignment network model to obtain the image-to-text order or the text-to-image order, thereby achieving alignment of the image and text.

[0035] After training the image-text alignment network model, it can be used to align the image and text to be aligned. The image and text to be aligned are fed into the trained network model, which extracts image and text features, respectively, and then calculates similarity. For images, each text has a similarity. By sorting these similarities, the text most similar to the image is obtained, thus achieving image-text alignment. Conversely, for text, each image has a similarity. By sorting these similarities, the image most similar to the text is obtained.

[0036] For example, for n images I={I1, I2, I3...I n}, and text T = {T1, T2, T3...T n}, calculate the similarity S = {S 12 , S 13 ,...S 1n ...S nn}, S ij Represents the similarity between the i-th image and the j-th text. Then for the similarity S of the t-th image t ={S t1 , S t2 , S t3 ,...S tn}, sort in descending order, for example, get S t ={S t3 , S t1 , S tt ,...S tn}. Then it is finally considered that the tth image and the third text are corresponding and aligned.

[0037] In this application, the technical solution of this application was experimentally verified. During the experiment, based on the Flickr dataset and the MSCOCO dataset, the current mainstream image-text alignment algorithm was compared, using the three standards of R@1, R@5, and R@10. Among them, R@X means the frequency of the correct text (image) found in the test set with the image (text) as the query being ranked in the top X.

[0038] Table 1. Experimental results on the MSCOCO (5K) dataset

[0039]

[0040] Table 2. Experimental results of Flickr30K dataset

[0041]

[0042] Tables 1 and 2 show the test results on the MSCOCO (5K) dataset and the Flickr 30K dataset, respectively. Comparison of the two tables demonstrates the superior performance of our method on the image-text alignment task. All metrics on MSCOCO (5K) outperform previous methods, and most metrics on Flickr 30K outperform previous methods.

[0043] The above-described embodiments merely represent several implementation methods of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the present invention. It should be noted that a person skilled in the art could make various modifications and improvements without departing from the spirit of the present application, all of which fall within the scope of protection of the present application. Therefore, the scope of protection of the present patent application shall be determined by the appended claims.

Claims

1. A method for image-text alignment based on mutual information, characterized in that: The mutual information-based image-text alignment method includes: Constructing an image-text alignment network model including a feature extraction module and a feature alignment module, wherein the feature extraction module includes an image feature extraction module and a text feature extraction module, respectively used to extract image features and text features, and the feature alignment module is used to calculate alignment loss, wherein the alignment loss includes intra-modality loss and inter-modality loss; Obtain training samples and train the constructed image-text alignment network model to obtain a trained image-text alignment network model; Input the image and text to be aligned into the trained image-text alignment network model to obtain the image-to-text order or the text-to-image order, thus achieving image-to-text alignment. The image feature extraction module includes an image cropping module and four layers connected in sequence. The first layer includes a fully connected layer and two Swin-Transformer Blocks, the second layer includes an image stitching module and two Swin-Transformer Blocks, the third layer includes an image stitching module and eighteen Swin-TransformerBlocks, and the fourth layer includes an image stitching module and two Swin-Transformer Blocks. The text feature extraction module includes a fully connected layer and twelve Transformer encoders connected in sequence.

2. The image-text alignment method based on mutual information according to claim 1, characterized in that: The image cropping module divides the input image into sections according to the preset image block size, and then arranges the obtained image library into a row and inputs it into the subsequent network module.

3. The image-text alignment method based on mutual information according to claim 1, characterized in that: The image stitching module is used to restore the output of the previous layer into a complete image.

4. The image-text alignment method based on mutual information according to claim 1, characterized in that: The intra-modal loss is used to calculate the intra-modal loss of different enhancements of the same image after passing through the feature extraction module; the inter-modal loss is used to calculate the inter-modal loss of different enhancements of the same image with the text respectively, and then take the maximum value as the final inter-modal loss.

Images