A pre-training method robust to text-image pair error matching

Abstract

The application discloses a pre-training method robust to wrong matching of image-text pairs, and aims to solve the wrong matching of image-text pairs encountered in visual-language pre-training, and designs a simple and effective robust pre-training method, which effectively solves the wrong matching of image-text pairs without increasing model parameters and training overhead, improves the visual-language representation ability of the model, and enhances the performance in tasks such as image-text retrieval, image positioning, visual question answering, visual reasoning and the like. Negative learning is first proposed in visual-language pre-training, so that the pre-training model is robust to wrong matching of image-text pairs, the representation ability is improved, and better performance is obtained in downstream tasks.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of image-text matching neural networks, and particularly relates to a pre-training method robust to image-text pair error matching. BACKGROUND

[0002] Visual language pre-training (VLP) aims to learn a general representation from large-scale image-text pairs, hoping to facilitate various V+L downstream tasks, such as image / text retrieval, visual question answering, and visual reasoning, etc. To this end, most existing visual language pre-training methods usually employ some self-supervised proxy tasks, such as masked language modeling task, image-text matching task, and image-text contrastive learning, etc. to explore the knowledge rooted in cross-modal pairs. Despite the remarkable achievements, almost all existing VLP methods inevitably encounter the problem of image-text pair error matching. It refers to the irrelevant image-text pairs being wrongly considered as matching, for example, a picture of an airplane taking off and the text "a car is driving on the road", which are irrelevant, but are wrongly associated together. However, visual language pre-training methods usually require large-scale correctly matched image-text pairs for model pre-training, and obviously these error-matched image-text pairs will have a negative impact on model training, causing the model to wrongly associate the visual image of the airplane with the "car" text, affecting the joint visual and language representation ability of the model, and thus impairing the performance in downstream tasks.

[0003] CLIP (Contrastive Language-Image Pre-training) model is a pre-training neural network model for matching images and texts released by OpenAI in early 2021. In the field of multi-modal research, it can be regarded as a classic work. The model exhibits the best performance in many tasks by directly pre-training using a large amount of Internet data. On the Internet, there are a large number of image-text pairs, usually in web pages, developers add text descriptions to images. In fact, these materials can be regarded as a labeled dataset, training using them can solve the high cost problem of obtaining labeled data, and because the Internet data is large and has great diversity, it is easier to obtain a model with good generalization ability. The CLIP model is based on the above concept, using the 400 million image-text pairs collected by OpenAI, the text and image are encoded respectively, and the metric learning is used for training. Its goal is to improve the similarity between image and text. The following figure is the flowchart of CLIP, which mainly obtains the text feature through the text encoder, obtains the image feature through the image encoder, and finally calculates the similarity of the two to perform contrastive learning, thereby realizing the joint representation of vision-image. Although CLIP uses a large number of image-text pairs, these data are obtained from the network, and there are a large number of incorrectly matched image-text pairs, and CLIP itself does not process and screen these data, which will inevitably affect the visual-linguistic representation ability of CLIP. SUMMARY

[0004] In view of the above problems in the prior art, the present application provides a pre-training method robust to incorrect matching of image-text pairs.

[0005] In order to achieve the above-mentioned purpose of the application, the technical scheme adopted by the present application is:

[0006] A pre-training method robust to incorrect matching of image-text pairs, comprising the following steps:

[0007] S1, initializing the model network structure and weight parameters, wherein the network structure comprises a visual encoder, a text encoder, a multi-modal fusion module and a momentum network;

[0008] S2, inputting a picture and a text corresponding to the picture into the corresponding visual encoder and text encoder respectively for feature extraction to obtain the features of the picture and the text, and inputting the picture and the text corresponding to the picture into the momentum network to obtain the model prediction;

[0009] S3. Input the image and text features obtained in S2 into the multimodal fusion module, and calculate the mask language modeling objective function and the image-text matching objective function. At the same time, obtain the model's prediction results for the image-text pair, and input the prediction results into the divider as the basis for judging whether the image and text match.

[0010] S4. The divider judges the noise of the image-text pairs based on the model prediction results. It uses the features of the noisy image-text pairs as negative contrastive learning for calculation, and uses the features of the non-noisy image-text pairs as positive contrastive learning for calculation. The calculation results of both are used as the calculation results after model prediction from the autovariable network as the learning target for image-text comparison.

[0011] S5. Using the mask language modeling objective function, image-text matching objective function from S3, and the image-text comparison results from S4, perform gradient optimization on the visual encoder, text encoder, and multimodal fusion module, and update the momentum network according to the corresponding weight momentum.

[0012] Furthermore, the mask language modeling objective function in S3 is expressed as:

[0013]

[0014] Where I and T are the input image and text, respectively, and D is the training dataset where I, T∈D. For the masked text, For real labels, The model predicts labels, and CE is the cross-entropy function. Let be the expectation of the cross-entropy function.

[0015] Furthermore, the image-text matching objective function in S3 is expressed as:

[0016]

[0017] Where I and T are the input image and text, respectively, and D is the training dataset where I, T∈D. For the masked text, y itm For image-text matching tasks, p itm To predict labels for the model in an image-text matching task. Let be the expectation of the cross-entropy function.

[0018] Furthermore, the specific method of forward learning in S3 is as follows:

[0019]

[0020] Where I and T are the input image and text, respectively, and N is the input text. b y represents the current batch size.itc and p nce represent the real label and the model predicted label respectively, I m is the m-th image in the current N b number of images, T m is the m-th text in the current N b number of texts, denotes the real label corresponding to I m and T m .

[0021] Further, the specific way of the negative learning in S3 is:

[0022]

[0023] wherein, I m is the m-th image in the current M m number of images, T nce is the m-th text in the current M m number of texts, p m is the model predicted label, is the real label of I t and T v obtained by sharpening the momentum network.

[0024] Further, the specific way of the image-text matching in S4 is:

[0025]

[0026]

[0027] wherein, M is the size of the positive sample pair and the negative sample pair to be calculated, s(·) is a similarity calculation function and f t and f v are text and image encoders respectively, v cls , t cls are [cls] tokens in the image and the text respectively, τ is a learnable temperature coefficient, p nce (I, T m ) and p nce (I m , T) are the model predicted labels of (I, T m ) and (I m , T) in the image-text contrast learning respectively.

[0028] Further, the specific way of the gradient optimization in S5 is:

[0029] y itc = αy itc + (1-α)Ψ

[0030]

[0031] where, is the prediction of the momentum network, is the momentum update coefficient, sharpen is the common label sharpening operation, is the real label after sharpening.

[0032] The present application has the following beneficial effects:

[0033] A simple and effective robust pre-training method is designed to solve the problem of incorrect matching of image-text pairs encountered in visual-linguistic pre-training. Without additional model parameters and training overhead, the method effectively solves the incorrect matching of image-text pairs, improves the visual-linguistic representation ability of the model, and enhances the performance in tasks such as image-text retrieval, image positioning, visual question answering, and visual reasoning. BRIEF DESCRIPTION OF DRAWINGS

[0034] Figure 1 is a pre-training method flow diagram for robustness to incorrect matching of image-text pairs. DETAILED DESCRIPTION

[0035] The specific embodiments of the present application are described below to facilitate understanding of the present application by those skilled in the art, but it should be clear that the present application is not limited to the scope of the specific embodiments. For those skilled in the art, it is obvious that various changes are within the spirit and scope of the present application as defined in the appended claims, and all inventions utilizing the concept of the present application are within the scope of protection.

[0036] A pre-training method for robustness to incorrect matching of image-text pairs, characterized in that it comprises the following steps:

[0037] S1, initialize the model network structure and weight parameters, wherein the network structure includes a visual encoder, a text encoder, a multi-modal fusion module and a momentum network;

[0038] In this scheme, the model mainly includes four, a visual encoder, a text encoder, a multi-modal fusion module, and a momentum network.

[0039] The visual encoder uses a 12-layer ViTbase model with 85.8M model parameters and is pre-trained on ImageNet.

[0040] The text encoder and the multi-modal fusion module use the first 6 layers and the last 6 layers of the BERTbase model, respectively, with a total of 123.7M model parameters.

[0041] The momentum network is a momentum update version of the visual encoder, the text encoder, and the multi-modal fusion module, which does not participate in gradient update, but is updated by momentum after the gradient update of the above three models.

[0042] S2, input a picture and text corresponding to the picture into the corresponding visual encoder and text encoder for feature extraction to obtain the features of the picture and the text, and input the picture and the text corresponding to the picture into the momentum network to obtain a model prediction;

[0043] S3, input the features of the picture and the text obtained in S2 into the multi-modal fusion module, and perform mask language modeling objective function calculation and picture-text matching objective function calculation, obtain a prediction result of the model on the picture-text pair, and input the prediction result into the partitioner as a basis for judging whether the picture-text pair matches;

[0044] In this embodiment, the target function and the model are introduced:

[0045] The three target functions of mask language modeling, picture-text matching, and picture-text comparison are defined as follows:

[0046] Mask language modeling objective function:

[0047]

[0048] I, T are input pictures and text respectively, D is a training data set and I, T ∈ D, is the masked text, is the true label, is the model prediction label, CE represents the cross-entropy function, which is used to calculate the difference between the prediction label and the true label to obtain a loss function value, is the expectation of the cross-entropy function. Mask language modeling first randomly masks T, i.e., replaces some words in the text with [mask], and then inputs the picture and the text into the model to let the model predict the replaced words.

[0049] Picture-text matching objective function:

[0050]

[0051] where y itm is the true label in the picture-text matching task, p itm is the model prediction label in the picture-text matching task. Picture-text matching predicts whether the input text and image match to obtain a prediction label and a true label, and calculates the cross-entropy to calculate the loss function value.

[0052] For forward learning:

[0053]

[0054] y itc and p nce represent the true label and the model predicted label respectively, I m is the m-th image in the current N b number of images, T m is the m-th text in the current N b number of texts, y m and T m corresponding to the true label, y itc will first be smoothed by the momentum network. The main purpose of the positive image-text contrast is to increase the similarity between positive sample pairs, thereby achieving the alignment of images and texts.

[0055] For negative learning:

[0056]

[0057] is the true label of I m and T m obtained by sharpening through the momentum network. The main purpose of negative learning is to reduce the similarity between negative sample pairs, thereby avoiding the possibility of false matching of positive sample pairs, and indirectly achieving the alignment of images and texts.

[0058] S4, the noise judgment of the image-text pair is performed according to the model prediction result, the noise image-text pair features are calculated for negative contrast learning, and the non-noise image-text pair features are calculated for positive contrast learning, and the calculation results of the two are used together to use the calculation result after the model prediction of the momentum network as the learning target to compare the image and the text;

[0059] Image-text contrast:

[0060] Image-text contrast consists of two parts, namely positive learning and negative learning, which mainly distinguish the input image-text pair according to the prediction value of the image-text matching, and the prediction value is high as positive learning and low as negative learning. Both of them first calculate the similarity of the image and the text:

[0061] Given an image I and a text T, we calculate the similarity of the image to the text and the similarity of the text to the image as follows

[0062]

[0063]

[0064] M represents the size of the positive sample pair and the negative sample pair to be calculated, and s(·) is a similarity calculation function: ft and f v represent text and image encoders respectively, cls represents the global representation, tau is a learnable temperature coefficient initialized to 0.07, p nce (I, T m ) and p nce (I m , T) are the model predicted labels for (I, T m ) and (I m , T) respectively in image-text contrastive learning.

[0065] S5, the mask language modeling objective function, the image-text matching objective function in S3 and the image-text matching result in S4 are used to perform gradient optimization on the visual encoder, the text encoder and the multi-modal fusion module, and the momentum network is updated according to the corresponding weight momentum.

[0066] The partitioner mainly partitions the input image-text pair according to the prediction of the image-text matching, and the prediction with high value is used for positive learning and the prediction with low value is used for negative learning.

[0067] The smoother and the sharpener are mainly used to smooth and sharpen the real label by using the momentum network, and the specific operation is as follows:

[0068] y itc = alpha * y itc + (1-alpha) * psi

[0069]

[0070] Wherein, psi is the prediction of the momentum network, alpha is the momentum update coefficient, sharpen is a common label sharpening operation, is the real label after sharpening.

[0071] Experimental verification

[0072] We use the visual-linguistic model pre-trained by our method in image-text retrieval, visual question answering, visual positioning and other tasks, and effectively improve the performance, as shown in the following table (NEVER represents our method):

[0073] Image-text retrieval, the test results on the COCO and Flickr30K datasets are shown in Table 1

[0074] Table 1

[0075]

[0076] The test results on the visual question answering (VQA), visual reasoning (NLVR) and visual entailment (SNLI-VE) tasks are shown in Table 2:

[0077] Table 2

[0078]

[0079] The test results on the visual positioning task are shown in Table 3:

[0080] Table 3

[0081]

[0082] The comparative experiments on multiple tasks effectively prove the effectiveness of the application.

[0083] The present application is described in reference to the flowcharts and / or block diagrams of the method, device (system), and computer program product according to the embodiments of the present application. It should be understood that each flow and / or block in the flowcharts and / or block diagrams, and the combination of the flows and / or blocks in the flowcharts and / or block diagrams can be implemented by computer program instructions. These computer program instructions can be provided to the processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device produce a device that implements the flowcharts and / or block diagrams. Figure 1 one or more flows and / or blocks Figure 1 an apparatus that performs the functions specified in one or more blocks.

[0084] These computer program instructions can also be stored in a computer-readable memory that can direct the computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including instruction apparatus, which implements the flowcharts and / or block diagrams. Figure 1 one or more flows and / or blocks Figure 1 an apparatus that performs the functions specified in one or more blocks.

[0085] These computer program instructions can also be loaded into a computer or other programmable data processing device, so that a series of operation steps are performed on the computer or other programmable device to produce a computer-implemented process, so that the instructions executed on the computer or other programmable device provide a process for implementing the flowcharts and / or block diagrams. Figure 1 one or more flows and / or blocks Figure 1 an apparatus that performs the functions specified in one or more blocks.

[0086] The principles and implementation methods of the present application are described in the specific embodiments, and the above embodiment descriptions are only used to help understand the method of the present application and its core idea; at the same time, for those skilled in the art, according to the idea of the present application, the specific implementation and application range will be changed, and the above description should not be understood as a limitation of the present application.

[0087] Those skilled in the art will appreciate that the embodiments described herein are presented for purposes of illustration and understanding of the principles of the application and should not be construed as limiting the scope of the application to such specifically outlined embodiments and examples. Various other specific embodiments and examples not described herein will be apparent to those skilled in the art in view of the teachings provided herein. The scope of the application should be determined from the claims.

Claims

1. A pre-training method robust to graph-text pair error matching, characterized in that, The method comprises the following steps: S1, initializing a model network structure and weight parameters, wherein the network structure comprises a visual encoder, a text encoder, a multi-modal fusion module and a momentum network; S2, inputting a picture and text corresponding to the picture into corresponding visual encoders and text encoders respectively for feature extraction to obtain features of the picture and the text, and inputting the picture and the text corresponding to the picture into the momentum network to obtain a model prediction; S3, inputting the features of the picture and the text obtained in S2 into the multi-modal fusion module, and performing mask language modeling objective function calculation and picture-text matching objective function calculation, obtaining a prediction result of the model on the picture-text pair, and inputting the prediction result into a divider as a basis for judging whether the picture-text pair matches, wherein the mask language modeling objective function is represented as: wherein, are inputted picture and text, respectively, is a training data set and , is a masked text, is a true label, is a model predicted label, and CE is a cross-entropy function, is an expectation of the cross-entropy function; The picture-text matching objective function is represented as: wherein, are an input picture and text, respectively, is a training dataset and , is a masked text, is a true label in a picture-text matching task, is a model predicted label in a picture-text matching task, is an expectation of a cross-entropy function; The specific way of forward learning is: wherein, are input pictures and text, respectively, is the current data batch size, and represent the true label and the model predicted label, respectively, is the current is the m-th picture in the number of pictures, , denotes and the corresponding true label; The specific way of negative learning is: wherein, is the current is the mth picture in the number of pictures, is the mth text in the number of texts, is the model predicted label, is the sharpened and is the true label. S4, the divider judges noise of the picture-text pair according to the model prediction result, and uses features of a noise picture-text pair as negative contrast learning for calculation, and uses features of a non-noise picture-text pair for forward contrast learning calculation, and the calculation results of the two are used together to use a calculation result of the model prediction of the momentum network as a learning target to compare the picture-text pair; S5, the mask language modeling objective function in S3, the picture-text matching objective function and the picture-text comparison result in S4 are used to perform gradient optimization on the visual encoder, the text encoder and the multi-modal fusion module, and the momentum network is updated according to the corresponding weight momentum.

2. The pre-training method robust to text-image pair error matching according to claim 1, characterized in that, The specific way of picture-text comparison in S4 is: where M is the size of positive and negative sample pairs to be calculated, is a similarity calculation function and , and are text and image encoders, respectively, , refer to [cls] tokens in images and texts, respectively, is a learnable temperature coefficient, and are model predicted labels for and in image-text contrastive learning, respectively.

3. The pre-training method robust to text-image pair error matching according to claim 1, characterized in that, The specific way of gradient optimization in S5 is: where, is the prediction of the momentum network, is the momentum update coefficient, sharpen is the common label sharpening operation, is the real label after sharpening.

Images