Cross-modal image-text retrieval method based on spatial prediction type VLPM and storage medium

Abstract

The application discloses a cross-modal image-text retrieval method based on a spatial prediction type visual language pre-training model and a storage medium. The method introduces local semantic prediction visual space positions into a visual language pre-training model, so that the pre-training model can learn more effective high-level semantics, and effectively improves the performance of a downstream cross-modal image-text retrieval task. Feature extraction is performed on downstream image-text retrieval data based on the spatial prediction type visual language pre-training model, a quadruple loss function is constructed, and supervised cross-modal image-text retrieval is performed. A quadruple construction module is used to mine false negative example samples with strong semantic correlation with positive samples, thereby reducing the influence of the false negative example samples on image-text retrieval.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer application, and particularly relates to a cross-modal image-text retrieval method based on a spatial prediction type vision and language pre-training model and a storage medium. BACKGROUND

[0002] Cross-modal image-text retrieval aims to realize information retrieval across image and text data, and builds a bridge between vision and natural language. At present, in cross-modal image-text retrieval, a vision and language pre-training model (VLPM) based image-text retrieval method has good performance and is widely used. The method includes two key processes: one is to train a vision and language pre-training model (VLPM) on a large-scale pre-training dataset; and the other is to fine-tune the vision and language pre-training model, so that the fine-tuned vision and language pre-training model can be applied to the image-text retrieval task.

[0003] For the first process, contrast learning is currently mainly used for model training. For the second process, the trained vision and language pre-training model is directly fine-tuned and applied to the downstream cross-modal image-text retrieval. Patent application CN115221276A discloses a Chinese image-text retrieval model training method based on CLIP: first, a sample is encoded by using a Chinese encoder, and a first similarity between positive samples and a second similarity between a positive sample vector and a negative sample vector set are calculated; then, the Chinese encoder is trained according to the first similarity and the second similarity, and the encoder is replaced with an English encoder in CLIP; finally, the Chinese image-text CLIP model is trained for image-text pair matching by using a Chinese image-text training set, and a target Chinese image-text CLIP retrieval model is obtained.

[0004] However, the above prior art still has the following shortcomings: (1) In the training process of the vision and language pre-training model, high-level semantics are not integrated into the model, which will seriously hinder the performance of the downstream image-text retrieval task. (2) Although the vision and language pre-training model is successfully fine-tuned to the unsupervised cross-modal image-text retrieval, the cross-modal retrieval performance needs to be improved due to the lack of necessary supervision information. (3) The influence of false negative samples on the performance of image-text retrieval is not considered. SUMMARY

[0005] In view of the shortcomings of the prior art, the present application provides a cross-modal image-text retrieval method based on a spatial prediction type vision and language pre-training model, which is embodied in the following:

[0006] (1) Construct a space prediction type visual language pre-training model, the purpose is to predict the visual space position by local semantic, make the model learn more effective high-level semantic, effectively improve the performance of downstream cross-modal image-text retrieval task.

[0007] (2) Take the space prediction type visual language pre-training model as the backbone network, and construct a four-tuple loss function to supervise the training of the cross-modal image-text retrieval model.

[0008] (3) By mining false negative example samples with strong semantic association with positive samples, reduce the influence of false negative example samples on image-text retrieval.

[0009] The technical scheme of the application specifically includes the following steps:

[0010] S1: Obtain a plurality of groups of pre-training data of image-text pairs, wherein, including training images and pre-training texts for describing pre-training images.

[0011] S2: Use the plurality of groups of image-text paired data to train the space prediction type visual language pre-training model, and obtain a visual language pre-training model capable of learning higher level image-text semantics. The specific training method is as follows:

[0012] S2-1: Use a local information extraction module to extract local information from the pre-training text, and use a first feature extractor to obtain local text features t.

[0013] Further, the step S2-1 can be specifically implemented as follows:

[0014] Tokenize the pre-training text; analyze the relationship between words, including subject-predicate relationship, verb-object relationship, adverbial structure, definite relationship, parallel relationship and adverbial relationship.

[0015] According to the relationship between the words, a pre-training text relationship structure tree is constructed.

[0016] Get the relationship structure sub-tree from the relationship structure tree, and form a sentence according to the order of each word in the sentence, which is the local information of the pre-training text.

[0017] S2-2: Crop the pre-training image into n*n blocks, n is a positive integer, and use a second feature extractor to obtain image block features x i ,i∈[0,n 2 -1] respectively.

[0018] S2-3: Calculate the cosine similarity c i of the local text features t and the image block features x i , and get the top K image blocks with the highest cosine similarity.

[0019] The formula for cosine similarity is as follows:

[0020]

[0021] S2-4: Calculate the ratio m of the overlapping area of ​​the local text feature t and the first K image patches, respectively. i .

[0022] S2-5: Calculate the cosine similarity c corresponding to the i-th image block. i and the ratio of overlapping areas m i The distance.

[0023] The distance formula is as follows:

[0024]

[0025] Where |·| represents absolute value.

[0026] S2-6: Use the spatial distance loss function to constrain the image patch location information matched by local text features to have high similarity, and complete the training of the spatial prediction visual language pre-training model.

[0027] The formula for the spatial distance loss function is as follows:

[0028]

[0029] S3: Obtain multiple sets of training data for image and text retrieval, including training images and training text used to describe the training images.

[0030] S4: Use the trained spatial prediction visual language pre-trained model to extract features from the training samples of image and text retrieval, and obtain more expressive deep features of image and text data.

[0031] S5: Use the quadruplet construction module to construct quadruplets from the training samples of image and text retrieval, including samples, positive example set, false negative example set, and negative example set.

[0032] S5-1: For image retrieval tasks, given an image query... The task is to retrieve the most relevant text set T from the database. The text set T is split into a positive set T0. p and the negative example set T n It is necessary to obtain it from the negative example set T. n Identify the positive example set T p Equivalent false negatives. Similar to the positive example t. i ∈T p Highly similar negative examples t j ∈T n It is considered a false negative. Furthermore, if there exists a... Highly similar images Then the image The relevant text is also included in the false negative set FN T middle.

[0033] FN T The formula is as follows:

[0034]

[0035] Here, hyperparameters α and β represent the similarity thresholds for text and image modalities, respectively, and are used to select false negatives from the negative example set. Function Indicates query image Retrieve relevant text. Neg X and Neg T These represent the negative instances of the image and the text, respectively.

[0036] S5-2: For text retrieval tasks, the query is text t i The task is to retrieve the most relevant set of images G from the database. This involves splitting the image set G into a set of positive examples Gi. p and negative example set G n From the negative example set G n Identify the positive example set G p Equivalent false negatives. (Similar to positive examples) Highly similar negative examples This is considered a false negative. Furthermore, if there exists a text t... i Highly similar text t s Then the text t s The relevant images should also be included in the false negative set FN. X middle.

[0037]

[0038] Among them, the function F(t) s ) indicates based on the query text t s Retrieve relevant images.

[0039] S5-3: The negative instance sets for images and text are redefined as follows:

[0040]

[0041]

[0042] S5-4: The four-tuples for image retrieval and text retrieval are as follows:

[0043]

[0044] Y T =[t i ,Xp FN X Neg X ].

[0045] where Y X represents the image retrieval quadruple, including the image query positive text set T p , false negative text set FN T and negative text set Neg T '. Y T represents the text retrieval quadruple, including the text query t i , positive image set X p , false negative image set FN X and negative image set Neg X '.

[0046] S6: Constructing a quadruple loss function to supervise the training of the cross-modal image-text retrieval model.

[0047] A preferred quadruple loss function is provided here, and the formula is as follows:

[0048]

[0049] where |T p |, |FN T | and |Neg T '| are the cardinalities of the positive set, false negative set and negative set in the image retrieval result. |X p |, |FN X | and |Neg X '| are the cardinalities of the positive set, false negative set and negative set in the text retrieval result. ρ1 and ρ2 are balancing factors. λ1 is used to balance the relative contributions of positive examples and false negative examples in image retrieval. λ2 is used to balance the relative contributions of positive examples and false negative examples in text retrieval. S(·,·) represents the cosine similarity measure function.

[0050] S7: Inputting the image or text to be retrieved into the trained cross-modal image-text retrieval model to obtain the corresponding retrieval result.

[0051] The application also provides a non-transitory computer readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the above cross-modal image-text retrieval method.

[0052] The technical features and beneficial effects of the application are as follows:

[0053] By introducing the local semantic prediction visual space position into the visual language pre-training model, the pre-training model can learn more effective high-level semantics, effectively improving the performance of the downstream cross-modal image-text retrieval task; the spatial prediction type visual language pre-training model is used for feature extraction of downstream image-text retrieval data, a four-tuple loss function is constructed, and supervised cross-modal image-text retrieval is performed; the four-tuple construction module is used to mine false negative example samples with strong semantic correlation with positive samples, reducing the influence of false negative example samples on image-text retrieval. BRIEF DESCRIPTION OF DRAWINGS

[0054] Figure 1 is a flowchart of the present application;

[0055] Figure 2 is a flowchart of the training process of the spatial prediction type VLPM, the target model training and testing process in the embodiment of the present application;

[0056] Figure 3 is a schematic diagram of the local information extraction module constructing a pre-training text relationship structure tree in the embodiment of the present application. DETAILED DESCRIPTION

[0057] The present application will be further described in detail below in combination with the drawings and embodiments, so as to better understand the technical solutions of the present application.

[0058] In a specific embodiment, the spatial prediction type visual language pre-training model is as follows:

[0059] The local information extraction module is used to extract local information from the pre-training text, and a first feature extractor is used to obtain local text features t;

[0060] The pre-training image is cropped into n*n blocks, and a second feature extractor is used to obtain image block features x i ,i∈[0,n 2 -1] respectively.

[0061] The cosine similarity c i of the local text features t and the image block features x i is calculated, and the top K image blocks with the highest cosine similarity are obtained.

[0062] The cosine similarity formula is as follows:

[0063]

[0064] The proportion m i of the overlapping area of the local text features t and the top K image blocks is calculated respectively.

[0065] The cosine similarity c iand the ratio of overlapping areas m i The distance;

[0066] The distance formula is as follows:

[0067]

[0068] Where |·| represents absolute value.

[0069] The spatial distance loss function is used to constrain the image patch location information matched by local text features to have high similarity, thus completing the training of a spatial prediction-type visual language pre-training model.

[0070] The formula for the spatial distance loss function is as follows:

[0071]

[0072] In one specific embodiment, the local information extraction module includes:

[0073] Perform word segmentation on the pre-trained text;

[0074] Analyze the relationships between words, including subject-predicate relationships, verb-object relationships, adverbial-head structures, attributive-head relationships, coordinate relationships, and prepositional phrase relationships;

[0075] Construct a pre-trained text relationship structure tree based on the relationships between the words;

[0076] Obtain the relation structure subtree from the relation structure tree and form a sentence according to the order in which each word appears in the sentence. This is the local information of the pre-trained text.

[0077] Taking the pre-training text "A man in black is walking a white dog on the beach" as an example, the relational structure tree of this pre-training text is as follows: Figure 3 As shown, partial information such as "a man wearing black clothes", "a white dog", "a man walking a dog", and "a man walking on the beach" can be obtained.

[0078] In one specific embodiment, constructing quadruples from the training samples of image-text retrieval using the quadruple construction module includes:

[0079] For image retrieval tasks, given an image query The task is to retrieve the most relevant text set T from the database. The text set T is split into a positive set T0. p and the negative example set T n It is necessary to obtain it from the negative example set T. n Identify the positive example set T p Equivalent false negatives. Similar to the positive example t. i ∈T p Highly similar negative examples tj ∈T n It is considered a false negative. Furthermore, if there exists a... Highly similar images Then the image The relevant text is also included in the false negative set FN T middle.

[0080] FN T The formula is as follows:

[0081]

[0082] Here, hyperparameters α and β represent the similarity thresholds for text and image modalities, respectively, and are used to select false negatives from the negative example set. Function Indicates based on the query image Retrieve relevant text. Neg X and Neg T These represent the negative instances of the image and the text, respectively.

[0083] For text retrieval tasks, the query is text t i The task is to retrieve the most relevant set of images G from the database. This involves splitting the image set G into a set of positive examples Gi. p and negative example set G n From the negative example set G n Identify the positive example G p Equivalent false negatives. (Similar to positive examples) Highly similar negative examples This is considered a false negative. Furthermore, if there exists a text t... i Highly similar text t s Then the text t s The relevant images should also be included in the false negative set FN. X middle:

[0084]

[0085] Among them, the function F(t) s ) represents the query text t s Retrieve relevant images.

[0086] The negative sets of images and text are redefined respectively. and

[0087]

[0088] The four-tuples for image retrieval and text retrieval are as follows:

[0089]

[0090] Y T=[t i ,X p ,FN X ,Neg′ X (7)

[0091] Among them, Y X This represents an image retrieval quadruple, including image queries. Positive example text set T p False negative example text set FN T Neg's negative example text set T Y T This represents a text retrieval quadruple, including the text query t. i , positive example image set X p False negative example image set FN X Neg′ and negative example image set X .

[0092] In one specific embodiment, the quadruplet loss function is as follows:

[0093]

[0094] Among them, |T p |、|FN T |and|Neg′ T | is the cardinality of the positive example set, false negative example set, and negative example set in the image retrieval results. |X p |、|FN X |and|Neg′ X | represents the cardinality of the positive, false negative, and negative example sets in the text retrieval results. ρ1 and ρ2 are balance factors. λ1 is used to balance the relative contributions of positive and false negative examples in image retrieval. λ2 is used to balance the relative contributions of positive and false negative examples in text retrieval. S(·,·) denotes the cosine similarity measure function.

Claims

1. A cross-modal image-text retrieval method based on spatial prediction type VLPM, characterized in that, The method comprises: Obtaining a plurality of groups of pre-training samples of image-text pairs, including pre-training images and pre-training texts for describing the pre-training images; Training a spatial prediction visual language pre-training model using the pre-training samples; Obtaining a plurality of groups of training samples for image-text retrieval, and performing feature extraction on the training samples for image-text retrieval using the trained spatial prediction visual language pre-training model; Using a four-tuple construction module to construct four-tuples for the training samples for image-text retrieval, including samples, positive sets, false negative sets and negative sets; Constructing a four-tuple loss function to supervise the training of the cross-modal image-text retrieval model; Inputting an image or text to be retrieved into the trained cross-modal image-text retrieval model to obtain a corresponding retrieval result; The training method of the spatial prediction visual language pre-training model comprises the following steps: (1) using a local information extraction module to extract local information from the pre-training samples, and using a first feature extractor to obtain local text features t; (2) the pre-training image is cropped into n x n blocks, n is a positive integer, and an image block feature x is obtained using a second feature extractor i respectively 2 ; i ∈ [0, n-1] (3) calculating the local text feature t and the image block feature x i cosine similarity c i obtaining the top K image blocks with the highest cosine similarity (4) calculating the local text feature t and the proportion m of the overlapping area of the top K image blocks, respectively i ; (5) calculating the cosine similarity c corresponding to the ith block of images i and the overlap area ratio m i the distance; (6) using a spatial distance loss function to constrain the position information of the image block matched by the local text features to have high similarity, and completing the training of the spatial prediction visual language pre-training model.

2. The method of claim 1, wherein, The formula of the four-tuple loss function is: Y T = [t i , X p , FN X , Neg′ X ]; where Y X denotes the image retrieval quadruple, including image query positive text set T p , false negative text set FN T and negative text set Neg′ T ; Y T denotes the text retrieval quadruple, including text query t i , positive image set X p , false negative image set FN X and negative image set Neg′ X ; |T p |, |FN T | and |Neg′ T | are the cardinalities of the positive set, false negative set and negative set in the image retrieval result respectively; |X p |, |FN X | and |Neg′ X | are the cardinalities of the positive set, false negative set and negative set in the text retrieval result respectively; ρ1 and ρ2 are balancing factors; λ1 is used to balance the relative contribution of positive and false negative in image retrieval; λ2 is used to balance the relative contribution of positive and false negative in text retrieval; S(·,·) denotes the cosine similarity measure function.

3. A non-transitory computer-readable storage medium having stored thereon a computer program, which, when executed by a processor, implements the steps of the cross-modal image-text retrieval method of claim 1 or 2.

Images