A multi-granularity image-text matching method and system based on deep fusion

Through the multi-grained image-text matching method based on deep fusion, the encoder is trained using in-modal and inter-modal features, the problems of mutual interference between modals and high computing power requirements in image text matching are solved, and more efficient Feature learning and matching accuracy are improved.

CN117093692BActive Publication Date: 2024-11-01GUANGZHOU SHUOGUO LELEI NETWORK TECHNOLOGY CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202311063453.4
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2023-08-23
Publication Date
2024-11-01
Estimated Expiration
2043-08-23

AI Technical Summary

Technical Problem

The existing image text matching technology has problems of mutual interference between modals and high computing resource requirements when learning image and text feature representation, and the multi-grained method has limited effect on complex correlation learning.

Method used

Using a multi-grained image-text matching method based on deep fusion, different encoders are trained through in-modal and inter-modal features to avoid mutual interference, and cross-modal feature representation is learned through multi-grained visual coding and deep fusion encoder. Combining hard negative pairs improves retrieval accuracy.

Benefits of technology

Effectively learn the feature representation of images and text, reduce the demand for computing power resources, improve the matching accuracy of multi-grained visual (text) encoders, and fully capture the complex relationships between different modes.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN117093692B_ABST
    Figure CN117093692B_ABST
Patent Text Reader

Abstract

The present application discloses a multi-granularity image-text matching method and system based on deep fusion, the method comprising: performing multi-granularity visual encoding on the image to be matched to obtain the encoded image; performing multi-granularity visual encoding on the text to be matched to obtain the encoded text; training an encoder based on the encoded image and the encoded text to obtain an initial encoder; performing deep fusion on the initial encoder to obtain a cross-modal encoder; training the cross-modal encoder to obtain a final model, and using the final model to complete the matching of the image and the text. The present application uses intra-modal and inter-modal features to train different encoders to avoid mutual interference between the two, thereby effectively learning the feature representation of each modality and reducing the requirements for computing resources. A deep fusion method is adopted to more fully learn the complex associations between different modalities. In the training process of the multi-granularity visual encoder, hard negative pairs are introduced to improve the matching accuracy during retrieval.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of image-text matching, and specifically to a multi-granularity image-text matching method and system based on deep fusion. Background Art

[0002] With the explosive growth of multimodal data uploaded on social media every day, cross-modal retrieval (CMR) is becoming an emerging need for people to find relevant images and texts conveniently. CMR is a fundamental task in multimodal vision and language understanding, which retrieves relevant data from one modality by querying from another modality. In recent years, image-text matching is an emerging task that matches instances of one modality with instances of another modality. This can connect vision and language, thereby improving the performance of other multimodal applications.

[0003] Image-text matching, also known as image-text retrieval, refers to matching an image with a piece of text to find the correlation between them. Specifically, for any input text-image pair, the purpose of image-text matching is to measure the semantic similarity between the image and the text. Existing image-text matching work can be roughly divided into two categories: 1) Global association: learning semantic associations with the entire text and image as objects; 2) Local association: learning semantic associations with fine-grained image salient regions and text words as objects. One-to-one matching methods usually extract global feature representations of images and texts respectively, and then use the objective function of structured or canonical correlation analysis to project their features into a common space, so that similar pairs of image texts are close in distance in space, that is, they have high similarity. After the multi-granularity method X-VLM, which considers both at the same time, was proposed, due to its amazing effect, multi-granularity-based methods gradually became mainstream.

[0004] The main idea of ​​the multi-granularity based image-text matching method is to locate the visual concept in the image given the relevant text, and to align the text with the visual concept in a multi-granularity manner. Summary of the invention

[0005] This application aims to address the deficiencies of the prior art and proposes a multi-granularity image-text matching method and system based on deep fusion, which utilizes intra-modal and inter-modal features to train different encoders to avoid mutual interference between the two, thereby effectively learning the feature representation of each modality while reducing the requirements for computing resources.

[0006] To achieve the above objectives, this application discloses the following solutions:

[0007] A multi-granularity image-text matching method based on deep fusion, the steps comprising:

[0008] Perform multi-granularity visual encoding on the image to be matched to obtain an encoded image;

[0009] Perform multi-granularity visual encoding on the text to be matched to obtain the encoded text;

[0010] Train an encoder based on the encoded image and the encoded text to obtain an initial encoder;

[0011] The initial encoder features are deeply fused and encoded to obtain a cross-modal encoder;

[0012] The initial encoder and the cross-modal encoder are trained and jointly obtained to obtain a final model, and the final model is used to complete the matching of the image and the text.

[0013] Preferably, the method for obtaining the encoded image includes: dividing the image to be matched into non-overlapping patches, and linearly embedding all the patches to obtain a number of patches; passing the patches to a pre-trained image encoder for multi-granularity visual encoding; creating N+1 conceptual representations at different granularities to obtain the encoded image.

[0014] Preferably, the method for performing the multi-granularity visual encoding includes: using a special mark [CLS] to separate a single sentence and a pair of sentences in the input text to complete the multi-granularity visual encoding.

[0015] Preferably, the method for obtaining the initial encoder includes: setting a pair (V, T), wherein T represents a positive example of V, and treating other (N-1) texts as negative examples; defining cosine similarity:

[0016] s(V, T) = g v (v cls )Tg w (w cls )

[0017] In the formula, w cls represents the output of the text encoder [CLS] embedding; g v and g w is a transformation that maps the [CLS] embedding to a normalized low-dimensional representation.

[0018] Preferably, the visual text similarity processed in batches includes:

[0019]

[0020] Similarities between text and vision include:

[0021]

[0022] Where τ is a learnable temperature parameter; yi2t and t2i represents the ground truth one-hot similarity, and the contrastive loss is defined as the cross entropy H between p and y:

[0023]

[0024] Where, L c1 is the contrast loss score, E V,T~D Indicates expected value.

[0025] Preferably, training the initial encoder and the cross-modal encoder comprises:

[0026] First, query the top k texts corresponding to each image and sort them in descending order;

[0027] Then, the retrieval results are optimized again using the trained cross-modal encoder;

[0028] Finally, the similarity within and between modes is expressed in the form of weights, and the weight calculation method is:

[0029]

[0030] Among them, top k _i2t_sim is the similarity of the top k most similar texts corresponding to the i-th image in the similarity matrix, top k _i2t_sim j for top k The similarity score between the i-th image and the j-th text in _i2t_sim.

[0031] Preferably, the method for querying the first k texts corresponding to each image is:

[0032] top k _i2t_sim, top k _i2t_idx=top k (p i2t )

[0033] The method to query the top k images corresponding to each text is:

[0034] top k _t2i_sim, top k _t2i_idx=top k (p t2i ).

[0035] Preferably, the text feature most similar to the i-th image is generated as:

[0036]

[0037] The image feature most similar to the i-th text is:

[0038]

[0039] Preferably, the cross-modal encoder backbone is a multi-way transformer, each of which consists of a shared self-attention module and a feed-forward network pool for different modalities; each input token is routed to an expert according to the way the input token is received, and each layer contains a visual expert and a language expert; in addition, the first three layers have visual-language experts designed for the fusion encoder, and the modality experts are used to encourage the model to capture modality-specific information; and the shared self-attention module is used to learn the alignment between different modes.

[0040] The present application also provides a multi-granularity image-text matching system based on deep fusion, comprising: an image encoding module, a text encoding module, a training module, a fusion module and a matching module;

[0041] The image encoding module is used to perform multi-granularity visual encoding on the image to be matched to obtain an encoded image;

[0042] The text encoding module is used to perform multi-granularity visual encoding on the text to be matched to obtain the encoded text;

[0043] The training module trains an encoder based on the encoded image and the encoded text to obtain an initial encoder;

[0044] The fusion module is used to perform deep fusion on the initial encoder to obtain a cross-modal encoder;

[0045] The matching module trains the initial encoder and the cross-modal encoder, and jointly obtains a final model, and uses the final model to complete the matching of the image and the text.

[0046] Compared with the prior art, the present invention has the following beneficial effects:

[0047] Different encoders are trained using intra-modal and inter-modal features to avoid mutual interference between the two, thereby effectively learning the feature representation of each modality and reducing the requirements for computing resources. Secondly, a deep fusion method is used to more fully learn the complex associations between different modalities. Finally, in the training process of multi-granular visual (text) encoders, hard negative pairs are introduced to improve the matching accuracy during retrieval. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] In order to more clearly illustrate the technical solution of the present application, the following briefly introduces the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0049] Figure 1 This is a method flow chart of an embodiment of the present application;

[0050] Figure 2 A schematic diagram of image encoding according to an embodiment of the present application;

[0051] Figure 3 A schematic diagram of text encoding according to an embodiment of the present application;

[0052] Figure 4 Schematic diagram of the system structure of an embodiment of the present application. DETAILED DESCRIPTION

[0053] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0054] In order to make the above-mentioned objects, features and advantages of the present application more obvious and easy to understand, the present application is further described in detail below in conjunction with the accompanying drawings and specific implementation methods.

[0055] Embodiment 1

[0056] like Figure 1 The figure shows a method flow framework diagram of an embodiment of the present application.

[0057] First, the image to be matched is subjected to multi-granularity visual encoding to obtain the encoded image.

[0058] For each input image, we first split an image into non-overlapping patches and linearly embed all patches to get several patches. Then, these patches are passed to the transformer layer to produce {v1, ..., v NI For an image with a resolution of 224×224 and a patch size of 32×32, NI=49 in this embodiment.

[0059] Assume v pi It encodes the information of the corresponding patch pi. A visual concept V is represented by a set of patches corresponding to the information between patches. j(object, region or image). The patch features are reshaped while retaining the location information of the patch features, expressed as Yes V j The average of the features is calculated to represent the entire visual concept, expressed as Create N+1 concept representations at different granularities, represented as j∈[0,N]. In this embodiment, let Itrans(V 0 ) indicates that all patch features are utilized. Figure 2 shown.

[0060] Afterwards, the text to be matched is subjected to multi-granularity visual encoding to obtain the encoded text.

[0061] The input text represents a single sentence or a pair of sentences in a sequence of tokens. A "sentence" can be an arbitrary span of continuous text, rather than an actual language sentence. A "sequence" refers to the sequence of input tokens to BERT, which can be a single sentence or two sentences packaged together.

[0062] The first token of each sequence is always a special classification token ([CLS]). The final hidden state corresponding to this token is used as the aggregate sequence representation for the classification task. Sentence pairs are packed into a single sequence. Figure 3 shown.

[0063] Based on the above steps, the encoder is trained based on the encoded image and the encoded text to obtain the initial encoder (multi-granularity visual / text encoder). The specific training process includes:

[0064] Predict (visual, textual) pairs, representing (V, T). In this embodiment, visual concepts include objects, regions, and images. Randomly extract a small batch of N pairs and calculate batch visual-to-textual similarity and text-to-visual similarity.

[0065] Suppose a pair (V, T), where T represents a positive example of V, and the other (N-1) texts are considered negative examples; define cosine similarity:

[0066] s(V, T) = g v (v cls )Tg w (w cls )

[0067] In the formula, w cls represents the output of the text encoder [CLS] embedding; g v and g wis a transformation that maps the [CLS] embedding to a normalized low-dimensional representation. Then, the visual similarity of the batched text is calculated:

[0068]

[0069] Similarly, the similarity between text and vision is:

[0070]

[0071] i∈[0,N].

[0072] Where τ is a learnable temperature parameter. i2t and t2i represents the ground truth one-hot similarity, where only positive pairs have probability 1. The contrastive loss is defined as the cross entropy H between p and y:

[0073]

[0074] Where, L c1 is the score of the contrast loss, E V,T~D Express expectations.

[0075] Afterwards, the initial encoder is trained:

[0076] 1) Matching training

[0077] Determine whether a pair of visual concepts and text matches. For each visual concept in the mini-batch, this embodiment calculates the matching value according to p v2t (V) Sample the hard negative texts in the batch. Texts that are more relevant to the concept are more likely to be sampled. This embodiment also samples a hard negative visual concept for each text. This embodiment uses x cls , that is, the output of the cross-modal encoder [CLS] is embedded to predict the matching probability p match , the loss is:

[0078]

[0079] where y match is a two-dimensional one-hot vector.

[0080] 2) Hard negative training

[0081] Given a matching image-text pair (V, T), the hard negative ranking loss only considers the nearest negative pair (T~, V~), and the similarity of positive pairs in a mini-batch d should be higher than that of negative pairs by a fixed margin γ, defined as:

[0082]

[0083] Where S(·,·) represents the calculated image-text pair matching score, γ is the marginal parameter, [x] + ≡max(x,0).

[0084] Finally, the overall visual / text encoder training objective is:

[0085]

[0086] In addition, based on the above steps, the initial encoder features are deeply fused to obtain a cross-modal encoder.

[0087] The backbone is a multi-way transformer. Each multi-way transformer consists of a shared self-attention module and a pool of feed-forward networks for different modalities (i.e., modality experts). Each input token is routed to an expert based on how it is received. Each layer contains a vision expert and a language expert. In addition, the first three layers have vision-language experts designed for the fusion encoder. Using a pool of modality experts encourages the model to capture more modality-specific information. The shared self-attention module learns alignment between different modalities.

[0088] This embodiment is trained by unified masked data modeling on unimodal data (i.e., image and text) and multimodal data (i.e., image-text pairs). During training, this embodiment randomly masks a certain percentage of text tags or image patches and trains the model to recover the masked tags. The unified mask-then predicts the alignment of different modes. The final model is obtained.

[0089] Finally, the initial encoder and the cross-modal encoder are trained to obtain the final model, which is used to complete the matching of images and texts.

[0090] Among them, training the initial encoder and the cross-modal encoder includes:

[0091] First, query the top k texts corresponding to each image and sort them in descending order;

[0092] Then, the retrieval results are optimized again using the trained cross-modal encoder;

[0093] Finally, in order to utilize the similarity representations within and between modalities, we combine them with weights, which are calculated as follows:

[0094]

[0095] Among them, top k _i2t_sim is the similarity of the top k most similar texts corresponding to the i-th image in the similarity matrix, top k _i2t_sim j for topk The similarity score between the i-th image and the j-th text in _i2t_sim.

[0096] Furthermore, the method to query the top k texts corresponding to each image is: top k _i2t_sim, top k _i2t_idx=top k (p i2t );

[0097] The method to query the top k images corresponding to each text is:

[0098] top k _t2i_sim, top k _t2i_idx=top k (p t2i );

[0099] Furthermore, the text feature most similar to the i-th image is generated as:

[0100]

[0101] Similarly, the image feature most similar to the i-th text is:

[0102]

[0103] This embodiment randomly masks 15% of unimodal text tokens and 50% of text tokens from image-text pairs. For images, this embodiment masks 40% of image patches using a block-level masking strategy.

[0104] Embodiment 2

[0105] like Figure 4 As shown, it is a schematic diagram of the system structure of this embodiment, including: an image encoding module, a text encoding module, a training module, a fusion module and a matching module; the image encoding module is used to perform multi-granularity visual encoding on the image to be matched to obtain an encoded image; the text encoding module is used to perform multi-granularity visual encoding on the text to be matched to obtain an encoded text; the training module trains the encoder based on the encoded image and the encoded text to obtain an initial encoder; the fusion module performs deep fusion on the initial encoder to obtain a cross-modal encoder; the matching module trains the cross-modal encoder to obtain a final model, and uses the final model to complete the matching of image and text.

[0106] The following will explain in detail how the present application solves technical problems in real life in conjunction with this embodiment.

[0107] Firstly, the image to be matched is visually encoded at multiple granularities using the image encoding module to obtain the encoded image.

[0108] For each input image, we first split an image into non-overlapping patches and linearly embed all patches to get several patches. Then, these patches are passed to the transformer layer to produce {v1, ..., v NI For an image with a resolution of 224×224 and a patch size of 32×32, NI=49 in this embodiment.

[0109] Assume v pi It encodes the information of the corresponding patch pi. A visual concept V is represented by a set of patches corresponding to the information between patches. j (object, region or image). The patch features are reshaped while retaining the location information of the patch features, expressed as Yes V j The average of the features is calculated to represent the entire visual concept, expressed as Create N+1 concept representations at different granularities, represented as j∈[0,N]. In this embodiment, let Itrans(V 0 ) indicates that all patch features are utilized. Figure 2 shown.

[0110] Afterwards, the text encoding module performs multi-granularity visual encoding on the text to be matched to obtain the encoded text.

[0111] The input text represents a single sentence or a pair of sentences in a sequence of tokens. A "sentence" can be an arbitrary span of continuous text, rather than an actual language sentence. A "sequence" refers to the sequence of input tokens to BERT, which can be a single sentence or two sentences packaged together.

[0112] The first token of each sequence is always a special classification token ([CLS]). The final hidden state corresponding to this token is used as the aggregate sequence representation for the classification task. Sentence pairs are packaged into a single sequence. In this example, two methods are used to distinguish these sentences: Figure 3 shown.

[0113] Based on the above process, the training module trains the encoder based on the encoded image and the encoded text to obtain the initial encoder (multi-granularity visual / text encoder). The specific training process includes:

[0114] Predict (visual, textual) pairs, representing (V, T). In this embodiment, visual concepts include objects, regions, and images. Randomly extract a small batch of N pairs and calculate batch visual-to-textual similarity and text-to-visual similarity.

[0115] Suppose a pair (V, T), where T represents a positive example of V, and the other (N-1) texts are considered negative examples; define cosine similarity:

[0116] s(V, T) = g v (v cls )Tg w (w cls )

[0117] In the formula, w cls represents the output of the text encoder [CLS] embedding; g v and g w is a transformation that maps the [CLS] embedding to a normalized low-dimensional representation. Then, the visual similarity of the batched text is calculated:

[0118]

[0119] Similarly, the similarity between text and vision is:

[0120]

[0121] i∈[0,N].

[0122] Where τ is a learnable temperature parameter. i2t and t2i represents the ground truth one-hot similarity, where only positive pairs have probability 1. The contrastive loss is defined as the cross entropy H between p and y:

[0123]

[0124] Afterwards, the initial encoder is trained:

[0125] 1) Matching training

[0126] Determine whether a pair of visual concepts and texts matches. For each visual concept in the mini-batch, this embodiment samples the hard negative texts in the batch according to p v2t(V). Texts that are more relevant to the concept are more likely to be sampled. This embodiment also samples a hard negative visual concept for each text. This embodiment uses X c1s , that is, the output of the cross-modal encoder [CLS] is embedded to predict the matching probability p match , the loss is:

[0127]

[0128] where y match is a two-dimensional one-hot vector.

[0129] 2) Hard negative training

[0130] Given a matching image-text pair (V, T), the hard negative ranking loss only considers the closest negative pair (T ~ , V ~ ), the similarity of positive pairs in a mini-batch d should be higher than that of negative pairs by a fixed margin γ, defined as:

[0131]

[0132] Where S(·,·) represents the calculated image-text pair matching score, γ is the marginal parameter, [x] + ≡max(x,0).

[0133] Finally, the overall visual / text encoder training objective is:

[0134]

[0135] In addition, based on the above process, the fusion module deeply fuses the initial encoder to obtain a cross-modal encoder.

[0136] The backbone is a multi-way transformer. Each multi-way transformer consists of a shared self-attention module and a pool of feed-forward networks for different modalities (i.e., modality experts). Each input token is routed to an expert based on how it is received. Each layer contains a vision expert and a language expert. In addition, the first three layers have vision-language experts designed for the fusion encoder. Using a pool of modality experts encourages the model to capture more modality-specific information. The shared self-attention module learns alignment between different modalities.

[0137] This embodiment is trained by unified masked data modeling on unimodal data (i.e., image and text) and multimodal data (i.e., image-text pairs). During training, this embodiment randomly masks a certain percentage of text tags or image patches and trains the model to recover the masked tags. The unified mask-then predicts the alignment of different modes. The final model is obtained.

[0138] Finally, the training module trains the initial encoder and the cross-modal encoder to obtain the final model, and uses the final model to complete the matching of images and texts.

[0139] Among them, training the initial encoder and the cross-modal encoder includes:

[0140] First, query the top k texts corresponding to each image and sort them in descending order;

[0141] Then, the retrieval results are optimized again using the trained cross-modal encoder;

[0142] Finally, in order to utilize the similarity representations within and between modalities, we combine them with weights, which are calculated as follows:

[0143]

[0144] Among them, top k _i2t_sim is the similarity of the top k most similar texts corresponding to the i-th image in the similarity matrix, top k _i2t_sim j for top k The similarity score between the i-th image and the j-th text in _i2t_sim.

[0145] Furthermore, the method for querying the top k texts corresponding to each image is:

[0146] top k _i2t_sim, top k _i2t_idx=top k (p i2t );

[0147] The method to query the top k images corresponding to each text is:

[0148] top k _t2i_sim, top k _t2i_idx=top k (p t2i );

[0149] Furthermore, the text feature most similar to the i-th image is generated as:

[0150]

[0151] Similarly, the image feature most similar to the i-th text is:

[0152]

[0153] This embodiment randomly masks 15% of unimodal text tokens and 50% of text tokens from image-text pairs. For images, this embodiment masks 40% of image patches using a block-level masking strategy.

[0154] The embodiments described above are only descriptions of the preferred methods of the present application, and are not intended to limit the scope of the present application. Without departing from the design spirit of the present application, various modifications and improvements made to the technical solutions of the present application by ordinary technicians in this field should fall within the scope of protection determined by the claims of the present application.

Claims

1. A multi-granularity image-text matching method based on deep fusion, characterized in that: The steps include: Perform multi-granularity visual encoding on the image to be matched to obtain an encoded image; The method for obtaining the encoded image includes: dividing the image to be matched into non-overlapping patches, and linearly embedding all the patches to obtain a plurality of patches; passing the patches to a pre-trained image encoder for multi-granularity visual encoding; creating N+1 concept representations at different granularities to obtain the encoded image; Performing multi-granular visual coding on the text to be matched to obtain coded text; the method for performing the multi-granular visual coding includes: using a special mark [CLS] to separate a single sentence and a pair of sentences in the input text to complete the multi-granular visual coding; An encoder is trained based on the encoded image and the encoded text to obtain an initial encoder; the method for obtaining the initial encoder includes: during the encoder training process, a hard negative pair is introduced, specifically, a pair (V, T) is set, wherein T represents a positive example of V, and other (N-1) texts are regarded as negative examples; and cosine similarity is defined as: s(V,T)=g v (v cls ) T g w (w cls ); In the formula, w cls represents the output of the text encoder [CLS] embedding; g v and g w is a transformation that maps [CLS] embeddings to a normalized low-dimensional representation; batched visual-textual similarity includes: Similarities between text and vision include: In the formula, Is a learnable temperature parameter; first define the cross entropy between the contrast loss p and y as H, and then calculate the contrast loss score: Where, L c1 is the contrast loss score, E V,T~D Indicates expected value; The initial encoder features are deeply fused and encoded to obtain a cross-modal encoder; the backbone of the cross-modal encoder is a multi-way transformer, each of which consists of a shared self-attention module and a feed-forward network pool for different modalities; each input token is routed to an expert according to the way the token is input, and each layer contains a visual expert and a language expert; in addition, the first three layers have visual-language experts designed for the fusion encoder, and the modality experts are used to encourage the model to capture modality-specific information; the shared self-attention module is used to learn the alignment between different modes; The initial encoder and the cross-modal encoder are trained and jointly obtained to obtain a final model, and the final model is used to complete the matching of the image and the text; the initial encoder and the cross-modal encoder are trained including: First, query the first k texts corresponding to each image and sort them in descending order; the method for querying the first k texts corresponding to each image is: top k _v2t_sim,top k _v2t_idx=top k (p v2t ); The method to query the top k images corresponding to each text is: top k _t2v_sim,top k _t2v_idx=top k (p t2v ); Then, the retrieval results are optimized again using the trained cross-modal encoder; Finally, the similarity within and between modes is expressed in the form of weights, and the weight calculation method is: The text features most similar to the cth image are generated as: The image features most similar to the cth text are:

2. A multi-granularity image-text matching system based on deep fusion, characterized in that: include: Image encoding module, text encoding module, training module, fusion module and matching module; The image encoding module is used to perform multi-granularity visual encoding on the image to be matched to obtain an encoded image; The process of obtaining the encoded image includes: dividing the image to be matched into non-overlapping patches, and linearly embedding all the patches to obtain a plurality of patches; passing the patches to a pre-trained image encoder for multi-granularity visual encoding; creating N+1 concept representations at different granularities to obtain the encoded image; The text encoding module is used to perform multi-granular visual encoding on the text to be matched to obtain the encoded text; the method for performing the multi-granular visual encoding includes: using a special mark [CLS] to separate a single sentence and a pair of sentences in the input text to complete the multi-granular visual encoding; The training module trains an encoder based on the encoded image and the encoded text to obtain an initial encoder; the process of obtaining the initial encoder includes: introducing a hard negative pair during the encoder training process, specifically, setting a pair (V, T), where T represents a positive example of V, and other (N-1) texts are regarded as negative examples; defining cosine similarity: s(V,T)=g v (v cls ) T g w (w cls ); In the formula, w cls represents the output of the text encoder [CLS] embedding; g v and g w is a transformation that maps [CLS] embeddings to a normalized low-dimensional representation; batched visual-textual similarity includes: Similarities between text and vision include: In the formula, Is a learnable temperature parameter; first define the cross entropy between the contrast loss p and y as H, and then calculate the contrast loss score: Where, L c1 is the contrast loss score, E v,T~D Indicates expected value; The fusion module is used to perform deep fusion on the initial encoder to obtain a cross-modal encoder; the backbone of the cross-modal encoder is a multi-way transformer, each of which consists of a shared self-attention module and a feed-forward network pool for different modalities; each input token is routed to an expert according to the way the input token is input, and each layer contains a visual expert and a language expert; in addition, the first three layers have visual-language experts designed for the fusion encoder, and the modality experts are used to encourage the model to capture modality-specific information; the shared self-attention module is used to learn the alignment between different modes; The matching module trains the initial encoder and the cross-modal encoder, and jointly obtains a final model, and uses the final model to complete the matching of the image and the text; training the initial encoder and the cross-modal encoder includes: First, query the first k texts corresponding to each image and sort them in descending order; the method for querying the first k texts corresponding to each image is: top k _v2t_sim,top k _v2t_idx=top k (p v2t ); The method to query the top k images corresponding to each text is: top k _t2v_sim,top k _t2v_idx=top k (p t2v ); Then, the retrieval results are optimized again using the trained cross-modal encoder; Finally, the similarity within and between modes is expressed in the form of weights, and the weight calculation method is: The text features most similar to the cth image are generated as: The image features most similar to the cth text are: