A cross-modal instance retrieval method based on instance-level soft label alignment

Abstract

The application discloses a cross-modal instance retrieval method based on instance-level soft label alignment, relates to the technical field of multimedia retrieval, and comprises the following steps: S100, data set preparation; S200, feature extraction; S300, instance feature set generation; S400, soft label generation; S500, feature alignment; S600, loss function integration; S700, model training completion; and S800, cross-modal instance retrieval. The application introduces an alignment mechanism of instance-level features, improves the understanding ability of a model for local fine-grained semantics and instance information through instance discovery and soft label guidance, effectively improves the matching precision of cross-modal retrieval, expands the application range of the model in a cross-modal instance retrieval task, and further enhances the multi-scene applicability and practical value.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of multimedia retrieval, and particularly relates to a cross-modal instance retrieval method based on instance-level soft label alignment. BACKGROUND

[0002] In recent years, instance retrieval methods have made great progress, especially in instance-level feature discovery and localization. The invention patent "Class-Independent Instance Retrieval Method Based on Hierarchical Semantic Region Decomposition" (CN117725243A, Zhao Wanlei, Sun Qiyin) discloses a method based on hierarchical semantic region decomposition, which gradually refines from global to local to mine potential instances in the image, and cooperates with reasonable initialization, termination condition and irrelevant region filtering mechanism, greatly improving the comprehensiveness and stability of instance discovery.

[0003] Although instance retrieval has made significant progress on the single modal (i.e. image instance retrieval) task, with the rapid development of artificial intelligence and multi-modal learning technology, existing methods have been difficult to meet the more complex and diversified application requirements, and it is urgent to expand to the multi-modal direction. The cross-modal instance retrieval task thus emerged, and gradually became an important research direction in the cross field of visual understanding and language understanding. Cross-modal instance retrieval aims to retrieve corresponding specific visual instances from an image database according to natural language description, rather than just retrieving the entire image, emphasizing fine-grained cross-modal semantic matching at the instance level. This task is of great significance to search engines, intelligent recommendation, open world recognition and other application scenarios.

[0004] At present, multi-modal related researches still mainly focus on the field of cross-modal retrieval. Cross-modal retrieval usually aligns features at the global semantic level, aiming to learn a shared feature space so that matching images and texts are closer in this space. This task mainly includes two basic forms: text retrieval image and image retrieval text. In terms of visual feature extraction, traditional methods usually use region-based extractors such as Faster R-CNN using a bottom-up attention mechanism, and in terms of text feature extraction, Bidirectional Gated Recurrent Unit (BiGRU) or Bidirectional Encoder Representations from Transformers (BERT) are commonly used to encode text sequences.

[0005] Early studies to improve the matching performance of cross-modal image-text retrieval, stacked cross attention network (SCAN), the paper “Stacked cross attention for image-text matching [C]” (Proceedings of the European Conference on Computer Vision. Lee KH, Chen X, Hua G, et al. 2018: 201-216) introduces cross attention mechanism, realizing the fine-grained association between image regions and text words; using cross-modal hard aligning network (CHAN), the paper “Fine-grained image-text matching by cross-modal hard aligning network [C]” (Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition. Pan Z, Wu F, Zhang B. 2023: 19275-19284) proposes a hard alignment strategy to strengthen the consistency of the corresponding regions of image and text; using negative-aware attention framework (NAAF), the paper “Negative-aware attention framework for image-text matching [C]” (Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition. Zhang K, Mao Z, Wang Q, et al. 2022: 15661-15670) further improves the matching accuracy by modeling positive and negative regions.

[0006] To further improve the performance of cross-modal retrieval, vision-language pretraining models (VLPs) have been widely used. Among them, the contrastive language-image pretraining (CLIP) proposed in the paper Learning transferable visual models from natural language supervision[C] (International Conference on Machine Learning. Radford A, Kim J W, Hallacy C, et al. 2021: 8748-8763) conducts contrastive learning through large-scale image-text pairs, respectively trains image encoder and text encoder, and maps different modal data into a unified feature space, thus showing excellent performance in zero-shot retrieval and open-class recognition. Based on CLIP and other models, various improvement methods have appeared in recent years, such as the local and generative-driven modality gap correction (LG-MGC) represented by the paper Overcoming the pitfalls of vision-language model for image-text retrieval[C] (Proceedings of the ACM International Conference on Multimedia. Zhang F, Qu S, Shi F, et al. 2024: 2350 2359), and the cross-modal and uni-modal soft-label alignment (CUSA) represented by the paper Cross-modal and uni-modal soft-label alignment for image-text retrieval[C] (Proceedings of the AAAI Conference on Artificial Intelligence: volume 38. Huang H, Nie Z, Wang Z, et al. 2024: 18298-18306). These methods further optimize the cross-modal feature alignment by introducing local feature modeling and soft label assistance.

[0007] In recent years, the related research on cross-modal instance retrieval has gradually attracted widespread attention in the academic community. Researchers have realized that matching images and texts only at the global level has been difficult to meet the demand for instance-level understanding and positioning in practical applications, and fine-grained cross-modal semantic alignment has become a new research focus. To support systematic research on this emerging task, benchmarks specifically for instance-level cross-modal retrieval have also been built, among which the most representative is ILIAS (Instance-level Image and Language Alignment Search). The ILIAS dataset is annotated based on instances, emphasizing the precise correspondence between specific targets in images and natural language descriptions, reflecting the high attention to fine-grained modeling and evaluation system construction in the current cross-modal instance retrieval field.

[0008] The existing cross-modal retrieval technology, especially the method based on visual-linguistic pre-training model (such as CLIP) fine-tuning, has the following main shortcomings:

[0009] 1. Only global feature alignment is performed, and there is a lack of instance-level fine-grained alignment. Existing methods usually perform cross-modal matching through global image feature and text feature contrast learning, but lack fine-grained modeling of specific instances and sub-regions in images, resulting in poor performance and insufficient retrieval accuracy when precise matching of specific instances (such as object local, detailed description) is required.

[0010] 2. The soft label guidance strategy does not target instance features, making it difficult to improve local semantic matching capability. Existing methods such as CUSA use single-modal soft labels to assist global alignment, but do not guide the fine-grained instance features generated by the instance discovery module, resulting in coarse soft label granularity and ineffective improvement of instance-level alignment effect.

[0011] 3. Relies on external detectors or pseudo labels, with limited generalization and openness. Some detector-based methods require pre-trained detectors to assist in extracting instance features, which is limited to existing categories and difficult to adapt to diverse instances in the open world.

[0012] 4. In the task of cross-modal instance retrieval, there is a problem of declining retrieval accuracy. Most existing cross-modal retrieval methods are designed for image-text global matching, and when migrated to instance-level retrieval, the lack of local and instance feature modeling leads to a decline in mAP (mean Average Precision) and other retrieval indicators, failing to meet the high requirements for fine-grained matching in practical applications.

[0013] Therefore, the skilled in the art is committed to developing a cross-modal instance retrieval method based on instance-level soft label alignment. SUMMARY

[0014] In view of the above defects of the prior art, the technical problem to be solved by the present application is that the instance-level feature alignment is insufficient and the local semantic matching capability is weak in the cross-modal instance retrieval method.

[0015] In order to improve the cross-modal matching accuracy at the instance level and enhance the matching capability between visual features and text features, the applicant divides the cross-modal instance retrieval method into a model training stage and a model inference stage, which correspond to two links of feature learning and retrieval inference respectively, and effectively improves the accuracy and detail matching capability of cross-modal retrieval by performing fine-grained alignment at the instance granularity.

[0016] In one embodiment of the present application, a cross-modal instance retrieval method based on instance-level soft label alignment is provided, comprising the following steps:

[0017] S100, data set preparation, performing public image-text retrieval, collecting image-text pair data sets, and constructing a cross-modal instance retrieval model based on instance-level soft label alignment;

[0018] S200, feature extraction, inputting the image-text pair data set into the above cross-modal instance retrieval model to extract image global features and text global features ;

[0019] S300, instance feature set generation, performing instance discovery on the image-text pair data set, discovering semantic candidate instances and generating an instance feature set;

[0020] S400, soft label generation, generating an instance-level semantic distribution of the image, and calculating a normalized semantic probability distribution as a soft label;

[0021] S500, feature alignment, constructing a soft label guided instance-level feature alignment loss function, performing instance-level feature alignment, constructing an image-text global feature alignment loss function, and performing global feature alignment;

[0022] S600, loss function integration, combining the instance-level feature alignment loss function and the image-text global feature alignment loss function according to a weight hyperparameter to form a joint loss function;

[0023] S700, model training is completed, the above cross-modal instance retrieval model is trained until the convergence condition of the joint loss function is met, a trained cross-modal instance retrieval model is obtained, otherwise the model parameters are adjusted and the learning rate is returned to step S200;

[0024] S800, cross-modal instance retrieval, using the trained cross-modal instance retrieval model to perform cross-modal instance retrieval on an image set, and obtaining an instance-level retrieval result ranking list.

[0025] Further, in the cross-modal instance retrieval method based on instance-level soft label alignment in the above embodiment, the image-text pair dataset includes image data and corresponding text data.

[0026] Further, in the cross-modal instance retrieval method based on instance-level soft label alignment in the above embodiment, the image data is a natural scene image containing multiple potential instances.

[0027] Further, in the cross-modal instance retrieval method based on instance-level soft label alignment in the above embodiment, the text data is a natural language description of the target instance.

[0028] Optionally, in the cross-modal instance retrieval method based on instance-level soft label alignment in any of the above embodiments, the cross-modal instance retrieval model based on instance-level soft label alignment is constructed based on a contrastive language-image pretraining (CLIP) model.

[0029] Optionally, in the cross-modal instance retrieval method based on instance-level soft label alignment in any of the above embodiments, the contrastive language-image pretraining (CLIP) model includes an image encoder and a text encoder, the image encoder adopts a Vision Transformer (ViT) architecture, the text encoder adopts a Transformer architecture, and an instance discovery module is set in an intermediate feature layer of an output of the image encoder.

[0030] Further, in the cross-modal instance retrieval method based on instance-level soft label alignment in the above embodiment, the instance discovery module divides the image into semantically consistent instance candidate regions and extracts local visual features of the instance candidate regions by using a Token-based hierarchical clustering method based on visual Transformer in an unsupervised manner.

[0031] Further, in the cross-modal instance retrieval method based on instance-level soft label alignment in the above embodiment, the image encoder includes an image embedding layer, l a Transformer layer, and an image linear projection layer, the calculation formula of the first i layer is as follows:

[0032] ;

[0033] wherein, MultiHead(·) represents a multi-head self-attention mechanism, MLP(·) is a multi-layer perceptron, LayerNorm(·) is a layer normalization operation, and the output of the image embedding layer isl After processing by the Transformer layer, the output image features are obtained. It contains global information and local information, that is ),in Represents global visual features. For the first i Local features of an image patch i From 1 to l Integers; global visual features Global image features are obtained by linear projection mapping onto the image-text embedding space. :

[0034] ;

[0035] in, For linear projection, global features of the image It contains global visual information of the image and serves as the final representation of the image for subsequent cross-modal feature alignment.

[0036] Optionally, in the cross-modal instance retrieval method based on instance-level soft tag alignment in the above embodiments, l It can be 12 or 24.

[0037] Preferably, in the cross-modal instance retrieval method based on instance-level soft tag alignment in the above embodiments, l It is 12.

[0038] Furthermore, in the cross-modal instance retrieval method based on instance-level soft tag alignment in the above embodiments, the text encoder includes a text embedding layer, m A Transformer layer and a text linear projection layer are used to pass through the text block sequence. m The global semantic features are obtained by processing the data through the Transformer layer. Global semantic features Global text features are obtained by mapping linearly to the image-text embedding space. .

[0039] Optionally, in the cross-modal instance retrieval method based on instance-level soft tag alignment in the above embodiments, m It can be 8, 12, or 16.

[0040] Preferably, in the cross-modal instance retrieval method based on instance-level soft tag alignment in the above embodiments, m It is 12.

[0041] Optionally, in the cross-modal instance retrieval method based on instance-level soft tag alignment in any of the above embodiments, step S200 includes:

[0042] S210, Image block division: The image encoder divides the input image data into multiple image blocks;

[0043] S220, Image patch sequence generation: Image patches pass through a linear projection layer, and together with category labels used to enhance the relationship between image patches, location encoding is added to embed spatial information to generate an image patch sequence;

[0044] S230, Image patch sequence processing: Processing the image patch sequence using a self-attention mechanism and a multilayer perceptron.

[0045] S240. Global Image Feature Extraction: Global visual features are mapped to the image-text embedding space through linear projection to extract global image features. ;

[0046] S250, Lexical Unit Decomposition: The text encoder decomposes text data into a series of lexical units to generate a text sequence.

[0047] S260. Text block sequence generation: Add start and end markers to the text sequence to generate a text block sequence after text embedding;

[0048] S270. Global semantic feature generation: The text block sequence is processed by Transformer to obtain global semantic features. Summarize the global semantic information of the text data;

[0049] S280. Global text feature extraction, global semantic features. Global text features are extracted by mapping to the image-text embedding space using linear projection. .

[0050] Furthermore, in the cross-modal instance retrieval method based on instance-level soft label alignment in the above embodiments, the instance feature set includes... K Instance-level feature descriptors , Indicates the first K The feature vector of each instance K It is the number of instance-level feature descriptors in the image. D For feature dimension, K and D It is a positive integer.

[0051] Preferably, in the cross-modal instance retrieval method based on instance-level soft tag alignment in the above embodiments, D =512.

[0052] Optionally, in the cross-modal instance retrieval method based on instance-level soft label alignment in any of the above embodiments, step S400 comprises:

[0053] S410, instance feature set average pooling, using a soft label generation module, performing average pooling on the instance feature set to generate an instance-level semantic distribution of the image;

[0054] S420, soft label generation, calculating a normalized semantic probability distribution as a soft label to describe the association degree of the candidate instance and the potential semantic category in the form of continuous probability distribution.

[0055] Optionally, in the cross-modal instance retrieval method based on instance-level soft label alignment in any of the above embodiments, the soft label generation module comprises:

[0056] An image encoding and image block generation unit encodes the image using an unsupervised trained feature encoder to generate an image block feature set of the image;

[0057] A hierarchical semantic region decomposition unit adopts a fast dichotomy clustering algorithm to iteratively decompose the image block feature set to construct a multi-level instance candidate region; specifically, the first decomposition takes the image block feature set as the first layer, and adopts the fast dichotomy clustering algorithm to decompose the image block feature set of the first layer into two image block feature sub-sets to obtain a second layer image block feature sub-set pair, and continues to adopt the fast dichotomy clustering algorithm to decompose the image block feature sub-sets in the second layer image block feature sub-set pair into two image block feature sub-sets respectively to obtain a third layer image block feature sub-set pair, and so on, to construct a multi-level instance candidate region; the termination condition of the decomposition iteration is that the similarity of each layer image block feature sub-set pair is calculated, and when the number of image block feature sub-sets with a similarity greater than 0.2 is greater than the number of all image block feature sub-sets*0.97, the decomposition iteration is terminated; each image block set in the structure can be regarded as a potential instance;

[0058] An instance screening unit introduces a saliency measurement mechanism to screen out dummy nodes with feature energy overlap rate as the criterion; specifically, the L1 norm of the image blocks in the image block feature set is calculated and sorted from large to small, and the top 30% of the image blocks are defined as energy salient image blocks; if the overlap rate of the image blocks in the current instance and the energy salient image blocks is less than 20%, the semantic is not significant, i.e. a dummy node, which is removed to obtain a candidate instance region;

[0059] An instance feature description unit, i.e. a descriptor, extracts image region features from the screened candidate instance region, and generates an instance-level feature expression through instance region extraction after cropping .

[0060] Optionally, in the cross-modal instance retrieval method based on instance-level soft label alignment in any of the above embodiments, the average pooling formula is as follows:

[0061] ;

[0062] wherein, is the average-pooled instance-level feature descriptor vector, K is the number of instance-level feature descriptors in the image, denotes the feature vector of the i-th instance. K

[0063] Optionally, in the cross-modal instance retrieval method based on instance-level soft label alignment in any of the above embodiments, the normalization semantic probability distribution formula is as follows:

[0064] ;

[0065] wherein, is the average-pooled instance-level feature descriptor vector, is the normalized soft label semantic probability distribution, is L2 normalization.

[0066] Optionally, in the cross-modal instance retrieval method based on instance-level soft label alignment in any of the above embodiments, step S500 comprises:

[0067] S510, instance-level feature alignment, aligning the image global feature with the soft label distribution, calculating the image global feature normalized generation probability distribution, calculating the KL (Kullback-Leibler) divergence, constructing the soft label guided instance-level feature alignment loss function, and performing instance-level feature alignment.

[0068] S520, global feature alignment, according to the image global feature and the text global feature , calculating the cross-modal contrast loss function, maximizing the similarity of positive sample pairs in a Batch (batch) of image-text pair samples, minimizing the similarity of negative sample pairs, constructing the image-text global-level feature alignment loss function, and performing global feature alignment. In the image-text pair sample, the correctly corresponding image-text pair is the positive sample pair, and the incorrectly corresponding image-text pair is the negative sample pair.

[0069] Optionally, in the cross-modal instance retrieval method based on instance-level soft label alignment in any of the above embodiments, the image global feature normalized generation probability distribution formula is as follows:

[0070] ;

[0071] ​in, It is a probability distribution generated by normalizing global image features. It is a global feature of the image. It is L2 normalization.

[0072] Optionally, in the cross-modal instance retrieval method based on instance-level soft label alignment in any of the above embodiments, the soft label-guided instance-level feature alignment loss function The formula is as follows:

[0073] ;

[0074] in, It is a probability distribution generated by normalizing global image features. D For feature dimension, It is the normalized semantic probability distribution of soft tags. Represents the symbol for calculating KL divergence. It is the global feature number 1 i The probability of each feature dimension Soft tag semantics i The probability of each feature dimension.

[0075] Optionally, in the cross-modal instance retrieval method based on instance-level soft label alignment in any of the above embodiments, the image-text global feature alignment loss function... The formula is as follows:

[0076] ;

[0077] in, and Let represent the loss function for image to text and the loss function for text to image, respectively. The formulas are as follows:

[0078] ;

[0079] ;

[0080] in, N The number of image-text pairs is τ, where τ is the temperature hyperparameter and contrast loss is τ. This can maximize the similarity of positive sample pairs and minimize the similarity of negative sample pairs, thereby enhancing the model's cross-modal discrimination ability. N It is the number of image-text pairs. It is the global image feature of the i-th sample. It is the text feature of the i-th sample.

[0081] Furthermore, in the cross-modal instance retrieval method based on instance-level soft tag alignment in the above embodiments, the joint loss function formula is as follows:

[0082] ,

[0083] wherein, is an empirical weight.

[0084] Preferably, in the cross-modal instance retrieval method based on instance-level soft label alignment in the above embodiment, .

[0085] Optionally, in the cross-modal instance retrieval method based on instance-level soft label alignment in any of the above embodiments, step S700 comprises:

[0086] S710, model training, training the cross-modal instance retrieval model based on instance-level soft label alignment based on the joint loss function through back propagation until the convergence condition of the joint loss function is met, obtaining the trained cross-modal instance retrieval model based on instance-level soft label alignment, otherwise executing step S720;

[0087] S720, adjusting the model parameters and adjusting the learning rate, returning to step S200.

[0088] Further, in the cross-modal instance retrieval method based on instance-level soft label alignment in the above embodiment, the convergence condition of the joint loss function is that the training reaches a set training round number or the joint loss function does not continue to decline for two consecutive rounds.

[0089] Optionally, in the cross-modal instance retrieval method based on instance-level soft label alignment in the above embodiment, the range of the set training round number is [6, 20].

[0090] Preferably, in the cross-modal instance retrieval method based on instance-level soft label alignment in the above embodiment, the set training round number is 12.

[0091] Optionally, in the cross-modal instance retrieval method based on instance-level soft label alignment in any of the above embodiments, the adjustment of the model parameters in step S700 uses the adamw optimizer.

[0092] Further, in the cross-modal instance retrieval method based on instance-level soft label alignment in the above embodiment, the adjustment of the learning rate in step S700 uses the cosine annealing learning rate optimization method.

[0093] Optionally, in the cross-modal instance retrieval method based on instance-level soft label alignment in any of the above embodiments, step S800 comprises:

[0094] S810, instance discovery and database construction, inputting the image set into the cross-modal instance retrieval model based on instance-level soft label alignment, extracting instances for each image to obtain the image and its corresponding instances;

[0095] S820, instance-level feature generation, input the image and its corresponding instance into the image encoder, extract the semantic features of each instance region to obtain instance features, and save them;

[0096] S830, text feature extraction, in response to the input natural language query for the instance, the text encoder extracts text semantic features;

[0097] S840, cross-modal similarity calculation and sorting, according to the instance features and the text semantic features, the similarity between the text semantic features and the instance features is calculated and sorted, and an instance-level retrieval result sorting list is output.

[0098] The application can model the fine-grained semantic relationship between instances by processing the local features output by the image encoder, generate continuous soft label guidance features for instance semantic distribution, improve the ability of local features to describe instance attributes, and adopt a combined optimization strategy in the training stage. The Kullback-Leibler divergence (KL divergence) minimizes the difference between the instance feature distribution and the soft label distribution at the instance level, and simultaneously optimizes the alignment of image global features and text global features through cross-modal contrast loss at the global level. It can simultaneously consider local fine-grained semantic modeling and global semantic association, significantly improve the precision and robustness of cross-modal matching, and retain the instance discovery mechanism in the training stage in the inference stage. Through unsupervised instance division, the features of potential instance regions are extracted, and the similarity calculation is performed with the features of the input query text, realizing instance-based cross-modal retrieval. Not only does it retain the efficiency of traditional global feature matching, but it also introduces fine-grained instance information, improving the retrieval accuracy under complex detailed description, and can balance local and global information, having a wider application potential in instance retrieval and detail matching tasks.

[0099] The application is based on large-scale image-text alignment training of vision-language pretraining models (VLPs), uses the cross-modal corresponding relationship implied in massive data, significantly enhances the generalization ability and open-world adaptability of the model, and overcomes the problem that the retrieval performance is limited due to the dependence of traditional methods on limited labeled data. Compared with methods based on other vision-language pretraining models, the application selects a contrastive language-image pretraining model (CLIP) with a double-stream structure as a basic model, uses independent image encoders and text encoders for feature extraction, and only realizes image-text matching through similarity calculation in the last stage, thereby reducing early complex modal interaction, having lower computational resource consumption and higher processing efficiency in the training and inference processes, and being suitable for resource-limited or large-scale retrieval application scenarios. The application innovatively introduces an instance-level feature alignment mechanism on the basis of original global feature alignment, improves the understanding ability of the model for local fine-grained semantics and instance information through instance discovery and soft label guidance, not only effectively improves the matching accuracy of cross-modal retrieval, but also expands the application range of the model in the cross-modal instance retrieval task, and further enhances the multi-scene applicability and practical value.

[0100] The concept, specific structure and technical effects of the application will be further described below with reference to the accompanying drawings, so as to fully understand the purpose, features and effects of the application. BRIEF DESCRIPTION OF DRAWINGS

[0101] Figure 1 is a flowchart of the cross-modal instance retrieval method based on instance-level soft label alignment of the embodiment. DETAILED DESCRIPTION

[0102] The technical content of the application will be more clear and easy to understand by introducing a plurality of preferred embodiments of the application with reference to the accompanying drawings of the specification. The application can be embodied in many different forms, and the protection scope of the application is not limited to the embodiments mentioned in the text.

[0103] In the drawings, the same components have the same reference numerals, and components with similar structures or functions have similar reference numerals. The size and thickness of each component shown in the drawings are arbitrarily shown, and the application does not limit the size and thickness of each component. In order to make the drawing clearer, the thickness of the components is appropriately exaggerated in some places in the drawing.

[0104] The application designs a cross-modal instance retrieval method based on instance-level soft label alignment, as shown in Figure 1 including the following steps:

[0105] S100. Dataset Preparation: For public image-text retrieval, collect an image-text pair dataset, including image data and corresponding text data. The image data consists of natural scene images containing multiple potential instances, and the text data is a natural language description of the target instance. Contrastive Language-Image Pretraining (CLIP) model is used to construct a cross-modal instance retrieval model based on instance-level soft label alignment. The CLIP model includes an image encoder and a text encoder. The image encoder uses a Vision Transformer (ViT) architecture, and the text encoder uses a Transformer architecture. An instance discovery module is set up in the intermediate feature layer of the image encoder's output. This module uses an unsupervised method, employing hierarchical clustering based on visual Transformer tokens, to divide the image into semantically consistent instance candidate regions and extract local visual features from these regions. The image encoder includes an image embedding layer... l A Transformer layer and an image linear projection layer, the first i The formula for calculating the layer is as follows:

[0106] ;

[0107] Where MultiHead(·) represents a multi-head self-attention mechanism, MLP(·) is a multilayer perceptron, and LayerNorm(·) is a layer normalization operation. l After processing by the Transformer layer, the output image features are obtained. It contains global information and local information, that is ),in Represents global visual features. For the first i Local features of an image patch i From 1 to l Integers; global visual features Global image features are obtained by linear projection mapping onto the image-text embedding space. :

[0108] ;

[0109] in, For linear projection, global features of the image It contains global visual information of the image and serves as the final representation of the image for subsequent cross-modal feature alignment. lis 12; the text encoder comprises a text embedding layer, m a Transformer layer and a text linear projection layer, the text block sequence is processed by m the Transformer layer to obtain global semantic features , m is 12, the global semantic features are mapped to an image-text embedding space by linear projection to obtain text global features .

[0110] S200, feature extraction, inputting the image data and the text data into a cross-modal instance retrieval model based on instance-level soft label alignment, extracting image global features and text global features ; specifically comprising:

[0111] S210, image block division, the image encoder divides the input image data into multiple image blocks;

[0112] S220, image block sequence generation, the image blocks are processed by the linear projection layer, together with the category labels used to enhance the relationship between the image blocks, and the position encoding embedding space information to generate an image block sequence;

[0113] S230, image block sequence processing, performing self-attention mechanism and multi-layer perceptron processing on the image block sequence;

[0114] S240, image global feature extraction, the global visual features are mapped to an image-text embedding space by linear projection to extract image global features ;

[0115] S250, vocabulary unit decomposition, the text encoder decomposes the text data into a series of vocabulary units to generate a text sequence;

[0116] S260, text block sequence generation, adding start and end markers to the text sequence, and generating a text block sequence after text embedding;

[0117] S270, global semantic feature generation, the text block sequence is processed by the Transformer to obtain global semantic features , summarizing the global semantic information of the text data;

[0118] S280, text global feature extraction, the global semantic features are mapped to an image-text embedding space by linear projection to extract text global features .

[0119] S300, Instance Feature Set Generation: This involves performing instance discovery on the image data, identifying semantically relevant candidate instances, and generating an instance feature set, including... K Instance-level feature descriptors , Indicates the first K The feature vector of each instance K It is the number of instance-level feature descriptors in the image. D For feature dimension, K and D It is a positive integer. D =512.

[0120] S400, Soft Tag Generation: Generates instance-level semantic distributions of images and calculates normalized semantic probability distributions as soft tags; specifically including:

[0121] S410. Instance feature set average pooling: Using the soft label generation module, the instance feature set is averaged to generate the instance-level semantic distribution of the image; the soft label generation module includes:

[0122] The image encoding and image patch generation unit uses an unsupervised trained feature encoder to encode the image and generate a set of image patch features.

[0123] The hierarchical semantic region decomposition unit employs a fast binary clustering algorithm to iteratively decompose the image patch feature set, constructing multi-level instance candidate regions. Specifically, the first decomposition uses the image patch feature set as the first layer, and the fast binary clustering algorithm decomposes the first-layer image patch feature set into two image patch feature subsets, obtaining the second-layer image patch feature subset pairs. The fast binary clustering algorithm is then used to further decompose the image patch feature subsets in the second-layer image patch feature subset pairs into two image patch feature subsets, obtaining the third-layer image patch feature subset pairs, and so on, constructing multi-level instance candidate regions. The termination condition for the decomposition iteration is: calculating the similarity between the two image patch feature subsets in each layer's image patch feature subset pair; the decomposition iteration terminates when the number of image patch feature subset pairs with a similarity greater than 0.2 is greater than the total number of image patch feature subset pairs multiplied by 0.97. Each image patch set in the structure can be considered as a potential instance.

[0124] The instance filtering unit introduces a saliency measurement mechanism to filter out dummy nodes based on the feature energy overlap rate. Specifically, it calculates the L1 norm of the image patches in the image patch feature set and sorts them from largest to smallest, defining the top 30% of image patches as energy saliency image patches. If the overlap rate between the image patch in the current instance and the energy saliency image patch is less than 20%, it is semantically insignificant, i.e., a dummy node. The dummy node is then removed to obtain the candidate instance region.

[0125] The instance feature description unit, or descriptor, extracts image region features from the selected candidate instance regions. By extracting the cropped instance regions, it generates instance-level feature representations. ;

[0126] The average pooling formula is as follows:

[0127] ;

[0128] in, It is the instance-level feature descriptor vector after average pooling. K It is the number of instance-level feature descriptors in the image. Indicates the first K Feature vectors of each instance;

[0129] S420. Soft label generation: Calculate the normalized semantic probability distribution as the soft label, which describes the degree of association between candidate instances and latent semantic categories in the form of a continuous probability distribution. The formula for the normalized semantic probability distribution is as follows:

[0130] ;

[0131] in, It is the instance-level feature descriptor vector after average pooling. It is the normalized semantic probability distribution of soft tags. It is L2 normalization.

[0132] S500, Feature Alignment, Aligning Global Image Features Align with the soft label distribution, calculate the normalized generation probability distribution of global image features, calculate the KL (Kullback-Leibler) divergence, construct a soft label-guided instance-level feature alignment loss function, and perform instance-level feature alignment; based on the global image features... With text global features Calculate cross-modal contrast loss, construct an image-text global feature alignment loss function, and perform global feature alignment; specifically including:

[0133] S510, instance-level feature alignment, aligns global image features Aligning with the soft label distribution, calculate the normalized generation probability distribution of global image features, as shown in the following formula:

[0134] ;

[0135] in, It is a probability distribution generated by normalizing global image features. It is a global feature of the image. It is L2 normalization;

[0136] Calculate the KL (Kullback-Leibler) divergence and construct a soft-label-guided instance-level feature alignment loss function. The formula is as follows:

[0137] ;

[0138] in, It is a probability distribution generated by normalizing global image features. D For feature dimension, It is the normalized semantic probability distribution of soft tags. Represents the symbol for calculating KL divergence. It is the global feature number 1 i The probability of each feature dimension Soft tag semantics i The probability of each feature dimension;

[0139] Perform instance-level feature alignment;

[0140] S520, Global Feature Alignment, based on global image features With text global features The cross-modal contrastive loss function is calculated by maximizing the similarity of positive image-text pairs and minimizing the similarity of negative image-text pairs within a batch of image-text pairs, thus constructing a global feature alignment loss function between the image and text. The formula is as follows:

[0141] ;

[0142] in, and Let represent the loss function for image to text and the loss function for text to image, respectively. The formulas are as follows:

[0143] ;

[0144] ;

[0145] in, N The number of image-text pairs is τ, where τ is the temperature hyperparameter and contrast loss is τ. This can maximize the similarity of positive sample pairs and minimize the similarity of negative sample pairs, thereby enhancing the model's cross-modal discrimination ability. N It is the number of image-text pairs. It is the global image feature of the i-th sample. These are the text features of the i-th sample;

[0146] Global feature alignment is performed. In the image-text pair samples, the correctly matched image-text pairs are positive sample pairs, and the incorrectly matched image-text pairs are negative sample pairs.

[0147] S600, loss function integration, combine the instance-level feature alignment loss function and the image-text global-level feature alignment loss function according to the weight parameter to form a joint loss function, and the formula is as follows:

[0148] ,

[0149] wherein, is an empirical weight, ;

[0150] The convergence condition of the joint loss function is that the training reaches a set training round number or the joint loss function does not decrease continuously for two rounds, and the set training round number is 12.

[0151] S700, model training is completed, the above cross-modal instance retrieval model is trained until the convergence condition of the joint loss function is met, and a trained cross-modal instance retrieval model is obtained, otherwise the model parameters are adjusted and the learning rate is adjusted to return to step S200; specifically including:

[0152] S710, model training, based on the joint loss function, the cross-modal instance retrieval model based on instance-level soft label alignment is trained by back propagation until the convergence condition of the joint loss function is met, and a trained cross-modal instance retrieval model based on instance-level soft label alignment is obtained, otherwise step S720 is executed;

[0153] S720, adjusting model parameters and adjusting learning rate, using the adamw optimizer to adjust the model parameters and using the cosine annealing learning rate optimization method to adjust the learning rate, returning to step S200.

[0154] S800, cross-modal instance retrieval, using the trained cross-modal instance retrieval model based on instance-level soft label alignment, performing cross-modal instance retrieval on the image set to obtain an instance-level retrieval result ranking list, specifically including:

[0155] S810, instance discovery and database construction, inputting the image set into the cross-modal instance retrieval model based on instance-level soft label alignment, extracting instances for each image to obtain images and their corresponding instances;

[0156] S820, instance-level feature generation, inputting the images and their corresponding instances into the image encoder to extract semantic features of each instance region to obtain instance features and save them;

[0157] S830, text feature extraction, in response to the input natural language query for the instance, the text encoder extracts text semantic features;

[0158] S840, cross-modal similarity calculation and sorting, calculating the similarity of the text semantic features and the instance features according to the instance features and the text semantic features, and sorting, outputting an instance-level retrieval result sorting list.

[0159] To verify the technical effects of the present application, comparative tests were conducted using the prior art CLIP (Contrastive Language-Image Pre-training, a breakthrough multi-modal pre-training model), +GFA (an enhanced grouping function agent using an image-text global level feature alignment loss function), +LG-MGC (Local and Generative driven Modality Gap Correction), and the present application (+GFA+IFA). The CLIP is a baseline model; the +GFA is an enhanced grouping function agent that introduces an image-text global level feature alignment loss function; the +GFA+IFA is an agent of a collaborative framework that fuses an enhanced grouping function agent (+GFA) and an enhanced agent fusion architecture (IFA), i.e., an agent using the cross-modal instance retrieval method based on instance level soft label alignment of the present embodiment. The mAP@50, mAP@100, and mAP@all were obtained.

[0160] The test data is as follows

[0161]

[0162] The above test result table shows that the mAP@50, mAP@100, and mAP@all obtained by the present embodiment are all better than those of other prior art.

[0163] As can be seen, the present application improves the model's understanding of local fine-grained semantic and instance information, not only effectively improving the matching accuracy of cross-modal retrieval, but also expanding the application range of the model in the cross-modal instance retrieval task, further enhancing the multi-scene applicability and practical value.

[0164] The above detailed the preferred embodiments of the present application. It should be understood that those skilled in the art can make many modifications and changes without creative labor based on the concept of the present application. Therefore, any technical solution obtained by logical analysis, reasoning or limited experiments based on the existing technology according to the concept of the present application shall be within the protection scope determined by the claims.

Claims

1. A cross-modal instance retrieval method based on instance-level soft label alignment, characterized in that, The method comprises the following steps: S100, data set preparation, performing open figure-text retrieval, collecting image-text pair data sets, and constructing a cross-modal instance retrieval model based on instance-level soft label alignment; S200、feature extraction, inputting the image-text pair dataset into the cross-modal instance retrieval model to extract image global features and text global features ; S300, instance feature set generation, performing instance discovery on the image-text pair data set, discovering semantic candidate instances, and generating an instance feature set; S400, soft label generation, generating an instance-level semantic distribution of the image, and calculating a normalized semantic probability distribution as a soft label; S500, feature alignment, constructing a soft label guided instance-level feature alignment loss function, performing instance-level feature alignment, constructing an image-text global feature alignment loss function, and performing global feature alignment; S600, loss function integration, combining the instance-level feature alignment loss function and the image-text global feature alignment loss function according to a weight hyperparameter to form a joint loss function; S700, model training completion, training the cross-modal instance retrieval model until a convergence condition of the joint loss function is met, obtaining a trained cross-modal instance retrieval model, or adjusting model parameters and adjusting a learning rate and returning to the step S200; S800, cross-modal instance retrieval, using the trained cross-modal instance retrieval model to perform cross-modal instance retrieval on an image set, and obtaining an instance-level retrieval result ranking list.

2. The cross-modal instance retrieval method based on instance-level soft label alignment according to claim 1, wherein, The image-text pair data set comprises image data and corresponding text data, the cross-modal instance retrieval model based on instance-level soft label alignment is constructed based on a contrast learning language-image pre-training model, the language-image pre-training model comprises an image encoder and a text encoder, the image encoder adopts a Vision Transformer architecture, the text encoder adopts a Transformer architecture, and an instance discovery module is arranged at an intermediate feature layer of an output of the image encoder.

3. The cross-modal instance retrieval method based on instance-level soft label alignment according to claim 2, wherein, The image encoder comprises an image embedding layer, l a Transformer layer and an image linear projection layer, i The calculation formula of the image linear projection layer is as follows: ； Wherein, MultiHead(·) represents a multi-head self-attention mechanism, MLP(·) is a multi-layer perceptron, LayerNorm(·) is a layer normalization operation, and after processing by the l-layer Transformer, an image feature is output , which contains global information and local information, that is , wherein represents a global visual feature, is a local feature of an i-th image block; the global visual feature i is mapped to an image-text embedding space through linear projection, and the image global feature is obtained : ； wherein, is a linear projection, image global features contain global visual information of the image and serve as the final representation of the image for subsequent cross-modal feature alignment.

4. The cross-modal instance retrieval method based on instance-level soft label alignment according to claim 3, wherein, The text encoder comprises a text embedding layer, m a Transformer layer and a text linear projection layer, the text block sequence is processed through the m Transformer layer to obtain global semantic features , the global semantic features are mapped to an image-text embedding space through linear projection to obtain the text global features .

5. The cross-modal instance retrieval method based on instance-level soft label alignment according to claim 4, wherein, The step S200 comprises: S210, image block division, the image encoder divides input image data into a plurality of image blocks; S220, image block sequence generation, the image block passes through the image linear projection layer, generates an image block sequence together with a category label for enhancing the relationship between image blocks, and adds position encoding embedding space information; S230, image block sequence processing, performing self-attention mechanism and multi-layer perceptron processing on the image block sequence; S240, image global feature extraction, the global visual feature is mapped to an image-text embedding space through linear projection, and the image global feature is extracted ; S250, vocabulary unit decomposition, the text encoder decomposes the text data into a series of vocabulary units to generate a text sequence; S260, text block sequence generation, adding start and end markers to the text sequence, generating a text block sequence after text embedding; S270, global semantic feature generation, the text block sequence is processed through a Transformer layer to obtain global semantic features , summarize the global semantic information of the text data; S280, text global feature extraction, the global semantic feature extracting the text global feature by linear projection mapping to image-text embedding space .

6. The cross-modal instance retrieval method based on instance-level soft label alignment according to claim 5, wherein, The step S500 comprises: S510, instance-level feature alignment, aligning the global features of the image Align with the soft label distribution, calculate the normalized generation probability distribution of the global features of the image, calculate the KL divergence, construct the instance-level feature alignment loss function guided by the soft label, and perform instance-level feature alignment; S520, aligning the global features, according to the image global features with the text global features , calculating a cross-modal contrast loss function, maximizing the similarity of positive sample pairs of images and texts within a batch of image-text sample pairs, minimizing the similarity of negative sample pairs, constructing the image-text global feature alignment loss function, and performing global feature alignment.

7. The instance-level soft label alignment based cross-modal instance retrieval method of claim 6, wherein, The soft label guided instance-level feature alignment loss function The formula is as follows: ； wherein, is the image global feature normalized generation probability distribution, D is the feature dimension, is the normalized soft label semantic probability distribution, represents the KL divergence calculation symbol, is the probability of the global feature in the i th feature dimension, is the probability of the soft label semantic in the i th feature dimension.

8. The cross-modal instance retrieval method based on instance-level soft label alignment of claim 7, wherein, The image-text global feature alignment loss function The formula is as follows: ； wherein, and respectively represent the loss function of image to text and the loss function of text to image, and the formula is as follows: ； ； wherein, N is the number of image-text pairs, τ is a temperature hyperparameter, N is the number of image-text pairs, is the image global feature of the i-th sample, is the text feature of the i-th sample.

9. The cross-modal instance retrieval method based on instance-level soft label alignment according to claim 8, wherein, The joint loss function formula is as follows: ； wherein, is an empirical weight.

10. The cross-modal instance retrieval method based on instance-level soft label alignment according to claim 9, wherein, The step S800 comprises: S810, instance discovery and database construction, inputting an image set into the cross-modal instance retrieval model, extracting instances for each image, and obtaining images and corresponding instances thereof; S820, instance-level feature generation, input the image and its corresponding instance into the image encoder, extract semantic features of each instance region, obtain instance features, and save them; S830, text feature extraction, in response to the input natural language query for the instance, the text encoder extracts text semantic features; S840, cross-modal similarity calculation and sorting, according to the instance features and the text semantic features, calculate the similarity between the text semantic features and the instance features and sort them, and output the instance-level retrieval result sorting list.

Images