Cross-modal instance retrieval method based on instance-level soft label alignment

Abstract

The invention discloses a cross-modal instance retrieval method based on instance-level soft tag alignment, which relates to the technical field of multimedia retrieval, and comprises the following steps: S100, preparing a data set; s200, carrying out feature extraction; s300, generating an instance feature set; s400, generating a soft label; s500, carrying out feature alignment; s600, integrating a loss function; s700, completing model training; and S800, carrying out cross-modal instance retrieval. According to the method, an alignment mechanism of instance-level features is introduced, the understanding ability of the model for local fine-grained semantics and instance information is improved through instance discovery and soft label guidance, the matching precision of cross-modal retrieval is effectively improved, the application range of the model in a cross-modal instance retrieval task is expanded, and the method is suitable for popularization and application. And the multi-scene applicability and the practical value are further enhanced.

Description

Technical Field

[0001] The present invention relates to the field of multimedia retrieval technology, and in particular to a cross-modal instance retrieval method based on instance-level soft label alignment. Background Art

[0002] In recent years, instance retrieval methods have made significant progress, particularly in instance-level feature discovery and localization. The invention patent, "A Class-Independent Instance Retrieval Method Based on Hierarchical Semantic Region Decomposition" (CN117725243A, Wanlei Zhao and Qiying Sun), discloses a method based on hierarchical semantic region decomposition. By gradually refining the method, it mines potential instances in images from a global to a local level. Combined with appropriate initialization and termination conditions and irrelevant region filtering mechanisms, this method significantly improves the comprehensiveness and stability of instance discovery.

[0003] Although instance retrieval has made significant progress in single-modality tasks (i.e., image instance retrieval), with the rapid development of artificial intelligence and multimodal learning technologies, existing methods are no longer able to meet the needs of more complex and diverse applications, and there is an urgent need to expand towards multimodality. Cross-modal instance retrieval has thus emerged as a key research direction at the intersection of visual understanding and language understanding. Cross-modal instance retrieval aims to retrieve specific visual instances from an image database based on natural language descriptions, rather than just retrieving the entire image. It emphasizes achieving fine-grained cross-modal semantic matching at the instance level. This task is of great significance to application scenarios such as search engines, intelligent recommendations, and open-world recognition.

[0004] Currently, multimodal research is still mainly focused on the field of cross-modal image and text retrieval. Cross-modal image and text retrieval usually performs feature alignment at the global semantic level, aiming to learn a shared feature space so that the matching images and texts are closer in this space. This task mainly includes two basic forms: text retrieval from images and image retrieval from text. In terms of visual feature extraction, traditional methods usually adopt detectors based on region extraction, such as Faster R-CNN using a bottom-up attention mechanism. In terms of text feature extraction, bidirectional gated recurrent units (BiGRU) or bidirectional encoder representation models (BERT) are often used to encode text sequences.

[0005] In order to improve the matching performance of cross-modal image-text retrieval, early studies have introduced a cross-attention mechanism to improve the matching performance of cross-modal image-text retrieval. The paper "Stacked cross attention for image-text matching[C]" (Proceedings of the European Conference on Computer Vision. Lee KH, Chen X, Hua G, et al. 2018: 201-216) has realized the fine-grained association between image regions and text words by introducing a cross-attention mechanism; using a cross-modal hard alignment network (CHAN), the paper "Fine-grained image-text matching by cross-modal hard aligning network[C]" (Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition. Pan Z, Wu F, Zhang B. 2023: 19275-19284) has proposed a hard alignment strategy to strengthen the consistency of the corresponding areas of the image and text; using a negative perception attention framework (Negative-Aware Attention Framework (NAAF), the paper "Negative-aware attention framework for image-text matching [C]" (Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition. Zhang K, Mao Z, Wang Q, et al. 2022: 15661-15670) further improved the matching accuracy by modeling positive and negative regions.

[0006] To further improve cross-modal retrieval performance, vision-language pretraining models (VLPs) have been widely used. Among them, the contrastive language–image pretraining (CLIP) model, presented in the paper "Learning transferable visual models from natural language supervision[C]" (International Conference on Machine Learning. Radford A, Kim JW, Hallacy C, et al. 2021: 8748-8763), proposes contrastive learning using large-scale image-text pairs. This approach trains image and text encoders separately, mapping data from different modalities into a unified feature space. This approach demonstrates excellent performance in zero-shot retrieval and open-category classification. Based on models such as CLIP, a variety of improved methods have emerged in recent years, such as Local and Generative-driven Modality Gap Correction (LG-MGC), a representative paper "Overcoming the pitfalls of vision-language model for image-text retrieval[C]" (Proceedings of the ACM International Conference on Multimedia. Zhang F, Qu S, Shi F, et al. 2024: 2350 2359), and Cross-modal and Uni-modal Soft-label Alignment (CUSA), a representative paper "Cross-modal and uni-modal soft-label alignment for image-text retrieval[C]" (Proceedings of the AAAI Conference on Artificial Intelligence: volume 38. Huang H, Nie Z, Wang Z, et al. 2024: 18298-18306), these methods further optimized cross-modal feature alignment by introducing local feature modeling and soft label assistance.

[0007] Research on cross-modal instance retrieval has gradually received widespread attention from the academic community in recent years. Researchers have realized that matching images and texts at a global level alone can no longer meet the needs of instance-level understanding and positioning in practical applications. Fine-grained cross-modal semantic alignment has become a new research focus. To support systematic research on this emerging task, benchmarks specifically for instance-level cross-modal retrieval have been constructed, among which ILIAS (Instance-level Image and Language Alignment Search) is a representative example. The ILIAS dataset is annotated with instances as the basic unit, emphasizing the precise correspondence between specific objects in the image and their natural language descriptions. This reflects the current emphasis on fine-grained modeling and evaluation system construction in the field of cross-modal instance retrieval.

[0008] Existing cross-modal retrieval methods, especially those based on fine-tuning of vision-language pre-trained models (e.g., CLIP), have the following major drawbacks: 1. Only global feature alignment is performed, lacking instance-level fine-grained alignment. Existing methods typically achieve cross-modal matching by comparative learning of global image features and text features. However, they lack fine-grained modeling of specific instances and their sub-regions in the image. This results in poor results when precise matching of specific instances (such as local object and detailed descriptions) is required, resulting in insufficient retrieval accuracy.

[0009] 2. Soft label guidance strategies do not target instance features, making it difficult to improve local semantic matching capabilities. Existing methods, such as CUSA, use single-modal soft labels to assist global alignment, but do not incorporate the fine-grained instance features generated by the instance discovery module for guidance. This results in the soft labels being too coarse-grained and unable to effectively improve instance-level alignment.

[0010] 3. Relying on external detectors or pseudo-labels, the generalization and openness are limited. Some detector-based methods require pre-trained detectors to assist in extracting instance features. This method is limited to existing categories and is difficult to adapt to diverse instances in the open world.

[0011] 4. In cross-modal instance retrieval tasks, there is a problem of decreased retrieval accuracy. Most existing cross-modal retrieval methods are mainly designed for full-image matching of images and texts. When migrating to instance-level retrieval, the lack of local and instance feature modeling leads to a decrease in retrieval indicators such as mAP (mean Average Precision), which cannot meet the high requirements for fine-grained matching in practical applications.

[0012] Therefore, technicians in this field are committed to developing a cross-modal instance retrieval method based on instance-level soft label alignment. Summary of the Invention

[0013] In view of the above-mentioned defects of the prior art, the technical problem to be solved by the present invention is the insufficient instance-level feature alignment and weak local semantic matching capability in the cross-modal instance retrieval method.

[0014] In order to improve the instance-level cross-modal matching accuracy and enhance the matching ability between visual features and text features, the applicant divides the cross-modal instance retrieval method into a model training stage and a model reasoning stage, which correspond to the two links of feature learning and retrieval reasoning respectively. By performing fine-grained alignment at the instance granularity, the accuracy and detail matching ability of cross-modal retrieval are effectively improved.

[0015] In one embodiment of the present invention, a cross-modal instance retrieval method based on instance-level soft label alignment is provided, comprising the following steps: S100: Dataset preparation: perform public image-text retrieval, collect image-text pair datasets, and build a cross-modal instance retrieval model based on instance-level soft label alignment; S200, feature extraction, input the image-text pair dataset into the above cross-modal instance retrieval model to extract the global features of the image and global features of text ; S300, instance feature set generation, performing instance discovery on the image text dataset, discovering candidate instances with semantic meaning and generating an instance feature set; S400, soft label generation, generating instance-level semantic distribution of the image, and calculating normalized semantic probability distribution as soft label; S500, feature alignment, constructing a soft-label guided instance-level feature alignment loss function, performing instance-level feature alignment, constructing an image-text global feature alignment loss function, and performing global feature alignment; S600, loss function integration, combining the instance-level feature alignment loss function and the image-text global feature alignment loss function according to weight hyperparameters to form a joint loss function; S700: Model training is completed. The cross-modal instance retrieval model is trained until the convergence condition of the joint loss function is met to obtain a trained cross-modal instance retrieval model. Otherwise, the model parameters and the learning rate are adjusted and the process returns to step S200. S800, cross-modal instance retrieval, using the trained cross-modal instance retrieval model to perform cross-modal instance retrieval on the image collection, and obtain a ranked list of instance-level retrieval results.

[0016] Furthermore, in the cross-modal instance retrieval method based on instance-level soft label alignment in the above embodiment, the image-text pair dataset includes image data and corresponding text data.

[0017] Furthermore, in the cross-modal instance retrieval method based on instance-level soft label alignment in the above embodiment, the image data is a natural scene image containing multiple potential instances.

[0018] Furthermore, in the cross-modal instance retrieval method based on instance-level soft label alignment in the above embodiment, the text data is a natural language description of the target instance.

[0019] Optionally, in the cross-modal instance retrieval method based on instance-level soft label alignment in any of the above embodiments, the cross-modal instance retrieval model based on instance-level soft label alignment is constructed based on a contrastive language-image pretraining (CLIP) model.

[0020] Optionally, in the cross-modal instance retrieval method based on instance-level soft label alignment in any of the above embodiments, the Contrastive Language-Image Pretraining (CLIP) model includes an image encoder and a text encoder, the image encoder adopts a Vision Transformer (ViT) architecture, the text encoder adopts a Transformer architecture, and an instance discovery module is set in the intermediate feature layer of the output of the image encoder.

[0021] Furthermore, in the cross-modal instance retrieval method based on instance-level soft label alignment in the above embodiment, the instance discovery module divides the image into semantically consistent instance candidate regions through an unsupervised method, using a hierarchical clustering method of visual Transformer-based Tokens, and extracts local visual features of the instance candidate regions.

[0022] Furthermore, in the cross-modal instance retrieval method based on instance-level soft label alignment in the above embodiment, the image encoder includes an image embedding layer, l A Transformer layer and an image linear projection layer, i The calculation formula of the layer is as follows: ; Among them, MultiHead(·) represents the multi-head self-attention mechanism, MLP(·) is the multi-layer perceptron, and LayerNorm(·) is the layer normalization operation. l After processing by the layer Transformer, the image features are output , contains global information and local information, that is, ),in Represents the global visual features, For the iThe local features of the image blocks, i 1 to l integer; global visual features The global features of the image are obtained by linear projection mapping to the image-text embedding space : ; in, is a linear projection, the global features of the image It contains the global visual information of the image and serves as the final representation of the image for subsequent cross-modal feature alignment.

[0023] Optionally, in the cross-modal instance retrieval method based on instance-level soft label alignment in the above embodiment, l 12 or 24.

[0024] Preferably, in the cross-modal instance retrieval method based on instance-level soft label alignment in the above embodiment, l is 12.

[0025] Furthermore, in the cross-modal instance retrieval method based on instance-level soft label alignment in the above embodiment, the text encoder includes a text embedding layer, m The text block sequence is passed through a Transformer layer and a text linear projection layer. m The Transformer layer is processed to obtain global semantic features , global semantic features The global features of the text are obtained by linear projection mapping to the image-text embedding space .

[0026] Optionally, in the cross-modal instance retrieval method based on instance-level soft label alignment in the above embodiment, m 8, 12 or 16.

[0027] Preferably, in the cross-modal instance retrieval method based on instance-level soft label alignment in the above embodiment, m is 12.

[0028] Optionally, in the cross-modal instance retrieval method based on instance-level soft label alignment in any of the above embodiments, step S200 includes: S210, image block division, the image encoder divides the input image data into multiple image blocks; S220, generating an image block sequence: the image blocks are passed through a linear projection layer, along with category labels for enhancing the relationship between image blocks, and position encoding is added to embed spatial information to generate an image block sequence; S230, image block sequence processing, performing self-attention mechanism and multi-layer perceptron processing on the image block sequence; S240, image global feature extraction, global visual features are mapped to the image-text embedding space through linear projection to extract image global features ; S250, vocabulary unit decomposition, the text encoder decomposes the text data into a series of vocabulary units to generate a text sequence; S260, generating a text block sequence, adding a start tag and an end tag to the text sequence, and generating a text block sequence after the text is embedded; S270, global semantic feature generation, the text block sequence is processed by Transformer to obtain global semantic features , summarize the global semantic information of text data; S280, global text feature extraction, global semantic features Extracting global features of text by linear projection mapping to image-text embedding space .

[0029] Furthermore, in the cross-modal instance retrieval method based on instance-level soft label alignment in the above embodiment, the instance feature set includes K instance-level feature descriptors , Indicates the K The feature vector of each instance, K is the number of instance-level feature descriptors in the image, D is the feature dimension, K and D Is a positive integer.

[0030] Preferably, in the cross-modal instance retrieval method based on instance-level soft label alignment in the above embodiment, D =512.

[0031] Optionally, in the cross-modal instance retrieval method based on instance-level soft label alignment in any of the above embodiments, step S400 includes: S410, average pooling of instance feature sets, using a soft label generation module to average pool the instance feature sets to generate instance-level semantic distribution of the image; S420 , soft label generation, calculating the normalized semantic probability distribution as the soft label, describing the degree of association between the candidate instance and the potential semantic category in the form of a continuous probability distribution.

[0032] Optionally, in the cross-modal instance retrieval method based on instance-level soft label alignment in any of the above embodiments, the soft label generation module includes: The image encoding and image block generation unit encodes the image using an unsupervised trained feature encoder to generate an image block feature set; The hierarchical semantic region decomposition unit uses a fast binary clustering algorithm to iteratively decompose the image block feature set to construct multi-level instance candidate regions. Specifically, the first decomposition takes the image block feature set as the first layer, and uses a fast binary clustering algorithm to decompose the image block feature set of the first layer into two image block feature subsets to obtain the second layer of image block feature subset pairs. The fast binary clustering algorithm is continued to decompose the image block feature subsets in the second layer of image block feature subset pairs into two image block feature subsets to obtain the third layer of image block feature subset pairs. And so on, to construct multi-level instance candidate regions. The termination condition of the decomposition iteration is: the similarity of the two image block feature subsets in each layer of image block feature subset pairs is calculated. When the number of image block feature subset pairs with a similarity greater than 0.2 is greater than the number of all image block feature subset pairs * 0.97, the decomposition iteration is terminated. Each image block set in the structure can be regarded as a potential instance. The instance screening unit introduces a saliency measurement mechanism to filter out dumb nodes based on feature energy overlap. Specifically, the L1 norm of the image blocks in the image block feature set is calculated and sorted from large to small. The top 30% of the image blocks are defined as energy-salient image blocks. If the overlap rate between the image block in the current instance and the energy-salient image block is less than 20%, it is considered semantically insignificant, i.e., a dumb node. The dumb node is removed to obtain the candidate instance region. The instance feature description unit, i.e., the descriptor, extracts image region features from the selected candidate instance regions and generates instance-level feature expressions through the cropped instance region extraction. .

[0033] Optionally, in the cross-modal instance retrieval method based on instance-level soft label alignment in any of the above embodiments, the average pooling formula is as follows: ; in, is the instance-level feature descriptor vector after average pooling, K is the number of instance-level feature descriptors in the image, Indicates the K The feature vector of each instance.

[0034] Optionally, in the cross-modal instance retrieval method based on instance-level soft label alignment in any of the above embodiments, the normalized semantic probability distribution formula is as follows: ; in, is the instance-level feature descriptor vector after average pooling, is the normalized soft label semantic probability distribution, is L2 normalization.

[0035] Optionally, in the cross-modal instance retrieval method based on instance-level soft label alignment in any of the above embodiments, step S500 includes: S510, instance-level feature alignment, global features of the image Align with the soft label distribution, calculate the probability distribution of the normalized global features of the image, calculate the KL (Kullback-Leibler) divergence, construct the soft label-guided instance-level feature alignment loss function, and perform instance-level feature alignment; S520, global feature alignment, based on the global features of the image and global features of text , calculate the cross-modal contrast loss function, maximize the similarity of positive image-text pairs and minimize the similarity of negative sample pairs within a batch of image-text pair samples, construct the image-text global level feature alignment loss function, and perform global feature alignment. In the image-text pair samples, the correctly corresponding image-text pairs are positive sample pairs, and the incorrectly corresponding image-text pairs are negative sample pairs.

[0036] Optionally, in the cross-modal instance retrieval method based on instance-level soft label alignment in any of the above embodiments, the global features of the image are normalized to generate a probability distribution, and the formula is as follows: ; in, is the normalized probability distribution of the global features of the image, is the global feature of the image, is L2 normalization.

[0037] Optionally, in the cross-modal instance retrieval method based on instance-level soft label alignment in any of the above embodiments, the soft label-guided instance-level feature alignment loss function , the formula is as follows: ; in, is the normalized probability distribution of the global features of the image, D is the feature dimension, is the normalized soft label semantic probability distribution, Represents the KL divergence calculation symbol, It is the global feature i The probability of feature dimensions, Soft label semantics i The probability of a feature dimension.

[0038] Optionally, in the cross-modal instance retrieval method based on instance-level soft label alignment in any of the above embodiments, the image-text global level feature alignment loss function The formula is as follows: ; in, and They represent the loss function of image to text and the loss function of text to image, respectively. The formulas are as follows: ; ; in, N is the number of image-text pairs, τ is the temperature hyperparameter, and the contrastive loss It can maximize the similarity of positive sample pairs, minimize the similarity of negative sample pairs, and enhance the cross-modal discrimination ability of the model. N is the number of image-text pairs, is the global image feature of the i-th sample, is the text feature of the i-th sample.

[0039] Furthermore, in the cross-modal instance retrieval method based on instance-level soft label alignment in the above embodiment, the joint loss function formula is as follows: , in, is the experience weight.

[0040] Preferably, in the cross-modal instance retrieval method based on instance-level soft label alignment in the above embodiment, .

[0041] Optionally, in the cross-modal instance retrieval method based on instance-level soft label alignment in any of the above embodiments, step S700 includes: S710: Model training: Based on the joint loss function, the cross-modal instance retrieval model based on instance-level soft label alignment is trained by back propagation until the convergence condition of the joint loss function is met, thereby obtaining a trained cross-modal instance retrieval model based on instance-level soft label alignment. Otherwise, step S720 is executed. S720: Adjust model parameters and learning rate, and return to step S200.

[0042] Furthermore, in the cross-modal instance retrieval method based on instance-level soft label alignment in the above embodiment, the convergence condition of the joint loss function is that the training reaches a set number of training rounds or the joint loss function no longer decreases for two consecutive rounds.

[0043] Optionally, in the cross-modal instance retrieval method based on instance-level soft label alignment in the above embodiment, the number of training rounds is set in the range of [6, 20].

[0044] Preferably, in the cross-modal instance retrieval method based on instance-level soft label alignment in the above embodiment, the number of training rounds is set to 12.

[0045] Optionally, in the cross-modal instance retrieval method based on instance-level soft label alignment in any of the above embodiments, the adjustment of model parameters in step S700 uses an adamw optimizer.

[0046] Furthermore, in the cross-modal instance retrieval method based on instance-level soft label alignment in the above embodiment, the learning rate is adjusted in step S700 using a cosine annealing learning rate optimization method.

[0047] Optionally, in the cross-modal instance retrieval method based on instance-level soft label alignment in any of the above embodiments, step S800 includes: S810: Instance discovery and database construction: Input the image collection into a cross-modal instance retrieval model based on instance-level soft label alignment to extract instances for each image, obtaining images and their corresponding instances. S820, instance-level feature generation: input the image and its corresponding instance into the image encoder, extract the semantic features of each instance region, obtain instance features, and save them; S830, text feature extraction: in response to the input natural language query for the instance, the text encoder extracts text semantic features; S840, cross-modal similarity calculation and sorting, based on instance features and text semantic features, calculate the similarity between text semantic features and instance features and sort them, and output a sorted list of instance-level retrieval results.

[0048] The present invention processes the local features output by the image encoder to generate continuous soft labels representing the semantic distribution of instances to guide feature learning. This method can delicately model fine-grained semantic relationships between instances and improve the ability of local features to characterize instance attributes. A combined optimization strategy is adopted in the training phase. The Kullback-Leibler divergence (KL divergence) is used to minimize the difference between the instance feature distribution and the soft label distribution at the instance level. At the global level, a cross-modal contrastive loss is used to optimize the alignment of global image features with global text features. This method can simultaneously take into account local fine-grained semantic modeling and global semantic associations, significantly improving the accuracy and robustness of cross-modal matching. In the inference phase, the instance discovery mechanism used in the training phase is retained. Features of potential instance regions are extracted through unsupervised instance partitioning and similarity is calculated with the features of the input query text to achieve instance-based cross-modal retrieval. This method not only retains the efficiency of traditional global feature matching but also introduces fine-grained instance information, improving retrieval accuracy under complex detailed descriptions. It can take into account both local and global information and has broader application potential in instance retrieval and detail matching tasks.

[0049] This method uses large-scale image-text alignment training based on Vision-Language Pretraining Models (VLPs). By leveraging the implicit cross-modal correspondences in massive amounts of data, it significantly enhances the model's generalization and open-world adaptability, overcoming the limitations of traditional methods that rely on limited annotated data and result in limited retrieval performance. Compared to other methods based on Vision-Language Pretraining Models, this method uses a dual-stream Contrastive Language-Image Pretraining (CLIP) model as its foundational model, employing independent image and text encoders for feature extraction. Image-text matching is achieved only in the final stage through similarity calculation, reducing complex early modal interactions. This reduces computational resource consumption and improves processing efficiency during training and inference, making it suitable for resource-constrained or large-scale retrieval applications. Based on the original global feature alignment, this invention innovatively introduces an instance-level feature alignment mechanism. Through instance discovery and soft label guidance, it improves the model's ability to understand local fine-grained semantics and instance information. It not only effectively improves the matching accuracy of cross-modal retrieval, but also expands the application scope of the model in cross-modal instance retrieval tasks, further enhancing its multi-scenario applicability and practical value.

[0050] The concept, specific structure and technical effects of the present invention will be further described below in conjunction with the accompanying drawings to fully understand the purpose, characteristics and effects of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] Figure 13 is a flowchart of a cross-modal instance retrieval method based on instance-level soft label alignment according to an embodiment of the present invention. DETAILED DESCRIPTION

[0052] The following describes several preferred embodiments of the present invention with reference to the accompanying drawings to make its technical content clearer and easier to understand. The present invention can be embodied in many different forms of embodiments, and the scope of protection of the present invention is not limited to the embodiments mentioned herein.

[0053] In the drawings, components with identical structures are denoted by the same reference numerals, and components with similar structures or functions are denoted by similar reference numerals. The size and thickness of each component shown in the drawings are arbitrary and are not limited by the present invention. To enhance clarity, the thickness of components in some places in the drawings is schematically exaggerated.

[0054] This application designs a cross-modal instance retrieval method based on instance-level soft label alignment, such as Figure 1 As shown, the following steps are included: S100. Dataset preparation, public image-text retrieval, image-text pair dataset collection, including image data and corresponding text data. Image data is a natural scene image containing multiple potential instances, and text data is a natural language description of the target instance. The contrastive language-image pretraining (CLIP) model based on contrastive learning builds a cross-modal instance retrieval model based on instance-level soft label alignment. The CLIP model includes an image encoder and a text encoder. The image encoder adopts the Vision Transformer (ViT) architecture, and the text encoder adopts the Transformer architecture. The instance discovery module is set in the intermediate feature layer of the image encoder's output. The instance discovery module uses an unsupervised method and a hierarchical clustering method based on the visual transformer's token to divide the image into semantically consistent instance candidate regions and extract local visual features of the instance candidate regions; the image encoder includes an image embedding layer, l A Transformer layer and an image linear projection layer, i The calculation formula of the layer is as follows: ; Among them, MultiHead(·) represents the multi-head self-attention mechanism, MLP(·) is the multi-layer perceptron, and LayerNorm(·) is the layer normalization operation. lAfter processing by the layer Transformer, the image features are output , contains global information and local information, that is, ),in Represents the global visual features, For the i The local features of the image blocks, i 1 to l integer; global visual features The global features of the image are obtained by linear projection mapping to the image-text embedding space : ; in, is a linear projection, the global features of the image Contains the global visual information of the image and serves as the final representation of the image for subsequent cross-modal feature alignment. l is 12; the text encoder includes a text embedding layer, m The text block sequence is passed through a Transformer layer and a text linear projection layer. m The Transformer layer is processed to obtain global semantic features , m is 12, global semantic features The global features of the text are obtained by linear projection mapping to the image-text embedding space .

[0055] S200, feature extraction, input image data and text data into the cross-modal instance retrieval model based on instance-level soft label alignment to extract global image features and global features of text ; Specifically include: S210, image block division, the image encoder divides the input image data into multiple image blocks; S220, generating an image block sequence: the image blocks are passed through a linear projection layer, along with category labels for enhancing the relationship between image blocks, and position encoding is added to embed spatial information to generate an image block sequence; S230, image block sequence processing, performing self-attention mechanism and multi-layer perceptron processing on the image block sequence; S240, image global feature extraction, global visual features are mapped to the image-text embedding space through linear projection to extract image global features ; S250, vocabulary unit decomposition, the text encoder decomposes the text data into a series of vocabulary units to generate a text sequence; S260, generating a text block sequence, adding a start tag and an end tag to the text sequence, and generating a text block sequence after the text is embedded; S270, global semantic feature generation, the text block sequence is processed by Transformer to obtain global semantic features , summarize the global semantic information of text data; S280, global text feature extraction, global semantic features Extracting global features of text by linear projection mapping to image-text embedding space .

[0056] S300, instance feature set generation, perform instance discovery on image data, discover candidate instances with semantics and generate instance feature sets, including K instance-level feature descriptors , Indicates the K The feature vector of each instance, K is the number of instance-level feature descriptors in the image, D is the feature dimension, K and D is a positive integer, D =512.

[0057] S400, soft label generation, generates instance-level semantic distribution of the image and calculates normalized semantic probability distribution as the soft label; specifically includes: S410: Average pooling of instance feature sets. Using a soft label generation module, average pooling is performed on the instance feature sets to generate instance-level semantic distribution of the image. The soft label generation module includes: The image encoding and image block generation unit encodes the image using an unsupervised trained feature encoder to generate an image block feature set; The hierarchical semantic region decomposition unit uses a fast binary clustering algorithm to iteratively decompose the image block feature set to construct multi-level instance candidate regions. Specifically, the first decomposition takes the image block feature set as the first layer, and uses a fast binary clustering algorithm to decompose the image block feature set of the first layer into two image block feature subsets to obtain the second layer of image block feature subset pairs. The fast binary clustering algorithm is continued to decompose the image block feature subsets in the second layer of image block feature subset pairs into two image block feature subsets to obtain the third layer of image block feature subset pairs. And so on, to construct multi-level instance candidate regions. The termination condition of the decomposition iteration is: the similarity of the two image block feature subsets in each layer of image block feature subset pairs is calculated. When the number of image block feature subset pairs with a similarity greater than 0.2 is greater than the number of all image block feature subset pairs * 0.97, the decomposition iteration is terminated. Each image block set in the structure can be regarded as a potential instance. The instance screening unit introduces a saliency measurement mechanism to filter out dumb nodes based on feature energy overlap. Specifically, the L1 norm of the image blocks in the image block feature set is calculated and sorted from large to small. The top 30% of the image blocks are defined as energy-salient image blocks. If the overlap rate between the image block in the current instance and the energy-salient image block is less than 20%, it is considered semantically insignificant, i.e., a dumb node. The dumb node is removed to obtain the candidate instance region. The instance feature description unit, i.e., the descriptor, extracts image region features from the selected candidate instance regions and generates instance-level feature expressions through the cropped instance region extraction. ; The average pooling formula is as follows: ; in, is the instance-level feature descriptor vector after average pooling, K is the number of instance-level feature descriptors in the image, Indicates the K The feature vector of each instance; S420, soft label generation, calculate the normalized semantic probability distribution as the soft label, and describe the degree of association between the candidate instance and the potential semantic category in the form of a continuous probability distribution. The normalized semantic probability distribution formula is as follows: ; in, is the instance-level feature descriptor vector after average pooling, is the normalized soft label semantic probability distribution, is L2 normalization.

[0058] S500, feature alignment, global features of the image Align with the soft label distribution, calculate the probability distribution of the normalized generation of the global features of the image, calculate the KL (Kullback-Leibler) divergence, construct the soft label guided instance level feature alignment loss function, and perform instance level feature alignment; according to the global features of the image and global features of text , calculate cross-modal contrast loss, construct image-text global feature alignment loss function, and perform global feature alignment; specifically including: S510, instance-level feature alignment, global features of the image Align with the soft label distribution and calculate the normalized probability distribution of the global features of the image. The formula is as follows: ; in, is the normalized probability distribution of the global features of the image, is the global feature of the image, is L2 normalization; Calculate KL (Kullback-Leibler) divergence and construct soft label guided instance-level feature alignment loss function , the formula is as follows: ; in, is the normalized probability distribution of the global features of the image, D is the feature dimension, is the normalized soft label semantic probability distribution, Represents the KL divergence calculation symbol, It is the global feature i The probability of feature dimensions, Soft label semantics i The probability of feature dimensions; Perform instance-level feature alignment; S520, global feature alignment, based on the global features of the image and global features of text , calculate the cross-modal contrast loss function, maximize the similarity of positive sample pairs and minimize the similarity of negative sample pairs within a batch of image-text pair samples, and construct the image-text global level feature alignment loss function , the formula is as follows: ; in, and They represent the loss function of image to text and the loss function of text to image, respectively. The formulas are as follows: ; ; in, N is the number of image-text pairs, τ is the temperature hyperparameter, and the contrastive loss It can maximize the similarity of positive sample pairs, minimize the similarity of negative sample pairs, and enhance the cross-modal discrimination ability of the model. N is the number of image-text pairs, is the global image feature of the i-th sample, is the text feature of the i-th sample; Perform global feature alignment. In the image-text pair samples, the correctly corresponding image-text pairs are positive sample pairs, and the incorrectly corresponding image-text pairs are negative sample pairs.

[0059] S600, loss function integration, the instance-level feature alignment loss function and the image-text global-level feature alignment loss function are combined according to the weight hyperparameters to form a joint loss function, the formula is as follows: , in, is the experience weight, ; The convergence condition of the joint loss function is that the training reaches the set number of training rounds or the joint loss function does not decrease for two consecutive rounds. The set number of training rounds is 12.

[0060] S700: Model training is completed. The cross-modal instance retrieval model is trained until the convergence condition of the joint loss function is met to obtain a trained cross-modal instance retrieval model. Otherwise, the model parameters and the learning rate are adjusted and the process returns to step S200. Specifically, the process includes: S710: Model training: Based on the joint loss function, the cross-modal instance retrieval model based on instance-level soft label alignment is trained by back propagation until the convergence condition of the joint loss function is met, thereby obtaining a trained cross-modal instance retrieval model based on instance-level soft label alignment. Otherwise, step S720 is executed. S720, adjust the model parameters and the learning rate, use the adamw optimizer to adjust the model parameters and use the cosine annealing learning rate optimization method to adjust the learning rate, and return to step S200.

[0061] S800, cross-modal instance retrieval, uses the trained cross-modal instance retrieval model based on instance-level soft label alignment to perform cross-modal instance retrieval on the image collection and obtain a ranked list of instance-level retrieval results, specifically including: S810: Instance discovery and database construction: Input the image collection into a cross-modal instance retrieval model based on instance-level soft label alignment to extract instances for each image, obtaining images and their corresponding instances. S820, instance-level feature generation: input the image and its corresponding instance into the image encoder, extract the semantic features of each instance region, obtain instance features, and save them; S830, text feature extraction: in response to the input natural language query for the instance, the text encoder extracts text semantic features; S840, cross-modal similarity calculation and sorting, based on instance features and text semantic features, calculate the similarity between text semantic features and instance features and sort them, and output a sorted list of instance-level retrieval results.

[0062] To verify the technical effect of the present invention, the present application conducted comparative experiments, respectively using the existing technology CLIP (C‌ontrastive ‌L‌anguage-‌I‌mage ‌P‌re-training, a breakthrough multimodal pre-training model‌), +GFA (enhanced grouping function agent, an agent using the image-text global-level feature alignment loss function), +LG-MGC (‌Local and Generative driven Modality Gap Correction, local and generative driven modality gap correction) and the present invention (+GFA+IFA) for experiments, where CLIP is the baseline model; +GFA is the enhanced grouping function agent that introduces the image-text global-level feature alignment loss function; +GFA+IFA is an agent in a collaborative framework that integrates the enhanced ‌grouping function agent (+GFA)‌ and the enhanced ‌agent fusion architecture (IFA)‌, that is, an agent using the cross-modal instance retrieval method based on instance-level soft label alignment of this embodiment; mAP@50, mAP@100 and mAP@all are obtained respectively.

[0063] The test data is as follows The above test result table shows that the mAP@50, mAP@100 and mAP@all indicators obtained in this embodiment are better than other existing technologies.

[0064] It can be seen that the present invention improves the model's ability to understand local fine-grained semantics and instance information, which not only effectively improves the matching accuracy of cross-modal retrieval, but also expands the application scope of the model in cross-modal instance retrieval tasks, further enhancing its multi-scenario applicability and practical value.

[0065] The above describes in detail the preferred embodiments of the present invention. It should be understood that those skilled in the art can make numerous modifications and variations based on the concepts of the present invention without inventive effort. Therefore, any technical solutions that can be derived by those skilled in the art through logical analysis, reasoning, or limited experimentation based on the concepts of the present invention and the prior art should be within the scope of protection defined by the claims.

Claims

1. A cross-modal instance retrieval method based on instance-level soft label alignment, characterized by: The steps include: S100: Dataset preparation: perform public image-text retrieval, collect image-text pair datasets, and build a cross-modal instance retrieval model based on instance-level soft label alignment; S200, feature extraction, inputting the image-text pair dataset into the cross-modal instance retrieval model to extract global image features and global features of text ; S300, generating an instance feature set, performing instance discovery on the image text in the dataset, discovering candidate instances with semantic meaning and generating an instance feature set; S400, soft label generation, generating instance-level semantic distribution of the image, and calculating normalized semantic probability distribution as soft label; S500, feature alignment, constructing the instance-level feature alignment loss function guided by the soft label, and performing instance-level feature alignment; Construct an image-text global feature alignment loss function to perform global feature alignment; S600, loss function integration, combining the instance-level feature alignment loss function and the image-text global-level feature alignment loss function according to weight hyperparameters to form a joint loss function; S700: Model training is completed. The cross-modal instance retrieval model is trained until the convergence condition of the joint loss function is met to obtain a trained cross-modal instance retrieval model. Otherwise, the model parameters and the learning rate are adjusted and the process returns to step S200. S800 , cross-modal instance retrieval: using the trained cross-modal instance retrieval model, performing cross-modal instance retrieval on the image collection to obtain a ranked list of instance-level retrieval results.

2. The cross-modal instance retrieval method based on instance-level soft label alignment according to claim 1, characterized in that: The image-text pair dataset includes image data and corresponding text data. The cross-modal instance retrieval model with instance-level soft label alignment is constructed based on a language-image pre-training model of contrastive learning. The language-image pre-training model includes an image encoder and a text encoder. The image encoder adopts a Vision Transformer architecture, and the text encoder adopts a Transformer architecture. An instance discovery module is set in the intermediate feature layer of the output of the image encoder.

3. The cross-modal instance retrieval method based on instance-level soft label alignment according to claim 2 is characterized in that: The image encoder includes an image embedding layer, l A Transformer layer and an image linear projection layer, i The calculation formula of the layer is as follows: ； Among them, MultiHead(·) represents the multi-head self-attention mechanism, MLP(·) is the multi-layer perceptron, and LayerNorm(·) is the layer normalization operation. After being processed by the l-layer Transformer, the output image features are , contains global information and local information, that is, ),in Represents the global visual features, For the i The local features of the image blocks, i 1 to l integer; global visual features The global features of the image are obtained by linear projection mapping to the image-text embedding space : ； in, is a linear projection, the global features of the image It contains the global visual information of the image and serves as the final representation of the image for subsequent cross-modal feature alignment.

4. The cross-modal instance retrieval method based on instance-level soft label alignment according to claim 3 is characterized in that: The text encoder includes a text embedding layer, m The text block sequence passes through the Transformer layer and the text linear projection layer. m The Transformer layer is processed to obtain global semantic features , the global semantic feature The global features of the text are obtained by linear projection mapping to the image-text embedding space .

5. The cross-modal instance retrieval method based on instance-level soft label alignment according to claim 4 is characterized in that: The step S200 includes: S210, image block division, the image encoder divides the input image data into multiple image blocks; S220, generating an image block sequence, wherein the image blocks are passed through the image linear projection layer, together with category labels for enhancing the relationship between image blocks, and position encoding is added to embed spatial information to generate an image block sequence; S230, image block sequence processing, performing self-attention mechanism and multi-layer perceptron processing on the image block sequence; S240, extracting global features of the image, mapping the global visual features to the image-text embedding space through linear projection, and extracting the global features of the image ; S250, vocabulary unit decomposition, the text encoder decomposes the text data into a series of vocabulary units to generate a text sequence; S260, generating a text block sequence, adding a start tag and an end tag to the text sequence, and generating a text block sequence after the text is embedded; S270, global semantic feature generation, the text block sequence is processed through the Transformer layer to obtain the global semantic feature , summarize the global semantic information of text data; S280, extracting global features of text, the global semantic features Extract the global features of the text by linear projection mapping to the image-text embedding space .

6. The cross-modal instance retrieval method based on instance-level soft label alignment according to claim 5, characterized in that: The step S500 includes: S510, instance-level feature alignment, the image global features Aligning with the soft label distribution, calculating the normalized probability distribution of the global features of the image, calculating the KL divergence, constructing the soft label-guided instance-level feature alignment loss function, and performing instance-level feature alignment; S520, global feature alignment, based on the global features of the image Global features with the text , calculate the cross-modal contrast loss function, maximize the similarity of positive image-text pairs and minimize the similarity of negative samples within a batch of image-text pair samples, construct the image-text global level feature alignment loss function, and perform global feature alignment.

7. The cross-modal instance retrieval method based on instance-level soft label alignment according to claim 6, characterized in that: The soft label guided instance-level feature alignment loss function , the formula is as follows: ； in, is the normalized probability distribution of the global features of the image, D is the feature dimension, is the normalized soft label semantic probability distribution, Represents the KL divergence calculation symbol, It is the global feature i The probability of feature dimensions, Soft label semantics i The probability of a feature dimension.

8. The cross-modal instance retrieval method based on instance-level soft label alignment according to claim 7, characterized in that: The image-text global level feature alignment loss function The formula is as follows: ； in, and They represent the loss function of image to text and the loss function of text to image, respectively. The formulas are as follows: ； ； in, N is the number of image-text pairs, τ is the temperature hyperparameter, and the contrastive loss It can maximize the similarity of positive sample pairs, minimize the similarity of negative sample pairs, and enhance the cross-modal discrimination ability of the model. N is the number of image-text pairs, is the global image feature of the i-th sample, is the text feature of the i-th sample.

9. The cross-modal instance retrieval method based on instance-level soft label alignment according to claim 8, characterized in that: The joint loss function formula is as follows: ； in, is the experience weight.

10. The cross-modal instance retrieval method based on instance-level soft label alignment according to claim 9, characterized in that: The step S800 includes: S810, instance discovery and database construction, inputting the image collection into the cross-modal instance retrieval model based on instance-level soft label alignment, extracting instances for each image, and obtaining images and their corresponding instances; S820, instance-level feature generation: input the image and its corresponding instance into the image encoder, extract semantic features of each instance region, obtain instance features, and save them; S830, text feature extraction: in response to the input natural language query for the instance, the text encoder extracts text semantic features; S840, cross-modal similarity calculation and sorting, based on the instance features and the text semantic features, calculate the similarity between the text semantic features and the instance features and sort them, and output the instance-level retrieval result sorting list.

Images