Adaptive text-image semantic alignment method based on nested multi-granularity representation

Abstract

This invention discloses an adaptive text-image semantic alignment method and system based on nested multi-granularity representations, relating to the field of cross-modal information retrieval technology. The invention includes: dynamically selecting different numbers of tokens from image patch tokens using a visual token selection strategy, concatenating them with classification tokens, and generating nested multi-granularity visual representations via a shared Transformer decoder layer; employing a contrastive learning strategy with granularity-difficulty forced mapping during training to couple sample difficulty with the granularity space, and utilizing a confidence consensus partitioning mechanism to suppress noise; and proposing an adaptive agile inference mechanism based on prediction entropy during inference, achieving precise matching of computational resources and query complexity by progressively calculating the entropy value of the candidate set similarity distribution and dynamically deciding whether to stop early. This invention supports seamless switching between efficiency-first and performance-first modes, significantly reducing computational overhead while ensuring retrieval accuracy, providing a flexible and scalable solution for edge deployment.

Description

Technical Field

[0001] This invention relates to the field of cross-modal information retrieval technology, and more specifically, to a text-to-image pedestrian re-identification technology. Background Technology

[0002] Text-to-Image Person Re-identification (T2I-ReID) is a key computer vision technology that aims to accurately retrieve corresponding pedestrian images from a vast image database based on a natural language description. This technology has significant application value in fields such as intelligent security, public safety, and unmanned surveillance. Imagine a scenario where law enforcement officers need to quickly identify a suspect in a crowded urban area based solely on eyewitness descriptions: "A middle-aged man wearing a black bomber jacket, carrying a water bottle, with a red collar." Such queries encompass both global attributes (e.g., "middle-aged man") and fine-grained details (e.g., "red collar"), requiring the retrieval system to dynamically adapt to different levels of complexity.

[0003] However, existing technologies generally face challenges in balancing retrieval accuracy and computational efficiency. On the one hand, natural language descriptions typically contain multi-layered information, ranging from global appearances like "wearing a red coat" to local details like "a backpack has a zipper"; on the other hand, real-world applications (especially edge computing devices) have strict limitations on computational resources and inference latency. Traditional methods usually learn a fixed-length high-dimensional visual representation for each image. This "one-size-fits-all" strategy leads to a fundamental contradiction: for simple text queries, the high-dimensional representation generates unnecessary computational redundancy; while for complex queries, the fixed representation may not be able to fully capture all the key fine-grained features, thus affecting retrieval accuracy. Figure 1 As shown, existing methods encode each image into a uniform high-dimensional vector, leading to redundant processing of simple descriptions (such as "a person wearing red clothes") while insufficient extraction of details for complex descriptions (such as "wearing a striped shirt and a zippered backpack"). The fundamental disconnect lies in the static nature of current paradigms, which fail to mimic human adaptive reasoning: starting with coarse visual cues (such as overall outlines) and progressively focusing on finer elements (such as shoe shape) as needed. This limitation is particularly acute in resource-constrained edge environments, such as deploying T2I-ReID in real-time applications (such as autonomous monitoring drones or mobile devices) requiring minimal latency. However, existing methods apply high-cost computation uniformly to all samples, wasting resources on simple queries and proving inadequate for complex ones.

[0004] The closest existing technology to this invention is a cross-modal retrieval scheme based on Matryoshka Representation Learning (MRL). The core idea of ​​MRL is to encode multi-granularity information within a single high-dimensional vector, so that different length prefixes of that vector can themselves serve as effective low-dimensional representations, allowing for flexible selection of feature dimensions based on computational budget during inference. Based on this idea, subsequent works such as the Matryoshka Query Transformer (MQT) extend it to visual language models, explicitly controlling the granularity of visual representations by dynamically adjusting the number of visual tokens; while the Matryoshka Query Transformer M... 3 Matryoshka Multimodal Models further applies the nested modeling concept to both visual and textual modalities. The implementation steps for these schemes typically involve: during the training phase, simultaneously optimizing features of different granularities (different dimensions or different token lengths) of the model's output; and during the inference phase, selecting an appropriate granularity for computation based on requirements, for example, first using coarse-grained features for rapid filtering, and then using fine-grained features for reordering.

[0005] However, the above methods are all based on the idea of ​​feature dimension truncation, meaning that the backbone network always outputs a complete high-dimensional vector, and the vector is sliced ​​only in the post-processing stage. While this approach can achieve multi-granularity representation, it cannot reduce the computational cost of the backbone network itself, and is still not user-friendly for resource-constrained edge devices. Furthermore, these existing solutions mainly verify the feasibility of nested representations in general visual question answering or image description tasks, and have not performed in-depth optimization for the characteristics of text-to-image person re-identification tasks, specifically in the following aspects:

[0006] (1) At the feature optimization level, although existing methods have introduced multi-granularity features, they have not forced the mapping between sample difficulty and granularity space. As a result, coarse-grained features still need to process difficult negative samples and fine-grained features still need to process simple negative samples, which cannot achieve differentiated and accurate alignment. There is a lack of effective interaction between multi-granularity features.

[0007] (2) At the level of reasoning strategy, the existing methods adopt a fixed two-stage cascade retrieval (first coarse screening and then rearrangement), which lacks the ability to dynamically judge the confidence of the retrieval results, resulting in simple queries still needing to perform a complete rearrangement, which results in computational redundancy. Summary of the Invention

[0008] The technical problem to be solved by this invention is to overcome the shortcomings of existing text-to-image pedestrian re-identification methods in that it is difficult to balance retrieval accuracy and computational efficiency, and to provide an adaptive retrieval method that can dynamically allocate computing resources according to query complexity and support efficient deployment of edge devices and lossless performance recovery.

[0009] The technical solution adopted by this invention to solve the above-mentioned technical problems is an adaptive text-image semantic alignment method based on nested multi-granularity representation, comprising the following steps:

[0010] Step 1: Constructing multi-granularity nested visual representations

[0011] The input image is encoded using a visual Transformer into a sequence containing a classification token and multiple image patch tokens. ;

[0012] Based on the attention weights of the classification token and each image patch token, a visual token selection strategy is used to dynamically select tokens from all image patch tokens. Given a number of image block tokens, obtain the selected token sequence. ,in A set of increasing cutoff lengths The i-th value in the array; i is the granularity index, m is the total number of granularities; the finer the granularity, the larger the index.

[0013] The classifying token and the selected token sequence are concatenated and then input into a shared Transformer decoder layer to generate a sequence corresponding to the truncation length. Local visual representation ;

[0014] The classification tokens are linearly mapped to obtain the global visual representation. ;

[0015] This yields nested, multi-granular visual representations. ;

[0016] Step 2: Multi-granularity comparative learning

[0017] The similarity between the text representation and the multi-granularity visual representation is calculated respectively, and the negative samples are divided into simple negative sample group, medium negative sample group and difficult negative sample group according to the similarity.

[0018] Different contrast loss constraints are applied to each group of negative samples, where the loss corresponding to the coarse-grained visual representation focuses on the simple negative sample group, and the loss corresponding to the fine-grained visual representation focuses on the difficult negative sample group.

[0019] During training, a confidence consensus partitioning mechanism is used to generate confidence weights for each image-text pair, and the contrast loss at each granularity is weighted using the confidence weights.

[0020] Step 3: Adaptive Search

[0021] During the inference phase, the coarsest-grained local visual representation is selected from the multi-granularity visual representations. A preliminary search of the image library was performed to obtain a candidate set;

[0022] For the current granularity i, calculate the similarity distribution entropy of the top N results in the candidate set. ;

[0023] The entropy Compared with a preset threshold: If If the value is less than the threshold, the search is terminated and the current sorting result is output; if... If the value is not less than the threshold, then switch to the next finer-grained visual representation. The similarity of the candidate set is recalculated and the above entropy comparison process is repeated until the termination condition is met or the finest granularity is reached.

[0024] Furthermore, the confidence consensus partitioning mechanism is as follows: using the dual loss information of the global branch and the local branch, a confidence weight is generated for each image-text pair through a Beta mixture model. High-weight samples contribute more to the loss calculation, while low-weight samples are adaptively downweighted.

[0025] Furthermore, to achieve lossless deployment flexibility, the method supports selecting either an efficiency-first mode or a performance-first mode during the inference phase based on hardware conditions or accuracy requirements; in efficiency-first mode, the adaptive retrieval step is executed; in performance-first mode, the finest-grained visual representation is directly used. Perform a one-time search of the image library.

[0026] Unlike existing technologies that rely on feature dimension truncation, this invention proposes a token-level dynamic selection mechanism, VTS (Visual Token Selection). This mechanism dynamically selects the most important image patches based on the attention weights of the [CLS] token and the image patch token, and generates multi-granularity nested representations through a shared Transformer decoder layer (TSE-Layer). This method not only achieves multi-granularity output but also substantially reduces the self-attention computation cost of the Transformer backbone network, solving the problem that existing methods, which only post-process slices, cannot save computational overhead.

[0027] This invention forces a mapping between sample difficulty and granularity space: coarse-grained features focus on quickly separating simple negative samples, while fine-grained features focus on finely distinguishing difficult negative samples. It also introduces a Confidence Consensus Division (CCD) mechanism, which uses a bi-branch loss to fit a Gaussian mixture model to dynamically evaluate sample confidence, effectively suppressing noise interference in the training data and achieving accurate alignment of text and images at multiple granular levels.

[0028] This invention abandons the traditional fixed two-stage cascaded retrieval and proposes an entropy-driven dynamic early stopping strategy. After each granularity of retrieval, the predicted entropy of the candidate set similarity distribution is calculated, and the retrieval is adaptively terminated or moved to a finer granularity based on the comparison between the entropy value and the threshold. This mechanism allows simple queries to stop early at the coarse-grained stage, significantly reducing the computational load of inference while maintaining accuracy.

[0029] This invention organically integrates the above three parts, supporting seamless and lossless switching between efficiency-first and performance-first modes. Unlike irreversible compression methods such as pruning and distillation, the nested representation structure of this invention allows for the direct use of the finest-grained representation to fully restore the original model performance when hardware upgrades or accuracy requirements increase, without the need for retraining.

[0030] The beneficial effects of this invention are:

[0031] 1. The dynamic balance between accuracy and efficiency:

[0032] By constructing nested multi-granularity visual representations from coarse to fine, this invention can adaptively allocate computing resources according to query complexity: simple queries use coarse-grained representations to reduce computational overhead, while complex queries use fine-grained representations to preserve local details, thereby significantly improving computational efficiency while ensuring retrieval accuracy.

[0033] 2. Differentiated collaborative optimization of multi-granularity features:

[0034] By forcibly mapping sample difficulty to granularity space, different granularity features perform their respective functions—coarse-grained features focus on distinguishing simple negative samples, while fine-grained features focus on distinguishing difficult negative samples. Combined with a dual-branch confidence evaluation mechanism, noise interference is effectively suppressed, achieving accurate alignment of text and images at multiple granular levels.

[0035] 3. Inference Adaptability and Deployment Flexibility:

[0036] An adaptive inference mechanism based on dynamic evaluation of retrieval confidence is proposed, which eliminates the redundant computation of fixed two-stage cascaded retrieval and allows simple queries to terminate the retrieval at the coarse-grained stage. At the same time, it supports lossless switching between efficiency-first and performance-first modes, overcomes the irreversible defects of traditional compression methods, and provides a flexible and scalable solution for edge deployment.

[0037] In summary, this invention transforms T2I-ReID from a static, fixed computing paradigm to an adaptive, on-demand computing paradigm, providing a scalable solution that balances accuracy and efficiency for practical deployment. Attached Figure Description

[0038] Figure 1 A comparison of existing methods with the method of this invention; Figure 1 In the text, (a) represents the existing method, and (b) represents the method of the present invention;

[0039] Figure 2 For the method global graph;

[0040] Figure 3 Visual modeling for Matryoshka;

[0041] Figure 4 Comparative learning for Matryoshka;

[0042] Figure 5 For efficiency analysis charts;

[0043] Figure 6 This is an example of adaptive retrieval based on multi-granularity contrastive learning. Detailed Implementation

[0044] I. Basic Framework

[0045] Matryoshka-ReID uses a CLIP-based two-stream architecture, such as... Figure 2 As shown, it includes image and text encoders. To achieve dynamic adaptation of query complexity and flexible switching between efficiency and performance, we introduce three core modules:

[0046] Matryoshka Visual Modeling (MVM): Constructing nested multi-granularity features {16,32,64,128,192}—simple queries use 16-32 tokens to capture the global appearance, while complex queries use 128-192 tokens to extract local details, achieving adaptive allocation of computing resources.

[0047] Matryoshka Contrastive Learning (MCL) encodes global semantics with coarse granularity and preserves local features with fine granularity. It achieves accurate alignment of text and images at an appropriate granularity through Grouped Contrastive Loss (GCL) and Confidence Consensus Divide (CCD).

[0048] Flexible Retrieval (FR): Supports one-click switching between efficiency and performance modes. Edge deployment uses coarse-to-fine cascading retrieval to reduce latency; after hardware upgrades, switching to the highest dimension fully restores performance—superior to traditional irreversible methods such as pruning and distillation.

[0049] 1.1 Token Representation

[0050] We use the Transformer encoder and ViT encoder from the CLIP ViT-B / 16 model as modality-specific encoders to extract text and image features, respectively.

[0051] Regarding text token representation, given a batch Input text We first segment the input text using BytePair Encoding (BPE). Then, we add special markers [SOS] and [EOS] to the beginning and end of the segmented sequence, respectively. The processed segmented sequence is then padded to a uniform length. According to the CLIP specification, It is set to 77. The preprocessed word segmentation sequence is input into the Transformer encoder to obtain the output batch set. :

[0052] ;

[0053] in Indicates the first Attention matrix of each text Indicates the first The global feature matrix of each text. Indicates the first The local text feature matrix of each text. This serves as a sample index, where the sample is either text or an image. The total number of samples.

[0054] Regarding image token representation, given a batch of input images Each image is first adjusted to Pixels, and are divided into Image patches of size . For a size of The image, total number of image blocks for:

[0055] ;

[0056] According to the CLIP specification, [CLS] tags are pre-added to learn global features. The combined sequence ([CLS] tags + (Number of image patches) are processed by the ViT encoder to obtain the output batch set. :

[0057] ;

[0058] in Indicates the first Attention matrix of images, Represents the global feature matrix. This represents the local characteristic matrix.

[0059] 1.2 Confidence Consistency Partitioning

[0060] To mitigate the impact of noise such as text-image mismatch and image quality defects in the training data, we employ a confidence consensus CCD partitioning method to evaluate sample quality. This method utilizes dual loss information from global and local branches, generating confidence weights for each image-text pair through a Beta mixture model. High-weighted samples are considered reliable matches and contribute more to subsequent loss calculations; low-weighted samples are adaptively downweighted, thereby suppressing noise interference in training. This dual-branch consensus mechanism is more robust than single-loss filtering and can effectively identify and reduce the negative impact of noisy samples.

[0061] II. Matryoshka Visual Modeling

[0062] like Figure 3 As shown, to achieve the goal of dynamically selecting the number of visual tokens based on the complexity of the text description, we designed MVM. This module solves two major problems of traditional fixed-length representations:

[0063] (1) Computational redundancy in simple queries (such as "people wearing red shirts");

[0064] (2) Insufficient detail in complex queries (such as "a person wearing glasses, a striped shirt, and a black backpack").

[0065] The core idea of ​​this module originates from MRL (Matryoshka Representation Learning), which constructs a multi-scale representation where the prefix sub-vectors of the high-dimensional embedding vector are themselves effective, low-dimensional representations. We generate a set of nested visual representations with different computational costs and representation granularities by dynamically selecting and processing the patch tokens output by the ViT backbone network.

[0066] Specifically, given an input image, the ViT encoder first converts it into a sequence of a [CLS] token and a series of image patch tokens. ,in is the number of patches, and D is the feature dimension.

[0067] Coarse-grained global representation: We directly use the Classification Token (CLS) to obtain the global representation through a linear mapping layer. It provides a coarse-grained description of the entire image, suitable for simple queries such as "wearing a red coat".

[0068] ;

[0069] in, This is the first element in the output sequence of the ViT encoder, namely the [CLS] token.

[0070] Fine-grained token-level representation: To obtain local features of different granularities to adapt to complex queries, we introduce the Matryoshka Representation Scale. This is a set of increasing cutoff lengths, where i is the granularity index, and m is the total number of granularities. For each granularity... We employ a Visual Token Selection (VTS) strategy, selecting the most important token based on its attention weight relative to all patch tokens (CLS). Each selected token is a patch token. These selected tokens are concatenated with the [CLS] token and then passed through a shared Transformer decoder layer (TSE-Layer) to ultimately generate a visual representation of the corresponding granularity. .

[0071] ;

[0072] ;

[0073] in, For the i-th granularity, the sequence of patch tokens selected by the VTS strategy; For the visual representation of the i-th granularity; This represents all elements from index 1 to the end; It is the sequence of all patch tokens, that is, the part of the ViT encoder output sequence after removing the [CLS] token.

[0074] Nested structure design: Due to the design of Matryoshka Representation Scale ( The selected subset of tokens is also nested, that is... This makes the final visual representation... , This is the top-level global representation, and it is the coarsest granularity. For progressive local representation, the granularity increases from coarse to fine; the final visual representation constitutes a nested representation structure with variable computational cost, ranging from coarse-grained global information to fine-grained local details, achieving:

[0075] Simple query optimization: Use 16-32 tokens to quickly capture the overall appearance and reduce computational costs.

[0076] Complex query support: Extract local details such as accessories and textures using 128-192 tokens.

[0077] Edge device adaptation: Flexibly adjust the number of tokens according to hardware conditions.

[0078] To jointly optimize these multi-granularity representations, we define a loss function for the output of each granularity. The overall training objective is a weighted sum of the losses from all granularities, which is the core idea of ​​MRL:

[0079] ;

[0080] in, It is the corresponding text representation. It is the loss function of the downstream task (the example uses Grouped Comparative Loss (GCL)). These are the weighting coefficients for the loss at each granularity. These are the weighting coefficients for the coarsest-grained loss in the global dataset. These are the weight coefficients of the i-th local granularity loss. This multi-task learning paradigm forces the model to learn discriminative features under different computational budgets, thus realizing the advantages of Matryoshka's nested representations.

[0081] III. Comparative Learning with Matryoshka

[0082] like Figure 4 As shown, in order to solve the problem of the difficulty in dynamically matching multi-granular semantics between text and images in traditional methods, we propose the MCL strategy.

[0083] This strategy achieves the following:

[0084] (1) Encode global semantics, such as human body outline and clothing color, in coarse-grained tokens;

[0085] (2) Preserve local features, such as logo and backpack shape, in fine-grained tokens;

[0086] (3) Make the text description and the image area accurately aligned at an appropriate granularity. When the text mentions the blue backpack, the model can focus on the backpack area by increasing the number of tokens, thus reducing background interference.

[0087] The core of this strategy is the Grouped Contrastive Loss (GCL), which interacts closely with the multi-granularity representations of the MVM module. It enhances the discriminative power of multi-granularity features by adaptively grouping them according to similarity and imposing differential contrast constraints. Furthermore, it incorporates confidence-based consensus-based CCD partitioning to further enhance robustness.

[0088] Given a batch of image-text pairs, we first calculate the normalized image features. and text features Similarity matrix between , is the a-th normalized image feature , is the b-th normalized text feature. .

[0089] Cooperative design of Grouped Contrast Loss (GCL) and multi-granularity representation: The core idea of ​​GCL is to adaptively divide training sample pairs into different groups based on their similarity in different granularity spaces, and impose differentiated contrastive constraints on different groups. This design deeply interacts with the multi-granularity representation generated by MVR.

[0090] coarse-grained space ( ): Includes the coarsest global representation and local coarsest-grained characterization It is mainly used to process simple negative samples and quickly separate obviously mismatched samples, such as gender and main clothing color differences.

[0091] Fine-grained space ( ): Focuses on difficult negative samples, finely distinguishing local attribute differences, such as accessory styles and texture details;

[0092] Medium-grained space: Handles samples of medium difficulty, achieving a smooth transition between coarse and fine granularity.

[0093] Specifically, it first bases its calculation on the global maximum value of intra-batch similarity. and minimum value Dynamically determine two thresholds, minimum threshold and maximum threshold :

[0094] ;

[0095] in It is a dynamic margin, for example Based on these two thresholds, negative sample pairs were divided into three groups:

[0096] Simple negative sample group Similarity is lower than The negative samples are mainly processed by coarse-grained features;

[0097] Medium negative sample group Similarity between and Negative samples between these points are processed using medium-granularity features;

[0098] Difficult negative sample group The similarity is higher than that of other similarities. Negative samples require fine-grained features for accurate differentiation.

[0099] Granularity-aware differential contrast constraints: GCL applies a different upper bound loss of TAL (Triplet Alignment Loss) to each group and adjusts the constraint strength according to the current granularity level. (Image-to-text direction) For example, for groups The loss corresponding to the current sample index b Defined as:

[0100] ;

[0101] in, Let be the adaptive weights in the direction from the b-th text to the a-th image. The similarity between the a-th image and the b-th text. It is the temperature coefficient; It is a negative sample mask. When the a-th sample and the b-th sample are a positive sample pair (matching text description and image content), the value is 0; when they are a negative sample pair, the value is 1. It is a specific set of margins, and exp is an exponential function. for Similarity after grouping. It is a specially processed similarity matrix that adjusts the weights of samples outside the current group based on the grouping, thereby achieving differentiated comparative learning. This indicates a truncation operation, taking the positive part.

[0102] Ultimately, the total GCL loss for a single sample It is the sum of the losses for all groups and all directions (image-to-text and text-to-image):

[0103] ;

[0104] To address the overall loss of MCL and multi-granularity collaborative optimization, we apply GCL to the representations of all granularities generated by MVM and use CCD to calculate the confidence weights. Weighting is performed to obtain the final MCL loss. :

[0105] ;

[0106] in, The confidence weights for global features. The confidence weight of the local i-th granularity.

[0107] This design enables the model to perform robust and targeted comparative learning at multiple granularities: coarse-grained features focus on global semantic matching, while fine-grained features are responsible for local detail alignment. This solves the problem of insufficient interaction between GCL loss and multi-granularity embedding in traditional methods, and achieves accurate dynamic alignment between text descriptions and image regions.

[0108] IV. Flexible Search

[0109] During the inference phase, we utilize nested multi-granularity representations generated by MVM to design an adaptive agile inference strategy based on predictive entropy (Matryoshka Agile Inference, MAI). The core innovation of this strategy lies in abandoning the traditional fixed two-stage cascaded retrieval and introducing an entropy-driven dynamic early stopping mechanism to achieve a precise match between computational resources and query complexity.

[0110] 4.1 Limitations of existing technology

[0111] Existing MRL-related works typically employ an adaptive retrieval method using a rigid two-pass strategy: the first stage uses coarse-grained features (such as low-dimensional vectors or a small number of tokens) to select a top-K candidate set from the entire graph database; the second stage unconditionally reorders this candidate set using the finest-grained features. This fixed process has the following problems:

[0112] Computational redundancy: For simple queries (such as "men wearing red shirts"), the results of the first stage may already be highly accurate, but the system still needs to perform the second stage of reordering, causing unnecessary computational overhead.

[0113] Lack of adaptability: The system cannot dynamically decide whether more granular calculations are needed based on the confidence level of the current search results;

[0114] Rigid granularity switching: It only supports two-stage switching from coarse to fine, and cannot flexibly progress between multiple granularity levels.

[0115] 4.2 The method of this invention: Adaptive Agile Inference (MAI) based on prediction entropy

[0116] This invention proposes a multi-level dynamic early stopping mechanism with algorithmic judgment capabilities, which constructs the retrieval process as a multi-granularity progressive adaptive loop, rather than a fixed-stage pipeline.

[0117] (1) Core Idea

[0118] The retrieval process begins with the coarsest-grained representation and progresses step by step. After each level of retrieval is completed, the system calculates the prediction entropy based on the similarity distribution of candidate results, thereby quantifying the prediction confidence of the current granularity. Only when the entropy value exceeds a preset threshold (indicating ambiguity in the candidate results) does the system automatically trigger the calculation of the next level of finer-grained representation; if the entropy value is below the threshold (indicating that the current results are sufficiently reliable), the retrieval is immediately terminated and the results are returned, achieving "early stopping." Figure 6 As shown.

[0119] (2) Specific implementation steps

[0120] Let the set of token counts corresponding to the nested multi-granularity representations generated by MVM be . , Indicates the first The number of tokens at each granularity level, of which In this embodiment Total number of particles =5, the finer the granularity, the larger the index.

[0121] Step 1: Initialize the search

[0122] Using the coarsest-grained local characterization ( =16 tokens) to perform fast similarity calculation on all images in the image library to obtain an initial similarity score vector. This step can quickly filter out a large number of obviously irrelevant candidates, such as coarse-grained differences like gender or main clothing color.

[0123] Step 2: Candidate Set Screening and Entropy Evaluation

[0124] from The top-K candidates are selected to form a candidate set (in this embodiment, the candidate set size K=100). Then, the predicted entropy of the similarity distribution is calculated for the top-N results in this candidate set (in this embodiment, the entropy evaluation window size N=10). This is to assess the reliability of the prediction results at the current granularity. This represents the similarity distribution entropy value of the top N candidate results at the i-th granularity, which is the confidence level of the search result at granularity i.

[0125] ;

[0126] in, Let be the similarity score of the j-th candidate at the i-th granularity. Temperature coefficient (in this embodiment) =0.1), Let be the normalized probability distribution, representing the probability value of the j-th ranked candidate at the i-th granularity after softmax normalization. In other words, it represents the probability weight of the j-th candidate result relative to the total similarity of the top N results at granularity i. Prediction entropy This reflects the determinism of the similarity distribution: the smaller the entropy value, the more obvious the difference in similarity between the top-N candidates, and the higher the confidence of the candidate ranked first; the larger the entropy value, the closer the similarity of multiple candidates, the more ambiguous they are, and the more granular features are needed to distinguish them.

[0127] Step 3: Adaptive Early Stopping Decision

[0128] The calculated entropy value With preset threshold Compare the results and decide whether to terminate the search based on the comparison outcome:

[0129] Early stopping branch: If Determine the current granularity The search results are sufficiently discriminative, and the confidence level of the top-ranked candidate is high enough that further calculations are unnecessary. At this point, the ranking of the current candidate set is returned, and the search terminates.

[0130] Progressive branch: If Determine the current granularity The search results are ambiguous; the similarity of the top N candidates is too close to be reliably distinguishable. In this case, the system automatically selects the next level of finer-grained representation and proceeds to step four for rearrangement.

[0131] Among them, threshold The settings can be flexibly configured according to the application scenario. This embodiment is configured as follows: This threshold can be dynamically adjusted based on hardware conditions and accuracy requirements: lowering the threshold can increase the early stopping rate and reduce the amount of computation (efficiency first); raising the threshold can reduce early stopping and increase the proportion of fine-grained retrieval (accuracy first).

[0132] Step 4: Granularity Progression and Cyclic Rearrangement

[0133] If a progressive branch is entered, the system will proceed with granularity progression according to the following process:

[0134] (1) Granularity switching

[0135] Select the next level of finer-grained characterization. Corresponding token quantity In this embodiment, the particle size progression order is as follows: .

[0136] (2) Rearrangement calculation

[0137] Use new granularity The representation For the current candidate set Recalculate the similarity score:

[0138] ;

[0139] (3) Cyclic evaluation

[0140] Updated similarity score As input, return to step two to re-select the candidate set and evaluate the entropy (candidate set). (Keep it unchanged, only update the similarity score), that is:

[0141] Calculate the prediction entropy from the top-N results. ;

[0142] Proceed to step three, Compare with a threshold;

[0143] (4) Termination conditions

[0144] This loop continues until one of the following conditions is met:

[0145] Early termination: at a certain particle size Down, The search has ended.

[0146] Granularity exhaustion termination: reaching the finest granularity (In this embodiment) At this point, regardless of the entropy value, the current sorting result will be forcibly output.

[0147] This flexible, coarse-to-fine retrieval paradigm perfectly aligns with the core advantages of Matryoshka's nested representations, allowing users to freely balance inference efficiency and retrieval accuracy based on specific application scenarios. Crucially, unlike traditional model compression methods, our approach supports lossless performance recovery, a unique advantage that pruning, distillation, and other techniques cannot achieve, providing unprecedented flexibility for practical deployments.

[0148] Unlike irreversible compression techniques such as pruning or distillation (which permanently sacrifice performance for efficiency), Matryoshka-ReID supports post-deployment adjustments without the need for retraining. It can seamlessly switch between high-performance and high-efficiency modes, ensuring full performance recovery during hardware upgrades—a key advantage for evolving real-world applications.

[0149] Experiments on CUHK-PEDES, ICFG-PEDES, and RSTPReid demonstrate that this method achieves effective performance while reducing computational cost by 92.75%.

[0150] The performance of experiments conducted on CUHK-PEDES is shown in Table 1:

[0151] Table 1

[0152] Method Reference R@1 R@5 R@10 mAP mINP SSAN ACMMM2021 61.37 80.15 86.73 – – IVT ECCV2022 65.59 83.11 89.21 – – CFine TIP2023 69.57 89.53 91.15 – – TBPS AAAI2024 73.54 88.19 92.35 65.38 49.25 IRRA CVPR2023 73.38 89.93 93.71 66.13 50.24 SAMC TIFS2024 74.03 89.18 93.31 68.42 55.01 ICLF VC2025 74.35 89.96 94.24 66.76 51.44 BCRA MS2025 75.03 89.93 93.84 66.71 50.31 RDE CVPR2024 75.94 90.14 94.12 67.56 51.44 Matryoshka Ours (192) Ours(192) 76.07 90.68 94.33 67.59 51.79 Matryoshka Ours(16) Ours(16) 74.22 89.82 93.65 66.18 49.02 Matryoshka Ours (16→192) Ours (16→192) 76.07 90.68 94.33 67.44 51.89

[0153] Performance comparisons on three benchmark datasets. Best results are highlighted in bold. R@K represents Rank-K accuracy (%). Our method supports flexible granularity: on CUHK-PEDES, full performance (67.34 GFLOPs) is achieved using 192 tokens, or efficient mode (4.88 GFLOPs, 92.75% reduction in computation) is achieved using 16 tokens.

[0154] Efficiency advantages such as Figure 5 The efficiency at the same particle size is shown in Table 2:

[0155] Table 2

[0156] Tokens R@1 mAP FLOPs (G) ↓ (%) 192 76.07 67.59 67.34 – 128 75.54 67.29 43.85 34.88 64 74.89 67.10 20.81 69.10 32 74.74 66.40 9.83 85.40 16 74.22 66.18 4.88 92.75

[0157] To verify the practical application value of the proposed method, we conducted an in-depth analysis of the trade-off between its efficiency and performance, and the results are as follows: Figure 5 As shown in the diagram, the experiments clearly demonstrate that Matryoshka-ReID achieves an excellent dynamic balance between efficiency and performance thanks to its nested, multi-granular design. When pursuing the highest accuracy, the model uses all 192 tokens, achieving 76.07% Rank-1 accuracy and 67.59% mAP with a computational cost of 67.34 GFLOPs. However, in high-efficiency scenarios, the model seamlessly switches to a lightweight mode using only 16 tokens, significantly reducing computation by 92.75% to just 4.88 GFLOPs, while Rank-1 accuracy and mAP only slightly decrease to 74.22% and 66.18%, respectively. This adaptive allocation of computational resources based on query complexity allows Matryoshka-ReID to be flexibly deployed between high accuracy and high efficiency, depending on different hardware resources and latency requirements, without the need for retraining. This core advantage is difficult to achieve with traditional fixed-computation models.

Claims

1. An adaptive text-image semantic alignment method based on nested multi-granularity representation, characterized in that, Includes the following steps: Step 1: Constructing multi-granularity nested visual representations The input image is encoded using a visual Transformer into a sequence containing a classification token and multiple image patch tokens. ; Based on the attention weights of the classification token and each image patch token, a visual token selection strategy is used to dynamically select tokens from all image patch tokens. Given a number of image block tokens, obtain the selected token sequence. ,in A set of increasing cutoff lengths The i-th value in the array; i is the granularity index, m is the total number of granularities; the finer the granularity, the larger the index. The classifying token and the selected token sequence are concatenated and then input into a shared Transformer decoder layer to generate a sequence corresponding to the truncation length. Local visual representation ; The classification tokens are linearly mapped to obtain a global visual representation. ; This yields nested, multi-granular visual representations. ; Step 2: Multi-granularity comparative learning The similarity between the text representation and the multi-granularity visual representation is calculated respectively, and the negative samples are divided into simple negative sample group, medium negative sample group and difficult negative sample group according to the similarity. Different contrast loss constraints are applied to each group of negative samples, where the loss corresponding to the coarse-grained visual representation focuses on the simple negative sample group, and the loss corresponding to the fine-grained visual representation focuses on the difficult negative sample group. During training, a confidence consensus partitioning mechanism is used to generate confidence weights for each image-text pair, and the contrast loss at each granularity is weighted using the confidence weights. Step 3: Adaptive Search During the inference phase, the coarsest-grained local visual representation is selected from the multi-granularity visual representations. A preliminary search of the image library was performed to obtain a candidate set; For the current granularity i, calculate the similarity distribution entropy of the top N results in the candidate set. ; The entropy Compared with the preset threshold: If If the value is less than the threshold, the search is terminated and the current sorting result is output; if... If the value is not less than the threshold, then switch to the next finer-grained visual representation. The similarity of the candidate set is recalculated and the above entropy comparison process is repeated until the termination condition is met or the finest granularity is reached.

2. The method as described in claim 1, characterized in that, The visual token selection strategy is as follows: Based on the attention weights of the classification token and each image patch token, the token with the highest weight is selected. Each image block token is used as the sequence of selected tokens.

3. The method as described in claim 1, characterized in that, The shared Transformer decoder layer is a single-layer Transformer decoder that includes a multi-head self-attention mechanism and a feedforward neural network.

4. The method as described in claim 1, characterized in that, The set of increasing cut-off lengths Values The corresponding total number of particle sizes is m=5.

5. The method as described in claim 1, characterized in that, Negative samples are divided into three groups: simple negative samples, moderate negative samples, and difficult negative samples. Based on the maximum similarity within the batch and minimum value and dynamic margins Determine two thresholds, the minimum threshold. and maximum threshold , ; Similarity lower Negative samples are classified into the simple negative sample group, with similarity between [missing information]. and Negative samples between these groups are classified into the intermediate negative sample group, which has a similarity higher than [missing information]. Negative samples are classified into the difficult negative sample group.

6. The method as described in claim 1, characterized in that, The confidence consensus partitioning mechanism is as follows: using the dual loss information of the global branch and the local branch, a confidence weight is generated for each image-text pair through a Beta mixture model. High-weight samples contribute more to the loss calculation, while low-weight samples are adaptively downweighted.

7. The method as described in claim 1, characterized in that, The similarity distribution entropy The calculation method is as follows: ； in, This represents the confidence level of the search results at granularity i. Let be the similarity score of the j-th candidate at the i-th granularity. For temperature coefficient, Let be the normalized probability value of the candidate ranked j at the i-th granularity, where j and k are both candidate ranking indices.

8. The method as described in claim 1, characterized in that, The termination condition for the search is: the entropy of the current granularity. The value is less than the preset threshold, or the current granularity has reached the finest granularity.

9. The method as described in claim 1, characterized in that, The method supports selecting either an efficiency-first mode or a performance-first mode during the inference phase based on hardware conditions or accuracy requirements; in efficiency-first mode, the adaptive retrieval step is executed; in performance-first mode, the finest-grained visual representation is used directly. Perform a one-time search of the image library.

Images