Weak alignment positive sample enhancement method and system for textpersonnel search
By identifying and dynamically increasing the weights of weakly aligned positive samples, the problem of ignoring alignment errors in existing methods is solved, improving the model's performance on real-world datasets, especially in terms of rank-1 accuracy and average precision.
Patent Information
- Application Number
- CN202511216881.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-08-28
- Publication Date
- 2025-11-21
AI Technical Summary
Existing text-based people search methods are limited in performance when dealing with weakly aligned positive samples, ignoring position alignment errors in real-world multimodal data, resulting in poor model performance on real-world datasets.
By identifying weakly aligned positive samples and dynamically increasing their weights, an enhancement loss function is designed to optimize model parameters, making the model pay more attention to these difficult samples. By utilizing the alignment of image and text features in the shared semantic space, an image encoder and a text encoder are used to generate embedding vectors. The similarity calculation module and the enhancement module are combined to identify and increase the weights of weakly aligned positive samples.
It improves the model's rank-1 accuracy and average precision on the benchmark TBPS dataset, enhances the model's robustness, and improves the alignment of image and text embeddings.
Smart Images
Figure CN120994857A_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of computer vision technology, specifically relating to a weakly aligned positive sample enhancement method and system for text-based people search. Background Technology
[0002] Text-Based Person Search (TBPS) aims to identify specific individuals in large image datasets using free-form natural language descriptions. This method provides a practical solution for scenarios such as surveillance, especially when visual references of the target person are lacking. Therefore, utilizing diverse natural language descriptions for person identification has become an important research direction.
[0003] In recent years, significant progress has been made in large-scale visual language pre-trained models (VLMs). These models are trained on massive datasets to establish a unified semantic space and achieve effective alignment between image and text embeddings. Depending on the training strategy, they can be categorized into adding new parameters and training from scratch, or partially or completely fine-tuning a large backbone network to optimize performance. Existing VLM-based methods typically employ a standard contrastive loss framework, bringing positive image-text pairs closer together while pushing negative embeddings further apart within a shared semantic space.
[0004] The TBPS task is inherently challenging due to the nature of its annotation process, where descriptive text can be ambiguous and image quality can be poor. Therefore, dynamic weighting or filtering processes have become common methods for handling noisy data points during model training. For example, RDE addresses the noise characteristics of data by introducing a noise correspondence process that dynamically weights outlier image-text pairs and treats them as noise to reduce their contribution to training. Existing methods primarily focus on aligning images and text to a common representation space, often ignoring the fact that positive image-text pairs in the real world exhibit varying degrees of similarity. This leads models to prioritize simpler sample pairs, and in some state-of-the-art methods, challenging samples are even discarded as noise during training.
[0005] Despite significant progress in architecture design and alignment loss in existing research, less attention has been paid to the characteristics of real-world people datasets. While methods have pointed out the variability and diversity of such data and designed noise filtering systems, the lack of clean data and the inherent variance of real-world data in the TBPS domain make handling weakly aligned positive image-text pairs crucial. Most existing multimodal methods detect targets under the alignment assumption that image pairs from different modalities are well aligned and have strong pixel-to-pixel correspondence, ignoring the position shift problem, or weak alignment problem, that is prevalent in real-world multimodal data. Summary of the Invention
[0006] To address the shortcomings of existing text-based people search processes in handling challenging weakly aligned positive samples, which limits model performance, this invention provides a weak positive sample enhancement system for text-based people search. This system dynamically assigns higher weights to correct text-image pairs (i.e., weak positive samples) that existing models fail to classify as best matches. In machine learning and object detection, positive samples refer to samples that are highly correlated with the ground truth and are primarily used to train models to identify specific targets.
[0007] To address the above problems, the present invention adopts the following technical solution: a weakly aligned positive sample enhancement method for text-based people search, comprising:
[0008] Obtain the image-text pair dataset, which includes images, corresponding text descriptions, and identity tags;
[0009] Extract image and text features from the image-text pair data, and encode the image and text features into a shared semantic embedding space. The image and text features are then used as embedding vector features.
[0010] Identifying weakly aligned positive sample pairs: For a given query text or image, if in the similarity retrieval results of another modality, the sample ranked first is an incorrect match, while the sample ranked k is a correct match, then the correct match sample pair ranked k is defined as a weakly aligned positive sample, where k>1.
[0011] Weight boosting is applied to the identified weakly aligned positive sample pairs;
[0012] Create a loss function to train the model, which takes into account weight boosting;
[0013] By iteratively optimizing the loss function and updating the model parameters, previously misranked, correctly matched weak positive samples can be more accurately ranked to higher positions. A sample refers to a single data entity, which can be an image or a text description. A sample pair refers to a pair consisting of an image and its corresponding text description.
[0014] As an improvement, the weights are increased to dynamically updated weights for weakly aligned positive sample pairs, giving them a larger weight in the loss calculation.
[0015] As an improvement, the process of dynamically updating the weights of weakly aligned positive sample pairs includes:
[0016] Initialize the weights of all sample pairs to 1;
[0017] For the identified weakly aligned positive sample pairs, their weights are assigned as exp(α), where exp(α) is greater than 1, while the weights of other sample pairs that do not match the query identity remain at 1.
[0018] Weight normalization is not performed during the weight update process.
[0019] As an improvement, exp(α) is approximately equal to 1.6.
[0020] As an improvement, the calculation process of the weighted loss function includes text-to-image retrieval loss and image-to-text retrieval loss.
[0021] As an improvement, the calculation formula for the text-to-image retrieval loss is as follows:
[0022]
[0023] Where B is the batch size. The boost weight for the i-th sample pair in the batch. and , respectively, are the embedding vectors of the image and the text, sim represents the similarity calculation, and τ is the temperature hyperparameter;
[0024] The formula for calculating the loss in image-to-text retrieval is as follows:
[0025]
[0026] The final combined loss function is:
[0027] As an improvement, for ID loss, an enhanced version of the loss function that considers weight boosting is:
[0028]
[0029] For the SDM loss, the enhanced version of the loss function that considers weight boosting is:
[0030]
[0031] As an improvement, k = 2.
[0032] As an improvement, the allocation of boost weights is performed dynamically and updated periodically over multiple training epochs; the boost weights are updated once every 4 training epochs.
[0033] Weakly aligned positive sample augmentation systems for text-based people search include:
[0034] An image encoder configured to generate multiple image embeddings from multiple images;
[0035] A text encoder configured to generate multiple text embeddings from multiple text descriptions, wherein the image embeddings and text embeddings reside in a shared semantic space;
[0036] A similarity calculation module is configured to determine a similarity score between a query embedding selected from the plurality of image embeddings or the plurality of text embeddings and a library embedding from another modality of the plurality of embeddings;
[0037] The enhancement module is configured to: (i) identify weak positive sample pairs based on the similarity score, wherein the weak positive sample pair includes a query embedding and a graph embedding that correctly matches the query, the graph embedding being ranked rank k relative to the query embedding, where k>1, and the graph embedding ranked rank-1 relative to the query embedding is a mismatch of the query; and (ii) assign an enhancement weight to the weak positive sample pair, the enhancement weight being greater than the default weight.
[0038] The training module is configured to update the parameters of the text-based people search model by optimizing a loss function that includes the augmentation weights assigned to weak positive sample pairs, thereby increasing the contribution of the weak positive sample pairs to the loss function.
[0039] The beneficial effects of the weakly aligned positive sample augmentation method for text-based people search in this invention are: dynamically assigning higher weights to correct image-text pairs that the model fails to classify as best matches for weakly aligned positive samples, thereby guiding the model to pay more attention to these difficult samples by increasing the contribution of these misranked positive sample pairs to the training loss; enhancing the robustness of the model by improving the alignment of image and text embeddings in the shared representation space; and achieving performance improvements, including higher rank-1 accuracy and mean average precision, on the benchmark TBPS dataset. Attached Figure Description
[0040] Figure 1 This is a flowchart of a weakly aligned positive sample enhancement method for text-based people search according to Embodiment 1 of the present invention.
[0041] Figure 2 This is an architecture diagram of a weakly aligned positive sample enhancement system for text-based people search according to Embodiment 1 of the present invention.
[0042] Figure 3 This is an ablation result diagram of key parameters (such as the enhancement rank k, enhancement coefficient exp(α), and weight update period) in the weakly aligned positive sample enhancement method for text-based people search according to Embodiment 1 of the present invention, used to illustrate the basis and influence of parameter selection.
[0043] Figure 4This is a comparison chart of the qualitative results of image retrieval using the baseline model (CLIP) and the model of this invention (CLIP+B). The green box indicates a correct match, showing that the correct match, which was originally ranked as rank-2, was improved to rank-1 after being enhanced by the method of this invention. Detailed Implementation
[0044] The technical solutions of the embodiments of the present invention will be explained and described below. However, the following embodiments are only preferred embodiments of the present invention and not all of them. Other embodiments obtained by those skilled in the art based on the embodiments in the implementation methods without creative effort are all within the protection scope of the present invention.
[0045] See Figures 1 to 4 The present invention provides a weakly aligned positive sample enhancement method for text-based people search, comprising:
[0046] Obtain the image-text pair dataset, which includes images, corresponding text descriptions, and identity tags;
[0047] Extract image and text features from the image-text pair data, and encode the image and text features into a shared semantic embedding space. The image and text features are then used as embedding vector features.
[0048] Identifying weakly aligned positive sample pairs: For a given query text or image, if in the similarity retrieval results of another modality, the sample ranked first is an incorrect match, while the sample ranked k is a correct match, then the correct match sample pair ranked k is defined as a weakly aligned positive sample, where k>1.
[0049] Weight boosting is applied to the identified weakly aligned positive sample pairs;
[0050] Create a loss function to train the model, which takes into account weight boosting;
[0051] By iteratively optimizing the loss function and updating the model parameters, correctly matched weak positive samples that were previously misranked can be more accurately ranked to higher positions.
[0052] The weakly aligned positive sample augmentation method for text-based people search in this embodiment dynamically assigns higher weights to the correct image-text pairs that the model fails to classify as best matches. By increasing the contribution of these misranked positive sample pairs to the training loss, the model is guided to pay more attention to these difficult samples. The robustness of the model is enhanced by improving the alignment of image and text embeddings in the shared representation space. Performance improvements, including higher rank-1 accuracy and mean average precision, are achieved on the benchmark TBPS dataset.
[0053] Example 1
[0054] See Figures 1 to 4 The weakly aligned positive sample enhancement method for text-based people search in Embodiment 1 of the present invention includes:
[0055] Obtain the image-text pair dataset, which includes images, corresponding text descriptions, and identity tags;
[0056] Extract image and text features from the image-text pair data, and encode the image and text features into a shared semantic embedding space. The image and text features are then used as embedding vector features.
[0057] Identifying weakly aligned positive sample pairs: For a given query text or image, if in the similarity retrieval results of another modality, the sample ranked first is an incorrect match, while the sample ranked k is a correct match, then the correct match sample pair ranked k is defined as a weakly aligned positive sample, where k>1.
[0058] Weight boosting is applied to the identified weakly aligned positive sample pairs;
[0059] Create a loss function to train the model, which takes into account weight boosting;
[0060] By iteratively optimizing the loss function and updating the model parameters, correctly matched weak positive samples that were previously misranked can be more accurately ranked to higher positions.
[0061] See Figure 2 The weakly aligned positive sample enhancement system for text-based people search in this embodiment begins with the feature extraction stage, that is, using an image encoder and a text encoder to transform the input image f. img and text description f txt Converted into a high-dimensional embedding vector in a shared semantic space. and Subsequently, the system uses these embedding vectors to calculate a similarity matrix to quantify the degree of matching between text and images. Based on the similarity score of this matrix, the reinforcement learning module retrieves and ranks a series of candidate images according to the given text description. During this process, the system identifies correctly matched images that are misranked (i.e., weak positive samples) and dynamically assigns them an enhancement weight. Finally, this vector containing the enhancement weight is fed into the training objective (loss function) along with the similarity matrix, thereby forcing the model to pay more attention to these difficult samples during model optimization to improve retrieval accuracy.
[0062] Specifically, the weakly aligned positive sample enhancement method for text-based people search in this embodiment first obtains the image-text pair dataset, represented as:
[0063]
[0064] Among them, I i Let T represent the i-th image in the dataset. i It corresponds to I i The text description, y i It is the category label or pedestrian identity label, and N is the total number of training samples.
[0065] Then, feature extraction. For each image I i via image encoder f img (e.g., CLIP ViT backbone network) obtains its embedding vector in Similarly, for each text T i via text encoder f txt (e.g., CLIP, BERT) generate text embedding vectors in The CLS token embeddings of a pre-trained CLIP model can be used as embedding vectors for both images and text. The input image patch size can be set to 384×128, the hidden dimension to 512, and the maximum text context length to 77 tokens.
[0066] Secondly, weak positive samples are identified and their weights are augmented. Drawing inspiration from AdaBoost (Adaptive Boosting, a classic ensemble learning algorithm proposed by Yoav Freund and Robert Schapire in 1995), we consider the cases of a single query and a graph set. Let q be the query, and g1, g2, ..., g K These are image samples from the library, sorted in descending order of similarity to q. Assume g... K g1 is a correct match (positive sample) at position k (k>1), while g1 is a wrong match (negative sample) at position 1.
[0067] This embodiment aims to emphasize the weights of these misranked positive sample pairs in order to push them towards rank-1. At the beginning of each training epoch, the sample pairs (q, g) are initialized. i The weight w i The value is 1. Then, the sample pair {(q,g} is defined. i The weight update rule for} is: Set exp(α) = 1.6. Therefore, this invention enhances samples that have a correct match at rank-k but an incorrect match at rank-1. For example, sample pair (q, g) k ) will obtain w kThe weights are updated to exp(α), while the weights of all other sample pairs not shared with the query q label remain at 1. Unlike AdaBoost, this embodiment does not perform weight normalization.
[0068] When processing a set of queries and a library sample, let T i To query the text, I i It is T i The image corresponding to rank-k (k>1). Let the image of rank-1 be I. i y Ii Define a set for its labels.
[0069]
[0070] The enhancement coefficient wk(i) corresponding to rank-k is dynamically assigned to emphasize the contribution of these rank-k samples during training, and is defined as follows:
[0071] wk(i)={exp(α)1if(Ti,Ii)∈Rkotherwise.
[0072] This formula ensures that the contribution of the rank-k sample is increased while maintaining the natural contribution of all other samples.
[0073] Optimal performance is achieved by enhancing only the case where k=2; performance degrades beyond k=2. Therefore, k=2. Weights are updated every four cycles.
[0074] Next, we design and apply training objectives that enhance perception. During training, we dynamically adjust the enhanced image-text contrastive (ITC) loss to ensure that the model can learn naturally from non-enhanced samples while focusing on challenging samples.
[0075] For text-to-image (t2i) retrieval, the enhanced perception ITC loss is defined as:
[0076]
[0077] in, By using samples in batch B and set R k The part obtained from wk(i) that establishes the correspondence between identical sample pairs in the batch represents the boost weight of the i-th sample pair in the batch, and τ is the temperature hyperparameter.
[0078] For image-to-text (i2t) retrieval, the loss is defined as:
[0079]
[0080] This determines the coefficient of each positive sample pair, therefore, they belong to set R. k The weights of these samples are enhanced. As a result, these samples contribute more to the loss, and the network pays more attention to them to improve its ranking.
[0081] The final formula for calculating the combined loss is as follows:
[0082]
[0083] In other embodiments, other loss functions are also used, such as ID loss and SDM loss. ID loss is a loss function used in deep learning for person re-identification (ReID) tasks, primarily used to train models to distinguish different pedestrian identities. Its core idea is to use pedestrian IDs as training labels and optimize the model's feature extraction capability through cross-entropy loss. SDM (Semantic Diffusion Model) is mainly used for image generation tasks. Its core is to generate high-quality, diverse images through a diffusion model and introduce semantic masks and normalization mechanisms to improve semantic expressiveness.
[0084] For ID loss, given a set of image-text pairs and their labels... The original loss is:
[0085]
[0086] In conjunction with the enhancement module of this embodiment, the loss is modified as follows:
[0087]
[0088] Similarly, the ID loss of the text can be calculated.
[0089] The enhanced SDM loss can be derived as follows:
[0090] L IRRA+B =L IRR+B +L SDM+B +L ID+B
[0091]
[0092] This scalability allows the enhancement method in this embodiment to be seamlessly integrated with existing model architectures employing different loss functions (such as IRRA and RDE). For example, the update objective after integration with IRRA is:
[0093] L IRRA+B =L IRR +L SDM+B +L ID+B .
[0094] For RDE, its original TAL can be enhanced in a similar way.
[0095] Finally, model training and evaluation were performed. The model was trained for 60 epochs with an initial learning rate of 1e-5. The method of this embodiment was tested on several publicly available pedestrian datasets, including CUHK-PEDES, ICFG-PEDES, RSTPReid, and IIITD-20K, using rank-k (1, 5, 10) accuracy and mean average precision (mAP) as evaluation metrics. The experimental results are shown in Table 1.
[0096]
[0097] In Table 1, "+B" * " indicates that it belongs to set R k The algorithm enhances weak positive samples. "+B" refers to a more comprehensive enhancement version that enhances not only "hard samples" (correct matches that have been incorrectly corrected) but also "easy samples" (correct matches that have already been paired). Bold text indicates best-case results. Comparison with current state-of-the-art methods.
[0098] Experimental results show that the enhanced models CLIP+B, IRRA+B, and RDE+B proposed in this embodiment demonstrate consistent performance improvements on all four datasets compared to their respective baseline models and other existing methods. For example, CLIP+B improves the rank-1 accuracy on CUHK-PEDES by 2.89% compared to the baseline CLIP and by 4% on ICFG-PEDES. RDE+B achieves an R1 accuracy of 76.12% on CUHK-PEDES and 65.55% on RSTPReid, outperforming all other methods.
[0099] See Figure 3 The ablation experiment verified the rationality of the parameter selection. Choosing k=2 for enhancement resulted in an enhancement coefficient exp(α)=1.6 (i.e., e...). α The algorithm involves updating the weights every four epochs. First, the optimal "boost rank k" is determined using the top-left chart: the model's R1 accuracy peaks when the boost target is limited to the second-ranked (k=2) weak positive sample. Second, the "boost coefficient exp(α)" is further analyzed using the top-right chart, determining that 1.6 is the optimal weight boost strength for achieving optimal model performance. Finally, the chart below explores the "weight update cycle," showing that updating the boost weights every four training epochs achieves the highest accuracy. In summary, these rigorous experiments provide solid empirical evidence for the key parameter configuration of the method in this embodiment.
[0100] See Figure 4 This example visually demonstrates the comparison between the enhancement method before and after enhancement: In both examples, the baseline model CLIP (Contrastive Language–Image Pre-training, a multimodal neural network model developed by OpenAI) ranks the correct matching error as rank-2, while the model CLIP+B (an enhanced version of CLIP) in this example improves the correct matching error ranked as rank-2 to rank-1.
[0101] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Those skilled in the art should understand that the present invention includes, but is not limited to, the content described in the above specific embodiments. Any modifications that do not depart from the functional and structural principles of the present invention will be included within the scope of the claims.
Claims
1. A weakly aligned positive sample augmentation method for text-based people search, characterized in that: Weakly aligned positive sample augmentation methods for text-based people search include: Obtain the image-text pair dataset, which includes images, corresponding text descriptions, and identity tags; Extract image and text features from the image-text pair data, and encode the image and text features into a shared semantic embedding space. The image and text features are then used as embedding vector features. Identifying weakly aligned positive sample pairs: For a given query text or image, if in the similarity retrieval results of another modality, the sample ranked first is an incorrect match, while the sample ranked k is a correct match, then the correct match sample pair ranked k is defined as a weakly aligned positive sample, where k>1. Weight boosting is applied to the identified weakly aligned positive sample pairs; Create a loss function to train the model, which takes into account weight boosting; By iteratively optimizing the loss function and updating the model parameters, correctly matched weak positive samples that were previously misranked can be more accurately ranked to higher positions.
2. The weakly aligned positive sample enhancement method for text-based people search according to claim 1, characterized in that: The weights are increased by dynamically updating the weights of weakly aligned positive sample pairs, giving them a larger weight in the loss calculation.
3. The weakly aligned positive sample enhancement method for text-based people search according to claim 2, characterized in that: The process of dynamically updating the weights of weakly aligned positive sample pairs includes: Initialize the weights of all sample pairs to 1; For the identified weakly aligned positive sample pairs, their weights are assigned as exp(α), where exp(α) is greater than 1, while the weights of other sample pairs that do not match the query identity remain at 1. Weight normalization is not performed during the weight update process.
4. The weakly aligned positive sample enhancement method for text-based people search according to claim 3, characterized in that: exp(α) is approximately equal to 1.
6.
5. The weakly aligned positive sample enhancement method for text-based people search according to claim 1, characterized in that: The calculation process of the loss function considering weight enhancement includes text-to-image retrieval loss and image-to-text retrieval loss.
6. The weakly aligned positive sample enhancement method for text-based people search according to claim 5, characterized in that: The formula for calculating the loss in text-to-image retrieval is as follows: Where B is the batch size. The boost weight for the i-th sample pair in the batch. and , respectively, are the embedding vectors of the image and the text, sim represents the similarity calculation, and τ is the temperature hyperparameter; The formula for calculating the loss in image-to-text retrieval is as follows: The final combined loss function is:
7. The weakly aligned positive sample enhancement method for text-based people search according to claim 1, characterized in that: For ID loss, the enhanced version of the loss function that considers weight boosting is: For the SDM loss, the enhanced version of the loss function that considers weight boosting is:
8. The weakly aligned positive sample enhancement method for text-based people search according to claim 1, characterized in that: k＝2。 9. The weakly aligned positive sample enhancement method for text-based people search according to claim 1, characterized in that: The allocation of boost weights is performed dynamically over multiple training cycles and updated periodically; the boost weights are updated every 4 training cycles.
10. A weakly aligned positive sample augmentation system for text-based people search, characterized in that: include: An image encoder configured to generate multiple image embeddings from multiple images; A text encoder configured to generate multiple text embeddings from multiple text descriptions, wherein the image embeddings and text embeddings reside in a shared semantic space; A similarity calculation module is configured to determine a similarity score between a query embedding selected from the plurality of image embeddings or the plurality of text embeddings and a library embedding from another modality of the plurality of embeddings; The enhancement module is configured to: (i) identify weak positive sample pairs based on the similarity score, wherein the weak positive sample pair includes a query embedding and a graph embedding that correctly matches the query, the graph embedding being ranked rank k relative to the query embedding, where k>1, and the graph embedding ranked rank-1 relative to the query embedding is a mismatch of the query; and (ii) assign an enhancement weight to the weak positive sample pair, the enhancement weight being greater than the default weight. The training module is configured to update the parameters of the text-based people search model by optimizing a loss function that includes the augmentation weights assigned to weak positive sample pairs, thereby increasing the contribution of the weak positive sample pairs to the loss function.