Pedestrian search method based on cross-modal consistency knowledge transfer of text description

Abstract

The application discloses a kind of pedestrian search methods based on text description cross-modal consistency knowledge transfer, it is related to image processing method technical field.The method comprises the following steps: using cross-modal knowledge transfer CMKT method is fine-tuned on the cutting good pedestrian picture-text pair contrast language-picture pretraining CLIP network, obtains an identity retrieval teacher model with cross-modal alignment capability;Then, cross-modal knowledge transfer CMKT is further migrated to end-to-end pedestrian search network by feature distillation with the cross-modal alignment capability of identity retrieval teacher model;Pedestrian picture-text pair is searched using pedestrian search network. The method can narrow the instance representation gap in end-to-end pedestrian search network, improve the accuracy of pedestrian search.

Description

Technical Field

[0001] This invention relates to the field of image processing methods, and in particular to a pedestrian search method based on cross-modal consistent knowledge transfer using text description. Background Technology

[0002] In text-based pedestrian search scenarios, inconsistencies often exist between the images and text descriptions corresponding to different instances of the same pedestrian. For example... Figure 1 As shown, pedestrian images and their labeled text have a complete and accurate correspondence, thus forming strong positive sample pairs. For example, (Image 1, Text 1) and (Image 2, Text 2) are two sets of strong positive sample pairs. However, there are often inconsistencies between the images and text corresponding to different instances of the same person. For example, the "white shoulder bag" in Image 2 has no corresponding description in Text 1, and the "brown shoes" in Text 2 is also missing in Image 1. Therefore, (Image 1, Text 2) and (Image 2, Text 1) constitute two sets of weak positive sample pairs. These weak positive sample pairs severely affect the model's cross-modal alignment performance, thus posing a significant challenge to text-based pedestrian search tasks. For instance, the InfoNCE loss ignores the modeling of weak positive sample pairs during training, causing the model to overfit to strong positive sample pairs, making it difficult to generalize to weak positive sample pairs during inference. The SDM loss treats strong and weak positive sample pairs equally during training, introducing a large amount of noise interference to cross-modal alignment. Therefore, in text-based pedestrian search, how to effectively utilize weak positive samples requires more refined design.

[0003] Furthermore, the inherent modal gap between images and text remains a significant challenge for text-based person search. This issue leads to insufficient alignment of identity representations extracted by person search networks across different modalities, resulting in a lack of necessary identity discrimination capabilities. Jiang et al. initialized a person retrieval model using Contrastive Language-Image Pre-training (CLIP) to inherit its strong cross-modal alignment capabilities, thereby addressing the modal gap problem in cross-modal person re-identification. Although such methods have achieved results in enhancing cross-modal alignment at the image level, the modal gap problem in cross-modal person search remains unresolved due to the current lack of instance-level image-text alignment pre-training models. Summary of the Invention

[0004] The technical problem to be solved by this invention is how to provide a pedestrian search method based on text description and cross-modal consistent knowledge transfer that can narrow the instance representation gap in end-to-end pedestrian search networks, reduce noise interference caused by cross-modal alignment, and improve search accuracy.

[0005] To solve the above-mentioned technical problems, the technical solution adopted by the present invention is: a pedestrian search method based on cross-modal consistent knowledge transfer using text description, comprising the following steps:

[0006] The cross-modal knowledge transfer (CMKT) method is used to fine-tune the contrastive language-image pre-trained CLIP network on cropped pedestrian image-text pairs to obtain an identity retrieval teacher model with cross-modal alignment capabilities.

[0007] Then, the cross-modal knowledge transfer CMKT further transfers the cross-modal alignment capability of the identity retrieval teacher model to the end-to-end pedestrian search network through feature distillation;

[0008] Use the pedestrian search network to perform pedestrian search on pedestrian image-text pairs.

[0009] The beneficial effects of adopting the above technical solution are as follows: This application proposes a cross-modal knowledge transfer method, which successfully narrows the instance representation gap in end-to-end pedestrian search networks through feature distillation. Furthermore, it proposes an inconsistency-aware contrastive loss, which imposes different degrees of constraints based on the inconsistency of positive sample pairs, thereby mitigating the noise interference caused by inconsistency issues in cross-modal alignment. Finally, the superiority of the proposed method is demonstrated experimentally on the PRW-TPS-EN and PRW-TPS-CN datasets, and its effectiveness is further demonstrated through a series of ablation experiments, hyperparameter experiments, and visualization experiments. Attached Figure Description

[0010] The present invention will now be described in further detail with reference to the accompanying drawings and specific embodiments.

[0011] Figure 1 This is an illustration showing inconsistencies between images and text corresponding to different instances of the same person;

[0012] Figure 2 This is the main flowchart of the method described in the embodiments of the present invention;

[0013] Figure 3 This is a schematic diagram of the strong positive constraints and weak positive constraints in the IAC loss of the method described in the embodiments of the present invention;

[0014] Figure 4 This is a schematic diagram of the structure of the identity retrieval teacher model in the method described in the embodiments of the present invention;

[0015] Figure 5 This is a diagram of the pedestrian search architecture for cross-modal knowledge transfer in the method described in the embodiments of the present invention;

[0016] Figure 6These are different L in the embodiments of the present invention. q Impact curves on model performance;

[0017] Figure 7 This illustrates the impact of different β7 values ​​on model performance in the embodiments of this invention.

[0018] Figure 8 This is a visualization of pedestrian search results on PRW-TPS-EN using the text method described in this embodiment of the invention;

[0019] Figure 9 This is a visualization of pedestrian search results on PRW-TPS-CN using the method described in this embodiment of the invention. Detailed Implementation

[0020] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present invention.

[0021] Many specific details are set forth in the following description in order to provide a full understanding of the invention. However, the invention may also be practiced in other ways different from those described herein, and those skilled in the art can make similar extensions without departing from the spirit of the invention. Therefore, the invention is not limited to the specific embodiments disclosed below.

[0022] like Figure 2 As shown in the figure, this invention discloses a pedestrian search method based on cross-modal consistent knowledge transfer using text description, comprising the following steps:

[0023] S1: Using the cross-modal knowledge transfer (CMKT) method, fine-tuning the contrastive language-image pre-trained CLIP network on cropped pedestrian image-text pairs to obtain an identity retrieval teacher model with cross-modal alignment capabilities;

[0024] S2: Then, cross-modal knowledge transfer CMKT further transfers the cross-modal alignment capability of the identity retrieval teacher model to the end-to-end pedestrian search network through feature distillation;

[0025] S3: Use the pedestrian search network to perform pedestrian search on pedestrian image-text pairs.

[0026] The steps described above will be explained in detail below with reference to specific content.

[0027] A pedestrian search framework enhanced by cross-modal alignment:

[0028] To address the modal gap problem at the instance level, this application proposes a Cross-Modal Knowledge Transfer (CMKT) method for text-query-based pedestrian search. Specifically, CMKT first fine-tunes CLIP on cropped pedestrian image-text pairs to obtain an identity retrieval teacher model with strong cross-modal alignment capabilities. Then, CMKT further transfers the cross-modal alignment capabilities of the teacher model to the end-to-end pedestrian search network through feature distillation, thereby significantly improving the cross-modal retrieval performance of the text-query-based end-to-end pedestrian search method.

[0029] Furthermore, to learn consistent identity representations across image and text modalities, this application proposes an Inconsistency-Aware Contrastive (IAC) loss. Given that weak positive pairs interfere with cross-modal alignment, this application introduces the concept of a cross-modal inconsistency score to measure the dissimilarity of weak positive pairs. The cross-modal inconsistency score is quantified by the similarity between the two text descriptions associated with a weak positive pair. Therefore, during cross-modal image-text matching, IAC dynamically imposes constraints on weak positive pairs based on the cross-modal inconsistency score. This means that weak positive pairs are subject to weaker constraints compared to strong positive pairs. Ultimately, IAC allows weak positive pairs to effectively participate in cross-modal alignment training and mitigates the negative impact of inconsistency issues.

[0030] 1) Contrast loss due to inconsistency perception

[0031] Given a batch of N image-text pairs Construct a set of image-text representations using an image encoder and a text encoder. Where v i and t i Let c represent the image embedding and text embedding after L2 norm normalization, respectively. i This indicates their corresponding identity tags;

[0032] Pedestrian images and their corresponding labeled text are considered as a set of strong positive sample pairs (v i , t i This treats the images and text corresponding to different instances of the same pedestrian as a set of weak positive sample pairs (v i , t j ), where c i =c j And i≠j.

[0033] To more effectively utilize strong and weak positive sample pairs in image-text matching training, this application improves upon InfoNCE by deriving the IAC loss. IAC is constrained by a strong positive sample pair. A weak positive sample pair constraint The composition, with orange and green graphic distributions representing image-text embeddings of two pedestrian identities, shows that a pair of graphics with consistent color depth represents a strong positive sample pair, while graphics with inconsistent color depth represent a weak positive sample pair. Based on AM-Softmax loss, The strong constraint requires that the similarity of positive sample pairs be greater than that of negative sample pairs by a specific margin m1. Therefore, The goal of image-to-text matching is to maximize p i,sp The expected log-likelihood is:

[0034]

[0035] Where τ is a temperature hyperparameter used to control the softness of the distribution, and K i Indicates that in the current batch, v i The set of texts that constitute negative sample pairs.

[0036] Based on the idea of ​​hard sample mining, The primary focus is on distinguishing the weakest positive and negative sample pairs within a batch. For example... Figure 3 As shown, The similarity of the weakest positive sample pair must be a certain margin larger than that of the negative sample pair, and this margin is dynamic, determined by the degree of inconsistency of the weakest positive sample pair. Specifically, in order to effectively model the complex relationships of weak positive sample pairs, this application provides... The concept of cross-modal inconsistency score is proposed. Cross-modal inconsistency score is used to reflect weak positive sample pairs (v... i , t j The degree of inconsistency between two texts (i.e., T) and weakly positive sample pairs is determined by the correlation between them. i and T j The similarity is measured by the similarity of T and other semantic embedding models, and this semantic similarity can be calculated using publicly available pre-trained sentence embedding models (such as MiniLM), denoted as sim(T). i ,T j Based on cross-modal inconsistency scores, An inconsistency-aware interval is introduced to constrain weak positive sample pairs. Therefore, It can be formalized as maximizing p i,wp Log-likelihood:

[0037]

[0038] Where m2 is the reference interval, J iThis indicates that within the current batch, it is related to v. i The set of texts that constitute weak positive sample pairs. Indicates the content within the current batch that is related to v i The text that constitutes the weakest positive sample pair, and γ representing text T. i and semantic similarity, i.e. Essentially, the inconsistency-aware interval γm2 controls the cross-modal inconsistency score of constraints and weakly positive sample pairs to be proportional, thus mitigating the problem of noise interference. Finally, the IAC loss from image to text... The calculation method is as follows:

[0039]

[0040] In contrast, the IAC loss from text to image This can be obtained by swapping v and t in formulas (1) and (2). Finally, the bidirectional IAC loss is calculated as follows:

[0041]

[0042] 2) Cross-modal knowledge transfer

[0043] To enhance the cross-modal alignment capability at the instance level of text query-based end-to-end pedestrian search methods, this application proposes the CMKT method. CMKT consists of two parts: an identity retrieval teacher model and a knowledge distillation end-to-end pedestrian search network.

[0044] Identity Retrieval Teacher Model. The first step in CMKT is to train an identity retrieval teacher model to obtain more powerful cross-modal identity retrieval capabilities. Specifically, this application obtains this identity retrieval teacher model by fine-tuning CLIP on cropped pedestrian image-text pairs.

[0045] like Figure 4 As shown, the cropped pedestrian image The image and text description T are processed by CLIP's image encoder and text encoder, respectively, where H, W, and C represent the image's height, width, and number of channels, respectively. For the pedestrian image V, it is first split into N = H × W / P. 2 A fixed-size, non-overlapping patch is then mapped to a token sequence of length N. Where P represents the patch size. After the injected location encoding and an additional [CLS] token, the token sequence... Input image encoder. Output of the last layer of the image encoder. After the embedding is further processed by a linear mapping and L2 norm normalization, the final representation f of the pedestrian image is obtained. v .

[0046] For text T, it is first tokenized into a sequence of text tokens using Byte Pair Encoding (BPE), and this sequence is enclosed in two special tokens, [SOS] and [EOS], to indicate the start and end of the sequence. It is important to note that the text token sequence cannot exceed a preset length L. t The portion will be truncated. Then, the text token... The input is fed into the text encoder to extract features. The output of the last layer of the text encoder... After the embedding is further processed by a linear mapping and L2 norm normalization, the final representation f of the text description is obtained. t Ultimately, f v and f t It is used to perform cross-modal identity matching, a process jointly supervised by the SDM loss and the IAC loss proposed in this paper, which is formalized as follows:

[0047]

[0048] β1 is used to balance the weights of the loss function. This set of loss functions guides the alignment of image-text pair representations, thereby improving the model's ability to retrieve identities across modalities.

[0049] The fine-tuned CLIP image encoder and text encoder are referred to as the identity retrieval image encoder and identity retrieval text encoder, respectively. Notably, during this training process, all parameters of both identity retrieval encoders are updated, allowing the model to fully inherit CLIP's powerful cross-modal alignment capabilities. Ultimately, the identity retrieval teacher model can extract more identity-discriminative and fully aligned re-recognition representations across modalities.

[0050] Knowledge distillation end-to-end pedestrian search network:

[0051] The second step involves CMKT constructing a knowledge distillation end-to-end pedestrian search network, aiming to improve cross-modal alignment capabilities at the instance level. Specifically, this is achieved by transferring the identity recognition capabilities obtained from the identity retrieval image encoder to the end-to-end pedestrian search network.

[0052] like Figure 5 The identity retrieval text encoder remains frozen and is responsible for generating representations f for the text query. tThe pedestrian search pipeline is built upon the Faster R-CNN detector. Initially, the original scene image is fed into the backbone network and a region proposal network to generate bounding boxes. Then, RoI-Align pools the bounding boxes to obtain fixed-size region-of-interest (ROI) feature maps. Inspired by the "re-identification priority" principle in AlignPS, this application directly uses a canonical-aware embedding method to extract the final pedestrian identity representation f from the flattened ROI feature maps. r Next, these region-of-interest features are fed into a detection head network to obtain more refined detection bounding boxes.

[0053] Regarding the training objective of identity recognition, traditional image-text retrieval methods calculate the loss function based on samples within a batch. However, this approach is inefficient for pedestrian search networks, as they typically use small batch sizes. Therefore, this application introduces an extended version of the IAC loss specifically designed for pedestrian search, denoted as IAC. ext Specifically, this application maintains two lengths of L. q Circular queue (Q) t and Q c The text representations from past iterations and their corresponding identity tags are stored separately. During each forward propagation, this application first uses the text features f of the current batch... t Q and its corresponding identity tag c are updated respectively. t and Q c Then, according to formulas (1) and (2), this application uses pedestrian instances to characterize f. r and Q t Calculate from image to text Loss, i.e. and The goals are to maximize and Log-likelihood:

[0054]

[0055] in, as well as Furthermore, m1, m2, and τ have the same meanings as in formulas (1) and (2). Additionally, it should be noted that since the identity retrieval text encoder is frozen during training, this application only employs image-to-text supervision for the end-to-end pedestrian search network. The calculation process is as follows:

[0056]

[0057] Ultimately, due to By incorporating more weak positive and negative sample pairs into the loss calculation, the model training process becomes more efficient and achieves higher convergence quality.

[0058] Furthermore, this application employs cross-modal feature distillation to transfer identity recognition knowledge to an end-to-end pedestrian search network. Specifically, this application first crops pedestrian instances with identity labels from the scene graph based on labeled bounding boxes, and then uses a frozen identity retrieval image encoder to generate identity representations f for these pedestrian instances. v These identity representations will serve as monitoring signals in the knowledge distillation process. Next, The loss is used to minimize the identity representation f of the region of interest. r and their corresponding f v The distance between them, this process can be formalized as:

[0059]

[0060] Where N l This indicates the number of regions of interest assigned identity labels in the current batch. The above method ensures that the identity representations extracted by the end-to-end pedestrian search network and the representations extracted by the identity retrieval image encoder are fully aligned, thereby reducing the modal gap between them and the identity retrieval text encoder, and ultimately improving the performance of cross-modal identity retrieval.

[0061] Finally, the CMKT method is trained in an end-to-end manner. Consistent with Faster R-CNN, the detection subnetwork uses a classification loss. A regression loss A region-based foreground / background classification loss A regional proposal for regression loss Joint supervision. Cross-modal identity retrieval is handled by... and cross-modal characteristic distillation loss Joint supervision. Therefore, the complete training objective can be formalized as:

[0062]

[0063] After training, the identity retrieval image encoder is discarded, leaving the identity retrieval text encoder and the end-to-end pedestrian search pipeline to perform inference. During inference, the query text is encoded by the identity retrieval text encoder to obtain query features, and the pedestrian search pipeline extracts detection bounding boxes and their corresponding re-identification representations on the scene graph. Then, the model outputs search results in order based on the similarity between the re-identification representations and the query features.

[0064] Experimental verification and analysis

[0065] This section first introduces the implementation and training details of the method described in this application. Then, comparative experiments with state-of-the-art methods are conducted on the English dataset PRW-TPS-EN and the Chinese dataset PRW-TPS-CN to demonstrate that the proposed method achieves a state-of-the-art level in text-based pedestrian search. Next, this section further conducts ablation experiments on key modules and hyperparameter experiments to verify the effectiveness of the method described in this application. In each set of experiments, mAP score and Rank-1 / 5 / 10 accuracy are used to evaluate the model's search performance. Finally, this section also provides a visualization experiment to visually demonstrate the search effect of the method described in this application.

[0066] 1) Model implementation and training details

[0067] For PRW-TPS-EN and PRW-TPS-CN, this application initializes the identity retrieval teacher model using pre-trained CLIP-ViT-B / 16 and Chinese-CLIP-ViT-B / 16 models, respectively. All images input to the identity retrieval teacher model are uniformly resized to 384×128, i.e., H=384, W=128, and the patch size P is set to 16. Based on the pre-trained model settings, the truncation length L of the input text token is set for both PRW-TPS-EN and PRW-TPS-CN. t The values ​​were set to 77 and 52 respectively. During training, this application employed three data augmentation methods for images: random horizontal flipping, random cropping, and random erasure. For text, 15% of the tokens were randomly masked for data augmentation. Regarding the hyperparameters in formulas (1) and (2), this paper sets m1 and m2 to 0.3 and 0.1 respectively, and sets the temperature coefficient τ to 0.02. This application uses MiniLM-L12-V2 as the statement embedding model for calculating cross-modal inconsistency scores in the IAC loss. The loss function weight β1 in formula (5) was set to 0.5. Finally, the identity retrieval teacher model was trained for 60 epochs using the Adaptive Moment Estimation (Adam) optimization algorithm. The training batch size was 128, the learning rate was initialized to 1e-5, and cosine decay was used.

[0068] This application uses a ResNet50 pre-trained on ImageNet to initialize an end-to-end pedestrian search network, where res1–res4 serve as the backbone network and res5 serves as the detection head network. For the pedestrian search network, the input scene images are uniformly resized to 1500×900 pixels, and the input text processing method is consistent with the identity retrieval teacher model. The size of the region of interest feature maps generated by RoI-Align pooling is set to 14×14. The re-identification representation output by the embedding head network has a dimension of 256. t and Qc queue length L q The value is set to 200. The hyperparameter settings in formulas (6) and (7) are consistent with those in formulas (1) and (2), respectively. In formula (10), the weights of the loss function are set to β2 = β4 = β5 = β6 = 1, β3 = 10, and β7 = 500. Regarding training details, the end-to-end pedestrian search network was trained for 24 epochs using a stochastic gradient descent optimization algorithm with a momentum of 0.9. The training batch size was 5, and the learning rate was heated to 0.002 in the first epoch and then decreased to 0.0002 in the sixteenth epoch. This experiment used a GeForce RTX 3090 GPU and an Intel Xeon Gold 6330 CPU for computation, and the code was written and implemented using the PyTorch framework and run on the Ubuntu 20.04 operating system.

[0069] Table 1 - Comparison of state-of-the-art methods on the PRW-TPS-EN dataset

[0070]

[0071] 2) Comparison Experiment of Advanced Methods

[0072] This section evaluates the proposed method and two existing text query-based person search methods: a two-stage method and a one-stage method. The two-stage method breaks down text query-based person search into two sub-tasks: first, it uses a detector to locate all pedestrian instances in the scene image; then, it performs text query-based person re-identification on the cropped pedestrian bounding boxes. Conversely, the one-stage method trains both the person detection and text query-based person re-identification sub-tasks simultaneously in an end-to-end network. These methods typically include a person search sub-network that processes the scene image and a text encoder that processes the text description.

[0073] For the two-stage approach, this application compares an existing method, PGPS, with several methods combining advanced detectors (e.g., RetinaNet, Faster R-CNN, and Deformable DETR) and the state-of-the-art cross-modal person re-identification model, IRRA. For the one-stage approach, this paper compares an existing method, SDPG, with several models extended from image-based query-based one-stage person search methods, including SeqNet and COAT, which combine BERT and DeBERTa text encoders.

[0074] Table 2 - Comparison of state-of-the-art methods on the PRW-TPS-CN dataset

[0075]

[0076] Table 1 presents the performance comparison of the above methods on the PRW-TPS-EN dataset. The experimental results show that the proposed method significantly outperforms all two-stage and one-stage methods, achieving state-of-the-art performance in text-based pedestrian search. Specifically, compared to the two-stage method PGPS proposed by Wu et al., the proposed method shows improvements of 5.33% and 10.16% in mAP score and Rank-1 accuracy, respectively. Compared to the one-stage method SDPG proposed by Zhang et al., the proposed method exhibits a more significant advantage, with improvements of 20.20% and 31.43% in mAP score and Rank-1 accuracy, respectively. More importantly, compared to the state-of-the-art one-stage method COAT+DeBERTa-large, the proposed method still maintains a significant performance advantage, with improvements of 4.16% and 13.19% in mAP score and Rank-1 accuracy, respectively. These experimental results demonstrate that the proposed method effectively solves the complex challenge of simultaneously optimizing three subtasks—pedestrian detection, pedestrian re-identification, and cross-modal retrieval—in an end-to-end model, thereby surpassing the performance of two-stage methods and achieving the best performance in text query-based pedestrian search to date.

[0077] Table 2 also presents a performance comparison of state-of-the-art methods on the PRW-TPS-CN dataset. It can be seen that the proposed method still significantly outperforms all two-stage and one-stage methods, demonstrating its superiority. Compared to the state-of-the-art two-stage method Deformable DETR+IRRA, the proposed method has advantages of 1.82% and 2.16% in mAP score and Rank-1 accuracy, respectively; compared to the state-of-the-art one-stage method COAT+DeBERTa-large, the proposed method has advantages of 2.73% and 10.58% in mAP score and Rank-1 accuracy, respectively. These results demonstrate that the proposed method is insensitive to query language and is highly effective in both English and Chinese text query scenarios.

[0078] Furthermore, Tables 1 and 2 also show that all the two-stage methods included in the comparison outperform the one-stage baseline methods. This indicates that the one-stage models do indeed face significant challenges from the instance-level modal gap and multi-task joint optimization, which severely limits their pedestrian search performance. However, the one-stage method proposed in this application outperforms all two-stage models, further demonstrating the effectiveness of this method in enhancing instance-level cross-modal alignment through cross-modal knowledge distillation.

[0079] 3) Modular ablation experiment

[0080] To verify the effectiveness of the inconsistency-aware contrastive loss and cross-modal knowledge transfer proposed in this application, as well as the role of the re-identification priority principle adopted in this application in improving the model search performance, this section conducted a series of ablation experiments on the PRW-TPS-EN dataset.

[0081] Table 3 - Comparison of different supervision losses for the identity retrieval teacher model

[0082]

[0083] Effectiveness of Inconsistency-Aware Contrast Loss. To evaluate the effectiveness of the IAC loss, this section compares the performance of the proposed method on the identity retrieval teacher model using different supervised losses. The experimental results are shown in Table 3. Among them, IAC... fixed express Using a fixed-interval IAC variant means that IAC imposes a fixed but weaker constraint on weak positive sample pairs than on strong positive sample pairs. As the table results show, compared to InfoNCE and SDM, IAC improves the mAP score by 3.14% and 0.98%, respectively, indicating that IAC's differentiated treatment of strong and weak positive sample pairs can mitigate noise interference in the cross-modal alignment process. Furthermore, compared to IAC… fixed IAC resulted in a 0.62% improvement in mAP score and a 0.60% improvement in Rank-1 accuracy. This demonstrates that IAC, by dynamically imposing constraints based on the inconsistency of positive sample pairs, can more accurately model the complex relationships between weak positive sample pairs, thus making more effective use of weak positive sample pairs in cross-modal alignment. Finally, it was found that using both IAC loss and SDM loss simultaneously can achieve better training results for the identity retrieval teacher model.

[0084] Table 4 - Comparison of different supervision losses in end-to-end pedestrian search networks

[0085]

[0086] Table 4 further analyzes the effectiveness of the IACext loss by comparing the supervised losses of different end-to-end pedestrian search networks. Among them, SDM... ext Used with IAC ext The same extension method makes the SDM loss more suitable for pedestrian search networks. Furthermore, since unlabeled pedestrian instances in the PRW-TPS-EN dataset lack corresponding text descriptions, the OIM loss here only uses a lookup table to store the features of labeled pedestrians and calculates the loss based on them, ignoring unlabeled pedestrians. Finally, the experimental results in Table 4 show that IAC... extCompared to OIM loss, the loss method showed improvements of 3.05% and 3.66% in mAP score and Rank-1 accuracy, respectively, compared to SDM. ext The loss improved mAP score and Rank-1 accuracy by 3.79% and 5.84%, respectively, further demonstrating the effectiveness of IAC in constraining inconsistencies in positive sample pairs for cross-modal alignment.

[0087] Effectiveness of cross-modal knowledge transfer: Table 4 also compares the performance of using the cross-modal feature distillation loss L1. Experimental results show that using cross-modal knowledge transfer improves the mAP score by 3.34% and the Rank-1 accuracy by 2.83%. These results demonstrate that the CMKT method proposed in this application successfully narrows the instance-level modal gap problem in end-to-end pedestrian search networks, thereby significantly improving pedestrian search performance. On the other hand, text-based end-to-end pedestrian search networks need to simultaneously optimize three objectives: pedestrian detection, pedestrian re-identification, and cross-modal retrieval, making it difficult for the model to fully learn the ability of cross-modal identity retrieval. This application enhances the learning of cross-modal identity retrieval in end-to-end networks through cross-modal knowledge transfer, ultimately achieving superior pedestrian search performance.

[0088] Table 5 - Comparison of the effectiveness of the re-identification priority principle

[0089]

[0090] Effectiveness of the Re-identification Priority Principle: To further explore the effectiveness of the re-identification priority principle adopted in this application, this application compared the effects of re-identification representation learning based on feature maps of different layers. The experimental results are shown in Table 5. Here, "res4" indicates re-identification representation learning based directly on the region of interest features obtained from the feature pooling of the last backbone network layer; "res5" indicates re-identification representation learning based on the detection box features output by the detection head network; and "res4+res5" indicates re-identification representation learning using the concatenation of the first two types of features. It should be noted that this set of experiments controlled the dimensionality of the final pedestrian re-identification representation to be the same. This shows that prioritizing the position of pedestrian re-identification representation learning in the pedestrian search network can effectively improve the training effect of the model's cross-modal identity retrieval. Given that re-identification representation learning based on the features of the last backbone network layer achieved the highest Rank-1, Rank-5, and Rank-10 accuracy, this application's method adopts this re-identification priority pedestrian search structure.

[0091] Hyperparameter Experiment

[0092] To explore IAC ext Loss of the circular queue length L qTo investigate the impact of the weight β7 of L1 loss on the performance of the proposed method, this section presents a series of hyperparameter experiments conducted on the PRW-TPS-EN dataset.

[0093] Q t (Q c The impact of queue length on model performance. To investigate L... q For IAC ext The impact of the loss is compared with L in this application. q The pedestrian search performance of the model when the number of pedestrians is increased from 10 to 300 is shown in the experimental results. Figure 6 It should be noted that, in order to more clearly display L... q Regarding the impact on model performance, this group of experiments did not use cross-modal knowledge transfer methods when evaluating model performance; that is, only detection loss and IAC were used. ext Loss training model. As can be seen from the figure, when L... q As L gradually increases from 10 to 100, the model performance shows a stable upward trend; q As Q increases further from 100, the model's performance almost stops improving. This reflects the fact that as Q increases... t (Q c The increase in queue length, IAC ext This method considers more inconsistent positive and negative sample pairs during cross-modal alignment, resulting in better training performance. However, this advantage saturates as the queue length increases to a certain size. Therefore, the above results demonstrate the reliability of the IAC loss expansion method and the effectiveness of IAC loss in utilizing inconsistent positive sample pairs for cross-modal alignment. Finally, the method in this application sets L... q =200.

[0094] The impact of cross-modal feature distillation loss weights on model performance. To further explore the impact of the weight β7 of the cross-modal feature distillation loss L1 on model performance, this application compares the pedestrian search performance of the model when β7 increases from 1 to 700, such as... Figure 7 As shown in the figure, the model performance gradually increases with the increase of β7, reaching its highest level at β7 = 500; further increasing β7 leads to a certain downward trend in model performance. This indicates that the cross-modal feature distillation loss can effectively guide the pedestrian search network to improve its cross-modal identity retrieval capabilities, and the model achieves optimal training results when this loss, detection loss, and IACext loss reach a balance. Finally, the method in this application sets β7 to 500.

[0095] Visualization Experiment:

[0096] Figure 8 and Figure 9The visualization results of pedestrian search based on text queries using the proposed method are presented on the PRW-TPS-EN and PRW-TPS-CN datasets. Specifically, for each query text, the top three search results are displayed from left to right, with each search result highlighted in the lower left corner of the scene graph. Green rectangles represent correct search results, while red rectangles represent incorrect answers. As can be seen from the figures, the proposed method can accurately locate the target pedestrian described in the text, even when the pedestrian scale is small and the pedestrian pose and background change drastically. Even in incorrect cases, some features of the search results are consistent with the text description. For example, Figure 8 The third search result corresponding to the query text 1 matches the characteristics described in the text, such as "a short-haired man wearing a white short-sleeved shirt, black trousers and black leather shoes, and carrying a black briefcase"; Figure 9 The third search result corresponding to the query text 2 matches the characteristics described in the text, such as "a short-haired man wearing gray shorts, dark gray shoes, and carrying a black backpack"; Figure 9 The first search result corresponding to query text 3 matches the features described in the text, such as "a woman with black hair, wearing a black short-sleeved shirt and blue denim shorts." These results intuitively demonstrate that the method in this application achieves good cross-modal alignment in pedestrian search tasks based on text queries, thus enabling the discovery of pedestrian instances that match the text description.

[0097] This application proposes a cross-modal knowledge transfer method that successfully narrows the instance representation gap in end-to-end pedestrian search networks through feature distillation. Furthermore, it proposes an inconsistency-aware contrastive loss function, which imposes varying degrees of constraint based on the inconsistency of positive sample pairs, thereby mitigating the noise interference caused by inconsistency issues in cross-modal alignment. Finally, experiments on the PRW-TPS-EN and PRW-TPS-CN datasets demonstrate the superiority of the proposed method, and a series of ablation experiments, hyperparameter experiments, and visualization experiments further demonstrate its effectiveness.

Claims

1. A pedestrian search method based on cross-modal consistent knowledge transfer using text description, characterized in that... Includes the following steps: The cross-modal knowledge transfer (CMKT) method is used to fine-tune the contrastive language-image pre-trained CLIP network on cropped pedestrian image-text pairs to obtain an identity retrieval teacher model with cross-modal alignment capabilities. Then, the cross-modal knowledge transfer CMKT further transfers the cross-modal alignment capability of the identity retrieval teacher model to the end-to-end pedestrian search network through feature distillation; Use a pedestrian search network to perform pedestrian image-text pairs for pedestrian search; In training the identity retrieval teacher model, we use the inconsistency-aware contrastive IAC loss method to learn consistent identity representations across image and text modalities. The contrastive IAC loss for inconsistency perception is: Given a batch Image-text pair A set of image-text representations is constructed using an image encoder and a text encoder. ,in and These represent the image embedding and text embedding after L2 norm normalization, respectively. This indicates their corresponding identity tags; The pedestrian image and its corresponding labeled text are considered as a set of strong positive sample pairs. , The images and text corresponding to different instances of the same pedestrian are considered as a set of weak positive sample pairs. , ),in and ; Inconsistency-aware contrastive IAC includes a strong positive sample pair constraint. A weak positive sample pair constraint Weak positive samples against constraints The strong constraint requires that the similarity of positive sample pairs be greater than that of negative sample pairs by a specific margin. Therefore, strong positive samples are constrained. The goal of image-to-text matching is to maximize... The expected log-likelihood is: （1） in, It is a temperature hyperparameter used to control the smoothness of the distribution. Indicates the current batch with The set of texts that constitute negative sample pairs; To effectively model complex relationships between weak positive sample pairs, constraints on weak positive sample pairs are required. The concept of cross-modal inconsistency score is introduced, which is used to reflect weak positive sample pairs ( , The degree of inconsistency between two texts associated with weak positive samples is determined by the correlation between them. and The similarity is measured by the semantic similarity, which can be calculated using a publicly available pre-trained sentence embedding model, denoted as . Based on cross-modal inconsistency scores, weak positive samples are used to constrain... Inconsistency-aware intervals are introduced to constrain weak positive sample pairs; therefore, weak positive sample pairs are constrained. Formalized as maximization Log-likelihood: （2） in, It is the reference interval. Indicates that within the current batch and The set of texts that constitute weak positive sample pairs. Indicates the current batch and The texts that constitute the weakest positive sample pairs, and Representing text and semantic similarity, i.e. = ; interval of inconsistency perception The control constraints and the cross-modal inconsistency score of weak positive sample pairs are directly proportional, ultimately leading to the IAC loss from image to text. The calculation formula is as follows: （3） Symmetrical IAC loss from text to image By exchanging the terms in formula (1) and formula (2) and Therefore, the final calculation method for bidirectional IAC loss is as follows: （4）。 2. The pedestrian search method based on cross-modal consistent knowledge transfer using text description as described in claim 1, characterized in that, The identity retrieval teacher model is obtained by fine-tuning the contrast language on cropped pedestrian image-text pairs and pre-training the CLIP model on images. Cropped pedestrian images The text description 𝑇 is processed by the image encoder and text encoder of the contrastive language-image pre-trained CLIP model, respectively, where 𝐻, W, and 𝐶 represent the height, width, and number of channels of the image, respectively; for pedestrian images First, it was broken down into A fixed-size, non-overlapping patch is then mapped to a length of... token sequence Where 𝑃 represents the patch size; after the injected location is encoded and an additional [CLS] token, the token sequence { , ,..., Input image encoder; The output of the last layer of the image encoder After embedding and then undergoing a linear mapping and L2 norm normalization, the final representation of the pedestrian image is obtained. ; For text 𝑇, it is first tokenized into a text token sequence using the byte-pair encoding BPE method. This text token sequence is enclosed in two special tokens, [SOS] and [EOS], to indicate the start and end of the sequence. The text token sequence must exceed a preset length. The portion will be truncated; Then, the text token { , ,..., The input is fed into the text encoder to extract features; the output of the last layer of the text encoder is... After the embedding is further processed by a linear mapping and L2 norm normalization, the final representation of the text description is obtained. ;final, and It is used to perform cross-modal identity matching, a process jointly supervised by SDM loss and IAC loss, and formalized as follows: （5） in, Weights used to balance losses; The fine-tuned CLIP image encoder and text encoder are called the identity retrieval image encoder and identity retrieval text encoder, respectively. Finally, the identity retrieval teacher model extracts a more identity-discriminative and fully aligned re-recognition representation.

3. The pedestrian search method based on cross-modal consistent knowledge transfer using text description as described in claim 1, characterized in that: By employing the cross-modal knowledge transfer (CMKT) method, the identity recognition capabilities acquired by the identity retrieval image encoder are transferred to an end-to-end pedestrian search network.

4. The pedestrian search method based on cross-modal consistent knowledge transfer using text description as described in claim 3, characterized in that: The identity retrieval text encoder remains frozen and is responsible for generating representations for text queries. The pedestrian search pipeline is built on top of the Faster R-CNN detector. First, the original scene image is fed into the backbone network and a region proposal network to generate proposal boxes. Then, RoI-Align is used to pool the proposal boxes to obtain fixed-size region of interest feature maps; the final pedestrian identity representation is directly extracted from the flattened region of interest feature maps using a canonical-aware embedding method. Next, these region-of-interest features are fed into a detection head network to obtain more refined detection bounding boxes.

5. The pedestrian search method based on cross-modal consistent knowledge transfer using text description as described in claim 3, characterized in that: The training objective for identity recognition in pedestrian search networks is to introduce an extended version of the IAC loss designed for pedestrian search, denoted as IAC. ext Maintain two lengths of circular queue and Store the text representations and their corresponding identity tags from past iterations respectively; during each forward propagation, first use the text features of the current batch. and their corresponding identity tags Update separately and ; Then, based on formulas (1) and (2), pedestrian instances are used for characterization. and Calculate from image to text Loss, i.e. and The goals are to maximize and Log-likelihood: （6） （7） in, = as well as = Because the identity retrieval text encoder is frozen during training, the end-to-end pedestrian search network only uses image-to-text supervision. The calculation process is as follows: (8) Ultimately, due to By incorporating more weak positive and negative sample pairs into the loss calculation, the model training process becomes more efficient and achieves higher convergence quality.

6. The pedestrian search method based on cross-modal consistent knowledge transfer using text description as described in claim 3, characterized in that: Cross-modal feature distillation is used to transfer identity recognition knowledge to an end-to-end pedestrian search network.

7. The pedestrian search method based on cross-modal consistent knowledge transfer using text description as described in claim 6, characterized in that: First, pedestrian instances with identity labels are cropped from the scene graph based on labeled bounding boxes. Then, a frozen identity retrieval image encoder is used to generate identity representations for these pedestrian instances. These identity representations will serve as monitoring signals in the knowledge distillation process; then, The loss was used to minimize the identity representation of the region of interest. and its corresponding The distance between them, the process is formalized as: （9） in, This indicates the number of regions of interest that have been assigned identity tags in the current batch; Finally, the cross-modal knowledge transfer (CMKT) method is trained in an end-to-end manner; in this method, the detection subnetwork is trained using a classification loss. A regression loss A region-specific foreground / background classification loss. A regional proposal for regression loss Joint supervision; cross-modal identity retrieval by and cross-modal characteristic distillation loss Joint supervision; therefore, the complete training objective is formalized as: （10） After training is complete, the identity retrieval image encoder is discarded, leaving the identity retrieval text encoder and the end-to-end pedestrian search pipeline to perform inference. During inference, the query text is encoded by the identity retrieval text encoder to obtain query features, and the pedestrian search pipeline extracts detection bounding boxes and their corresponding re-identification representations on the scene graph. Then, the model outputs the search results in order based on the similarity between the re-identification representations and the query features.

8. The pedestrian search method based on cross-modal consistent knowledge transfer using text description as described in claim 1, characterized in that: The proposed method was compared with state-of-the-art methods on the English dataset PRW-TPS-EN and the Chinese dataset PRW-TPS-CN.

Images