Pedestrian re-identification method, electronic device, and computer program product

Abstract

The present disclosure provides a pedestrian re-identification method, an electronic device and a computer program product. The pedestrian re-identification method of the present disclosure comprises: realizing pedestrian re-identification based on a student model and a teacher model, the student model and the teacher model being multi-modal large language models, the feature space of the student model being aligned with the feature space of the teacher model through knowledge distillation, the pedestrian re-identification method comprising: inputting the query data of a target pedestrian input by a user into the student model for feature extraction to determine a feature vector of the query data; the query data being an RGB image, an infrared image, a hand-drawn sketch or a text description; performing similarity matching between the feature vector of the query data and the feature vector of a pedestrian image in a database to obtain a candidate image; and filtering the candidate image based on the similarity of the candidate image to determine a target image with a similarity greater than or equal to a similarity threshold.

Description

Technical Field

[0001] This disclosure relates to a pedestrian re-identification method, electronic device, and computer program product. Background Technology

[0002] Pedestrian re-identification technology aims to retrieve specific pedestrians across different cameras. Existing methods generally suffer from fragmentation issues, requiring the design of dedicated models for different modalities, resulting in complex systems that are difficult to scale. Meanwhile, while multimodal large language models possess the potential for unified multimodal understanding, current applications are mostly limited to indirect methods (such as visual question answering or text description generation), constrained by context length and prone to illusions. Furthermore, the massive scale of multimodal large language models makes them difficult to deploy on edge devices, and traditional knowledge distillation methods are poorly adapted to the complex structures of multimodal large language models. Summary of the Invention

[0003] This disclosure provides a pedestrian re-identification method, electronic device, and computer program product.

[0004] According to one aspect of this disclosure, a pedestrian re-identification method is provided, which realizes pedestrian re-identification based on a student model and a teacher model. The student model and the teacher model are multimodal large language models. The feature space of the student model is aligned with the feature space of the teacher model through knowledge distillation. The pedestrian re-identification method includes: inputting query data of a target pedestrian input by a user into the student model for feature extraction to determine the feature vector of the query data; the query data is an RGB image, an infrared image, a hand-drawn sketch, or a text description; performing similarity matching between the feature vector of the query data and the feature vector of pedestrian images in a database to obtain candidate images; and filtering the candidate images based on the similarity of the candidate images to determine the target image whose similarity is greater than or equal to a similarity threshold.

[0005] According to the pedestrian re-identification method disclosed herein, a student model and a teacher model work collaboratively. Utilizing knowledge distillation technology, the lightweight student model can output highly discriminative feature vectors aligned with the feature space of the powerful teacher model during inference. This enables unified processing of query data from multiple modalities, such as RGB images, infrared images, hand-drawn sketches, or text descriptions. This overcomes the technical obstacle of inconsistent data distribution across modalities in cross-modal pedestrian re-identification, allowing a single framework to handle four modalities of retrieval tasks end-to-end without requiring dedicated models for different modalities, significantly improving the system's uniformity and scalability. Furthermore, through feature space alignment and similarity calculation, accurate matching from any modal query to candidate images is achieved, effectively solving the problems of severe fragmentation and cross-modal alignment difficulties in traditional methods, thus improving retrieval accuracy.

[0006] The pedestrian re-identification method according to at least one embodiment of the present disclosure further includes: inputting the target image with a similarity greater than or equal to a similarity threshold into the teacher model for secondary matching to determine the final target image.

[0007] According to at least one embodiment of the pedestrian re-identification method of this disclosure, the training process of the teacher model includes: constructing a paired dataset containing four modalities of data: RGB images, infrared images, hand-drawn sketches, and text descriptions, with each pair of paired data containing different modalities of the same pedestrian; extracting features from the modal data based on instructional prompts to obtain feature vectors of the modal data; and constructing a loss function for the teacher model based on the feature vectors of the modal data to optimize the training of the teacher model; the loss function includes identity recognition loss, triplet loss, and / or similarity distribution matching loss.

[0008] According to at least one embodiment of the pedestrian re-identification method of this disclosure, a trainable low-rank adapter is added to the linear transformation matrix of the target level in the teacher model for feature extraction of modal data.

[0009] According to at least one embodiment of the pedestrian re-identification method of this disclosure, feature extraction is performed on the modal data based on instruction prompts to obtain the feature vector of the modal data, including: performing forward propagation on the modal data based on instruction prompts to obtain the final hidden state of the modal data; and performing pooling operation on the final hidden state of the modal data to obtain the feature vector of the modal data.

[0010] According to at least one embodiment of the pedestrian re-identification method of this disclosure, the training process of the student model includes: constructing a paired dataset containing four modal data: RGB images, infrared images, hand-drawn sketches, and text descriptions, with each paired dataset containing different modal data of the same pedestrian; extracting features from the modal data to obtain feature vectors of the modal data; and using the feature vectors of the modal data generated by the teacher model as supervision signals to construct principal component mapping loss, feature relation loss, and cosine similarity matching loss to optimize the training of the student model.

[0011] A pedestrian re-identification method according to at least one embodiment of the present disclosure includes: performing singular value decomposition on a feature matrix composed of feature vectors generated by the teacher model to obtain principal components of a target dimension; projecting feature vectors generated by the student model and feature vectors generated by the teacher model onto a vector space composed of principal components of the target dimension; calculating the similarity loss between the projected feature vectors generated by the student model and the projected feature vectors generated by the teacher model, and determining the principal component mapping loss.

[0012] According to at least one embodiment of the pedestrian re-identification method of this disclosure, the user-inputted query data of the target pedestrian is input into the student model for feature extraction to determine the feature vector of the query data, including: preprocessing the query data, the preprocessing including size normalization, color space conversion and / or text segmentation; and performing feature extraction based on the preprocessed query data to determine the feature vector of the preprocessed query data.

[0013] According to another aspect of this disclosure, an electronic device is provided, comprising: a memory storing execution instructions; and a processor executing the execution instructions stored in the memory, causing the processor to perform a pedestrian re-identification method according to any embodiment of this disclosure.

[0014] According to another aspect of this disclosure, a readable storage medium is provided, wherein execution instructions are stored therein, which, when executed by a processor, are used to implement the pedestrian re-identification method of any embodiment of this disclosure.

[0015] According to another aspect of this disclosure, a computer program product is provided, including a computer program that, when executed by a processor, implements a pedestrian re-identification method according to any embodiment of this disclosure. Attached Figure Description

[0016] The accompanying drawings illustrate exemplary embodiments of the present disclosure and, together with the description thereof, serve to explain the principles of the present disclosure. These drawings are included to provide a further understanding of the present disclosure and are incorporated in and constitute a part of this specification.

[0017] Figure 1 This is a schematic diagram illustrating an application scenario of a pedestrian re-identification method according to one embodiment of the present disclosure.

[0018] Figure 2 This is a schematic diagram of the overall process of a pedestrian re-identification method according to one embodiment of the present disclosure.

[0019] Figure 3 This is a flowchart illustrating the determination of feature vectors in preprocessed query data in a pedestrian re-identification method according to one embodiment of the present disclosure.

[0020] Figure 4 This is a flowchart of the loss function for constructing a teacher model in a pedestrian re-identification method according to one embodiment of the present disclosure.

[0021] Figure 5 This is a flowchart illustrating the process of obtaining feature vectors of modal data in a pedestrian re-identification method according to one embodiment of the present disclosure.

[0022] Figure 6This is a flowchart of the loss function for constructing a student model in a pedestrian re-identification method according to one embodiment of the present disclosure.

[0023] Figure 7 This is a flowchart of determining the principal component mapping loss in a pedestrian re-identification method according to one embodiment of the present disclosure.

[0024] Figure 8 This is a pedestrian re-identification paradigm diagram of a multimodal large model in a pedestrian re-identification method according to one embodiment of the present disclosure.

[0025] Figure 9 This is a framework diagram of a pedestrian re-identification method according to one embodiment of the present disclosure.

[0026] Figure 10 This is a singular value decomposition analysis diagram of pedestrian re-identification features in a cloud-based teacher model in a pedestrian re-identification method according to one embodiment of this disclosure.

[0027] Figure 11 This is a t-SNE visualization result of the cloud-based teacher model and the edge-based student model in a pedestrian re-identification method according to one embodiment of the present disclosure.

[0028] Figure 12 This is a diagram showing the recognition results of the cloud-based teacher model and the edge-based student model in a pedestrian re-identification method according to one embodiment of this disclosure.

[0029] Figure 13 This is a schematic structural block diagram of a pedestrian re-identification device according to one embodiment of the present disclosure.

[0030] Figure 14 This is a schematic structural block diagram of an electronic device according to one embodiment of the present disclosure. Detailed Implementation

[0031] The present disclosure will now be described in further detail with reference to the accompanying drawings and examples. It should be understood that the specific examples described herein are for illustrative purposes only and are not intended to limit the scope of the disclosure. Furthermore, it should be noted that, for ease of description, only the parts relevant to the present disclosure are shown in the accompanying drawings.

[0032] It should be noted that, where there is no conflict, the embodiments and features described in this disclosure can be combined with each other. The technical solutions of this disclosure will now be described in detail with reference to the accompanying drawings and embodiments.

[0033] In smart security scenarios, pedestrian images captured by infrared cameras at night need to be matched with candidate images in a daytime candidate image database. However, the imaging mechanisms of infrared and visible light images differ significantly, and there are significant semantic gaps and distribution differences between different modalities, making it difficult to achieve a unified representation using traditional single feature spaces. In text retrieval scenarios, the ambiguity of natural language descriptions and the diversity of visual information in images are difficult to align, easily leading to semantic bias.

[0034] To address this issue, this disclosure proposes a pedestrian re-identification method. By employing a multimodal large language model as both the student and teacher models, and utilizing knowledge distillation techniques to align the feature space of the student model with that of the teacher model, it achieves unified and efficient processing of various modal data, including RGB images, infrared images, hand-drawn sketches, and text descriptions. In smart security scenarios, it effectively overcomes significant imaging differences between nighttime infrared images and daytime visible light images, or semantic discrepancies between text descriptions and actual image visual information. By accurately extracting the feature vectors of the query data and performing similarity matching with the feature vectors of pedestrian images in the database, highly similar target images are ultimately selected. This achieves accurate and reliable pedestrian re-identification under cross-modal conditions, significantly improving retrieval efficiency and accuracy, and solving the problem that traditional single feature spaces struggle to handle different modal data.

[0035] Figure 1 This is a schematic diagram illustrating an application scenario of a pedestrian re-identification method according to one embodiment of this disclosure. For example... Figure 1 As shown, the system may include a client 100 and a server 200. The client can be a terminal device such as a computer, mobile phone, or tablet, or it can be different programs or apps on the same terminal device. There may be one or more clients. The server 200 may be various computing nodes within the server. The client can send data to and request data from the server, and the server can store the received data, enabling data access between the client and the server.

[0036] Figure 2 A schematic diagram illustrating the overall flow of a pedestrian re-identification method according to one embodiment of this disclosure is shown. Figure 2 The method M100 shown includes steps S110 to S130. This method can be executed by electronic devices such as mobile phones and tablets, or by a server. Furthermore, it achieves pedestrian re-identification based on a student model and a teacher model, both of which are multimodal large language models. The feature space of the student model is aligned with the feature space of the teacher model through knowledge distillation.

[0037] In step S110, the query data of the target pedestrian input by the user is input into the student model for feature extraction to determine the feature vector of the query data; the query data can be an RGB image, an infrared image, a hand-drawn sketch or a text description.

[0038] Preferably, the student model uses an appropriate encoding branch for query data of different modalities, with visual modalities processed by a visual encoder and text modalities processed by a text encoder.

[0039] Preferably, the teacher model is the Qwen2-VL-2B multimodal large language model.

[0040] Preferably, the student model is a multimodal model with CLIP (ViT-L / 14) architecture.

[0041] Optionally, end-to-end data processing can be performed on the query data using either a teacher model or a student model.

[0042] Preferably, during the feature extraction process of the student model on the input query data, the feature vectors of the query data may have inconsistent image feature distributions in the original feature space with those of the feature vectors extracted by the teacher model due to modal differences (such as infrared having no color or text having no pixels). Therefore, it is necessary to transfer or map the feature vectors of the query data from the original feature space to the semantic feature space dominated by the teacher model, so that they are comparable to the feature vectors extracted by the teacher model under the same dimension and semantic scale.

[0043] Preferably, the feature vectors of the query data are mapped to the feature space of the teacher model through a linear projection layer or an adapter module.

[0044] In step S120, the feature vector of the query data is matched with the feature vector of the pedestrian images in the database to obtain candidate images.

[0045] The aforementioned database is a pre-built and stored collection of images containing a large amount of known pedestrian identification information, typically sourced from real-world data such as surveillance cameras, deployed cameras, or historical case archives. The pedestrian images cover pedestrian information under different times, locations, lighting conditions, and viewing angles, and each image is associated with a unique identification tag (such as an ID number) for subsequent matching and comparison.

[0046] The pedestrian images mentioned above can be RGB visible light images, infrared images, or multi-angle snapshots, etc., aiming to cover the diversity and complexity in practical applications.

[0047] Preferably, the similarity between the feature vectors of the query data and the feature vectors of pedestrian images in the database is calculated. Cosine similarity is used as the core metric, measuring directional consistency by calculating the cosine of the angle between the two feature vectors. Euclidean distance or Mahalanobis distance can be combined for multi-dimensional complementarity. After calculation, each pedestrian image in the database receives a numerical similarity score, representing the probability that it is the same pedestrian as the target pedestrian.

[0048] In step S130, candidate images are filtered based on their similarity to determine target images with a similarity greater than or equal to a similarity threshold.

[0049] Preferably, candidate images are sorted in descending order of similarity score, and the candidate images with the highest scores in the Top-K (e.g., top 10 or top 100) are selected as the initial screening results according to the application scenario requirements. Further quality filtering is performed using a preset similarity threshold to eliminate low-confidence matches below the threshold, avoiding false positives. The final output is the sorted and filtered target image, which can be accompanied by a similarity score as a confidence indicator for use by downstream business systems or manual review.

[0050] In one specific embodiment, a target image with a similarity greater than or equal to a similarity threshold is input into a teacher model for secondary matching to determine the final target image. Through the teacher model's powerful semantic understanding and more refined feature discrimination capabilities, the target image is re-verified, resulting in a more accurate and reliable output, maximizing recognition accuracy while ensuring efficiency.

[0051] Therefore, the pedestrian re-identification method disclosed in this paper adopts a multimodal large language model as both the student and teacher models, and uses knowledge distillation technology to align the feature space of the student model with that of the teacher model, achieving efficient and unified processing of various modal data such as RGB images, infrared images, hand-drawn sketches, or text descriptions. By adapting the encoding branches of different modal query data and using linear projection layers or adapter modules to adjust the spatial distribution of feature vectors, cross-modal data can be compared at the same semantic scale. Furthermore, it calculates the similarity between the query data and pedestrian images in the database, and selects high-confidence target images based on the similarity scores. This not only overcomes the problem of traditional single feature spaces being unable to handle cross-modal data, but also significantly improves the accuracy and reliability of pedestrian re-identification.

[0052] In step S110, the user-inputted query data of the target pedestrians is input into the student model for feature extraction to determine the feature vector of the query data. In some embodiments of this disclosure, this may include, for example... Figure 3 Steps S1101 to S1102 are shown.

[0053] In step S1101, the query data is preprocessed, including size normalization, color space conversion, and / or text segmentation.

[0054] Data dimensional consistency is ensured through size normalization (e.g., scaling all images to a fixed resolution of 224×224); interference caused by differences in imaging devices is eliminated through color space conversion (e.g., converting RGB images to grayscale or YUV, and standardizing infrared images to 8-bit grayscale); and for text descriptions, text segmentation and encoding are performed to convert them into discrete sequences.

[0055] In step S1102, feature extraction is performed based on the preprocessed query data to determine the feature vector of the preprocessed query data.

[0056] Therefore, size normalization ensures all query data are on a uniform spatial scale, color space conversion effectively suppresses imaging noise caused by device differences, and text segmentation and encoding transform natural language into structured sequences. Based on this, feature extraction transforms the preprocessed query data into high-dimensional dense semantic vectors, accurately capturing pedestrian identity-discriminating attributes (such as clothing, body type, accessories, or semantic keywords) while filtering out irrelevant interference information. This not only improves compatibility with cross-modal data but also ensures the accuracy and consistency of subsequent mapping, similarity calculation, and target selection processes.

[0057] In some embodiments of this disclosure, the training process of the teacher model may include, for example: Figure 4 Steps S410 to S430 are shown.

[0058] In step S410, a paired dataset containing four modalities of data—RGB images, infrared images, hand-drawn sketches, and text descriptions—is constructed. Each paired dataset contains different modalities of the same pedestrian.

[0059] During the training phase, data samples belonging to the same pedestrian identity but presented in different formats (such as daytime visible light photos, nighttime infrared captures, eyewitness sketches, and natural language descriptions) are collected or synthesized to form a multimodal associated sample set. This provides cross-modal alignment supervision signals for the teacher model, enabling the teacher or student model to learn the semantic consistency between different modalities during training.

[0060] In step S420, feature extraction is performed on the modal data based on the instruction prompts to obtain the feature vector of the modal data.

[0061] The aforementioned instructional prompts are determined based on specific application scenarios and are used to explicitly embed the task objective of pedestrian re-identification into modal data. They can focus on the discriminative attributes of pedestrian identity (such as clothing, body shape, accessories, etc.) during the feature extraction process, and suppress modality-specific interference information such as imaging conditions and background environment, thereby generating task-specific feature vectors and achieving semantic alignment and consistent understanding of cross-modal data.

[0062] By concatenating task-oriented instructional prompts with modal data and inputting them into the teacher model, the model is guided to focus on semantic attributes strongly related to the pedestrian re-identification task during forward propagation, while suppressing irrelevant interference factors such as imaging conditions, background noise, or ambiguity in expression.

[0063] In step S430, a loss function for the teacher model is constructed based on the feature vectors of the modality data to optimize the training of the teacher model; the loss function includes identity recognition loss, triplet loss and / or similarity distribution matching loss.

[0064] During training, multiple supervisory signals are jointly optimized to guide the teacher model to learn pedestrian representation capabilities with strong discriminativeness and cross-modal consistency: Identity Loss ensures that different modal data of the same identity are mapped to similar positions in the feature space; Triplet Loss strengthens the distance boundary between positive sample pairs (different modalities of the same person) and negative sample pairs (different people), improving inter-class separation; while Similarity Distribution Matching Loss (SDM Loss) further narrows the semantic distribution consistency between cross-modal samples, enabling heterogeneous data such as images, text, and sketches to form a unified and compact clustering structure in the shared space.

[0065] In one specific embodiment, a trainable low-rank adapter (i.e., a LoRA adapter) is added to the linear transformation matrix of the target level in the teacher model to extract features from the modal data.

[0066] Specifically, a hierarchical LoRA-SFT strategy is adopted, applying LoRA adapters to the top layers (especially the last four layers) of the visual Transformer model and the Large Language Model (LLM) in the teacher model, and applying them to lower layers through sparse (α) settings. That is, for the lower layers beyond the last four layers of the visual Transformer model and the LLM, LoRA is performed on the first layer every α layers. This allows the teacher model to quickly adapt to the Cross Modal Person Re-identification (CM-ReID) task while retaining the general knowledge of the base model. Adapters are densely deployed only at the top layers to capture high-level semantic features, and sparsely inserted at lower layers to retain the model's general capabilities, thus significantly improving the model's representation and matching ability for heterogeneous modal data such as RGB, infrared, sketches, and text with extremely low parameter increments. This avoids the high computational cost and overfitting risk of full parameter fine-tuning while ensuring the model's generalization ability.

[0067] Therefore, by constructing a four-modal pairing dataset containing RGB images, infrared images, hand-drawn sketches, and text descriptions, and combining it with instructional prompts to guide feature extraction, the teacher model achieved strong discriminative representation learning of heterogeneous pedestrian data within a unified semantic space. By jointly optimizing the identity recognition loss, triplet loss, and similarity distribution matching loss, the teacher model not only strengthened the semantic consistency of cross-modal samples of the same person but also improved the inter-class separation between different identities, significantly enhancing the robustness of cross-modal retrieval. Furthermore, a hierarchical LoRA adapter strategy was introduced, enabling the teacher model to efficiently adapt to cross-modal pedestrian re-identification tasks with only minor parameter adjustments, retaining the general capabilities of the basic model while achieving task-oriented and accurate feature modeling.

[0068] Regarding step S420, based on the instruction prompts, feature extraction is performed on the modal data to obtain the feature vector of the modal data. In some embodiments of this disclosure, this may include, for example: Figure 5 Steps S510 to S520 are shown.

[0069] In step S510, the modal data is forward-propagated based on the instruction prompt words to obtain the final hidden state of the modal data.

[0070] The instruction prompts are concatenated with the original modal data and input into the teacher model. The model then processes the data layer by layer through a multi-layer Transformer encoder, dynamically guiding it to focus on key attributes relevant to the person re-identification task while preserving contextual semantic relationships. The final output is a set of final hidden states containing rich semantic information. These final hidden states represent a deep semantic abstraction of the input data under specific task instructions by the teacher model.

[0071] In step S520, pooling is performed on the final hidden state of the modal data to obtain the feature vector of the modal data.

[0072] From the final hidden state of the modal data, high-dimensional, sequential semantic representations are compressed into a fixed-dimensional dense vector—the feature vector of the modal data—through methods such as Global Average Pooling (GAP), [CLS] token extraction, or attention-weighted pooling. This eliminates differences in sequence length, preserves the most discriminative global semantic information, and allows different modal data, even with varying input formats, to be mapped to a unified dimensional space, facilitating subsequent similarity calculations and cross-modal matching.

[0073] Therefore, it ensures the high comparability and discriminative power of cross-modal features, and also improves the generalization ability and retrieval accuracy of the teacher model in complex real-world scenarios, providing a high-quality and highly stable semantic foundation for subsequent similarity calculation, spatial mapping and target selection.

[0074] In some embodiments of this disclosure, the training process for the student model may include, for example: Figure 6 Steps S610 to S630 are shown.

[0075] In step S610, a paired dataset containing four modalities of data—RGB images, infrared images, hand-drawn sketches, and text descriptions—is constructed. Each paired dataset contains different modalities of the same pedestrian.

[0076] Step S610 is the same as step S410, and will not be described in detail here.

[0077] In step S620, feature extraction is performed on the modal data to obtain the feature vector of the modal data.

[0078] In step S630, the feature vectors of the modal data generated by the teacher model are used as supervision signals to construct principal component mapping loss, feature relation loss and cosine similarity matching loss to optimize the training of the student model.

[0079] Using the feature vectors of modal data generated by the teacher model as supervision signals, principal component mapping loss, feature relation loss, and cosine similarity matching loss are constructed. The aim is to guide the lightweight student model to efficiently inherit the teacher model's powerful cross-modal semantic representation capabilities at the edge through a multi-level knowledge distillation mechanism.

[0080] Principal component mapping loss ensures that the features extracted by the student model are consistent with those of the teacher model in terms of global distribution, capturing the main directions of change in the semantic space. Feature relation loss constrains the pairwise similarity structure between samples (such as the distance relationship between positive and negative sample pairs), enabling the student model to learn the discriminative ranking logic implicit in the teacher model. Cosine similarity matching loss further narrows the directional consistency between students and teachers in the semantic space, strengthening cross-modal semantic alignment.

[0081] Therefore, through a triple distillation mechanism of principal component mapping loss, feature relation loss, and cosine similarity matching loss, the lightweight student model is guided to efficiently inherit the teacher model's powerful cross-modal semantic representation capabilities at the edge. The synergistic effect of these three mechanisms improves the accuracy and deployment efficiency of heterogeneous queries (such as infrared images, sketches, and text) without increasing the number of parameters in the student model.

[0082] In some embodiments of this disclosure, determining the principal component mapping loss may include, for example: Figure 7 Steps S710 to S730 are shown.

[0083] In step S710, singular value decomposition is performed on the feature matrix composed of feature vectors generated by the teacher model to obtain the principal components of the target dimension.

[0084] Singular value decomposition (SVD) is performed on the feature matrix formed by the feature vectors generated by the teacher model in order to extract the most discriminative and representative low-dimensional principal component directions from the high-dimensional semantic space, thereby constructing a compact and semantically rich subspace.

[0085] Optionally, the feature vectors of the modal data extracted from the teacher model are arranged in rows or columns to form an N×D dimensional matrix (N is the number of modal data and D is the feature dimension). The left singular vector, singular value and right singular vector are obtained by singular value decomposition. The right singular vectors corresponding to the first k largest singular values ​​are selected as the principal components of the target dimension. These principal components capture the most important change directions in the semantic space of the teacher model (such as clothing differences, body shape, accessory style, etc.).

[0086] In step S720, the feature vectors generated by the student model and the feature vectors generated by the teacher model are projected onto the vector space composed of the principal components of the target dimension.

[0087] Using the first k principal component directions obtained from singular value decomposition (i.e. the first k columns of the right singular matrix), linear transformations are performed on the high-dimensional feature vectors output by the teacher model and the student model, respectively, compressing and mapping them into a low-dimensional, semantically compact subspace.

[0088] In step S730, the similarity loss between the feature vectors generated by the projected student model and the feature vectors generated by the projected teacher model is calculated, and the principal component mapping loss is determined.

[0089] The principal component mapping loss is determined by quantifying the degree of difference between the feature vectors generated by the projected student model and the feature vectors generated by the projected teacher model in the core semantic direction using a pre-defined contrast loss function (such as cosine similarity loss, L2 distance, or KL divergence).

[0090] Specifically, the teacher model exhibits significant low-rank characteristics in the person re-identification feature space. To address this low-rank characteristic, a principal component mapping loss is designed. This loss maps the person re-identification features generated by the teacher and student models (i.e., feature vectors of the query data or image features of candidate images) to the person re-identification features of the teacher model, specifically the features containing the first... The vector space of principal components guides the student model to prioritize capturing the most critical discriminative information, namely: in, This represents the principal component mapping loss. It is any contrastive distillation loss function, such as the cosine contrastive loss function. This refers to the extraction of features from the right singular matrix after performing singular value decomposition on the pedestrian re-identification features of the teacher model. Principal components, This represents the feature vector extracted by the teacher model from the m-th modality of data. This represents the feature vector extracted by the student model from the m-th modality data.

[0091] Therefore, the core discriminative semantic structure of the teacher model is accurately inherited. While retaining key identity attributes such as clothing, body shape, and accessories, redundant and noisy dimensions are effectively suppressed, enabling the student model to achieve semantic representation capabilities highly consistent with the teacher model at the edge with minimal parameter cost. By calculating the similarity loss between projected teacher and student features, the approximation ability of the student model in the dominant semantic direction is further enhanced, improving the retrieval accuracy, robustness, and generalization ability in cross-modal person re-identification tasks.

[0092] The technical solution of this disclosure will be further explained below with a specific implementation example.

[0093] like Figure 8As shown, existing methods indirectly apply multimodal large-scale models to visual question-answering-based retrieval (which is prone to illusion problems in long visual contexts due to limitations in image database size) or text distillation paradigms (only applicable to pure text person re-identification). The person re-identification method proposed in this disclosure directly uses a cloud-based multimodal large-scale language model (Cloud MLLM) as a unified teacher model, adapting it to multimodal inputs. Its unified knowledge is distilled into a lightweight edge student model to meet practical deployment requirements.

[0094] To overcome the challenges of fragmentation, limited application of MLLM, and difficult edge deployment in existing cross-modal person re-identification (CM-ReID) technologies, this paper proposes an innovative cloud-based teacher model fine-tuning strategy and an edge student model knowledge distillation mechanism to achieve end-to-end unified processing and efficient deployment of multimodal (RGB, infrared, hand-drawn sketch, text) person re-identification tasks.

[0095] like Figure 9 As shown, the cloud-based teacher model includes integrated task instructions and modal cue words, a multimodal large-scale model backbone network (Qwen2-VL), pooling operations, identity recognition loss (ID loss), triplet learning (Triplet loss), and a similarity distribution matching (SDM) module. The edge student model mainly includes a visual language model backbone network (CLIP (ViT-L / 14)), modal projection layers, distillation matching loss (such as cosine loss), principal component mapping loss (PCM loss), and feature relation loss (FR loss). Both the cloud-based teacher model and the edge student model can uniformly complete cross-modal person re-identification tasks end-to-end.

[0096] Among them, the Qwen2-VL-2B multimodal large language model (i.e., the teacher model) is used as the backbone. The multimodal large language model is guided to generate a unified embedding space among multiple modalities such as RGB, infrared (IR), sketch, and text through instruction-based prompting.

[0097] To efficiently fine-tune a multimodal large language model, a hierarchical LoRA-SFT strategy is adopted, applying the LoRA adapter to the top layers (especially the last four layers) of the visual Transformer and LLM, and through sparse ( The setting applies to lower layers, specifically: for layers other than the last four layers of the Visual Transformer and LLM, every [percentage missing]. The first layer is used for LoRA. This allows the multimodal large language model to quickly adapt to cross-modal person re-identification tasks while retaining the general knowledge of the base model.

[0098] At the same time, a combined task loss method is adopted ( The loss includes Identity Loss, Triplet Loss, and Similarity Distribution Matching Loss (SDM Loss) to ensure that the multimodal large model generates a highly discriminative and modality-invariant unified feature representation, effectively solving the cross-modal alignment problem.

[0099] In addition, such as Figure 10 As shown, this disclosure is the first to discover and empirically demonstrate that the cloud-based teacher model exhibits significant low-rank characteristics in the pedestrian re-identification feature space, that is, discriminative information is concentrated in a few principal components, which provides a theoretical basis for efficient distillation.

[0100] To address the low-rank characteristic of the teacher model, this disclosure designs a principal component mapping loss, which maps the person re-identification features of the cloud model (i.e., the teacher model) and the edge model (i.e., the student model) to the pre-inclusion features of the person re-identification features of the cloud model. The vector space of principal components guides the edge model to prioritize capturing the most critical discriminative information, namely: in, This represents the principal component mapping loss. It is any contrastive distillation loss function, such as the cosine contrastive loss function. This refers to the extraction of features from the right singular matrix after performing singular value decomposition on the pedestrian re-identification features of the teacher model. Principal components, This represents the feature vector extracted by the teacher model from the m-th modality of data. This represents the feature vector extracted by the student model from the m-th modality data.

[0101] To further preserve the relational structure in the feature space of the teacher model while also considering non-principal components, this disclosure introduces Feature Relation Loss (FR Loss). By aligning the semantic relationships between features, Feature Relation Loss indirectly enables the student model to learn information in less important dimensions, compensating for the loss of detail that may be caused by principal component mapping loss and improving the comprehensiveness of distillation.

[0102] Furthermore, the distillation process also includes a basic cosine comparison loss, used to directly and globally align pedestrian re-identification features between cloud and edge models, together forming the complete distillation loss function. This ensures that the edge model effectively inherits the unified intelligence of the cloud model.

[0103] Specifically, during inference, the cloud-based teacher model processes multimodal data through instructional prompts and extracts globally perceptive pedestrian re-identification features from its final hidden state using pooling operations. The edge student model adopts the CLIP (ViT-L / 14) architecture, using a visual Transformer and a text encoder for visual modalities (RGB, infrared, sketch) and text modalities respectively, and aligns its feature space with that of the cloud-based teacher model through a linear projection layer, thereby achieving end-to-end processing of cross-modal tasks.

[0104] Therefore, this paper innovatively adapts multimodal large language models end-to-end as a unified backbone for the CM-ReID task, enabling efficient feature extraction from multimodal data (RGB, infrared, sketches, and text). It reveals the low-rank characteristics of the feature space of multimodal large language models and designs a novel knowledge distillation strategy based on this. Through principal component mapping loss and feature relation loss, it achieves structured and efficient transfer of knowledge from the cloud-based teacher model to the lightweight edge student model. Finally, a complete edge-cloud collaborative solution is constructed to ensure high-performance deployment of unified intelligence at the multimodal large language model level on resource-constrained devices.

[0105] In terms of performance, the cloud-based teacher model disclosed herein achieves state-of-the-art results in multiple cross-modal tasks on the QrCM-ReID dataset (CUHK-PEDES, ICFG-PEDES, and RSTPReid obtained through modal augmentation and merging). For example, in the infrared image to visible light image (IR→R) task, the model achieves R1 and mAP of 92.87% and 90.28% respectively on the CUHK-PEDES dataset, 86.77% and 59.34% respectively on the ICFG-PEDES dataset, and 85.81% and 72.91% respectively on the RSTPReid dataset. In the text description to visible light image (T→R) task, the cloud-based model outperforms all existing methods in mAP on the three test sets, and its Rank-n accuracy is very close to the state-of-the-art level, even without introducing any custom modules and operations specifically for text-to-image ReID tasks.

[0106] The lightweight edge student model, after efficient knowledge distillation, also maintains excellent performance on the visual CM-ReID task. For example, in the S→R task, the edge model achieves an R1 of 89.14% and an mAP of 85.23% on CUHK-PEDES; an R1 of 88.10% and an mAP of 58.90% on ICFG-PEDES; and an R1 of 87.01% and an mAP of 75.82% on RSTPReid. This is thanks to the invention's revelation of the low-rank characteristics of the MLLM feature space and the design of innovative distillation strategies (PCM Loss and FR Loss) based on this, which effectively solves the problem of poor adaptability of traditional distillation methods to the complex structure of MLLMs, and achieves efficient structured knowledge transfer.

[0107] like Figure 11 As shown, the left image is the t-SNE visualization result of the cloud-based teacher model, and the right image is the t-SNE visualization result of the edge-based student model. Different shapes of scatter points correspond to different modalities of data, and different colors of scatter points represent different pedestrian identification numbers.

[0108] like Figure 12 As shown, the left image is the recognition result of the cloud-based teacher model, and the right image is the recognition result of the edge-based student model. In the figure, rows (1) and (2) correspond to the infrared image to visible light image (IR→R) task, and rows (3) and (4) correspond to the text description to visible light image (T→R) task. In row (3), the text description A is "The woman has a ponytail, is wearing a black down jacket, black trousers and black boots, a purple scarf and glasses, and has one hand in her pocket while walking." In row (4), the text description B is "The woman is wearing a black coat, black trousers and black shoes, glasses and a purple scarf, and is looking at her phone while walking." In summary, the pedestrian re-identification method disclosed in this paper proposes an innovative approach at the theoretical level and demonstrates significant advantages in practical applications, providing efficient and reliable technical support for the field of cross-modal pedestrian re-identification.

[0109] Based on any of the above embodiments, this disclosure also provides a pedestrian re-identification device.

[0110] Figure 13 This is a schematic block diagram of the structure of a pedestrian re-identification device according to one embodiment of the present disclosure.

[0111] like Figure 13 As shown, the pedestrian re-identification device includes: The feature extraction module 1302 inputs the query data of the target pedestrians input by the user into the student model to extract features and determine the feature vector of the query data; the query data can be an RGB image, an infrared image, a hand-drawn sketch or a text description. The similarity calculation module 1304 performs similarity matching between the feature vector of the query data and the feature vector of the pedestrian images in the database to obtain candidate images; The target determination module 1306 filters candidate images based on their similarity and determines the target image whose similarity is greater than or equal to the similarity threshold.

[0112] The aforementioned pedestrian re-identification device can be in the form of computer software, and each module of the aforementioned pedestrian re-identification device can be implemented through computer software modules.

[0113] The specific implementation process of the functions and roles of each module in the above-mentioned pedestrian re-identification device can be found in the implementation process of the corresponding steps in the above method, and will not be repeated here.

[0114] This disclosure also provides an electronic device 1000. Figure 14 A schematic diagram of the hardware implementation using the processing system is shown.

[0115] The hardware structure of electronic device 1000 can be implemented using a bus architecture. The bus architecture can include any number of interconnect buses and bridges, depending on the specific application and overall design constraints of the hardware. Bus 1100 connects various circuits including one or more processors 1200, memory 1300, and / or hardware modules. Bus 1100 can also connect various other circuits 1400 such as peripherals, voltage regulators, power management circuits, external antennas, etc. Bus 1100 can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Component (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of representation, only one connection line is used in this figure, but this does not indicate that there is only one bus or one type of bus.

[0116] For ease of explanation, certain steps of the above method are described in relation to modules. It should be understood that the corresponding module performing one or more steps of the above method may be one or more hardware modules specifically configured to perform the corresponding step, or implemented by a processor configured to perform the corresponding step, or stored in a computer-readable medium for implementation by a processor, or implemented by some combination thereof.

[0117] This disclosure also provides a readable storage medium storing a computer program that, when executed by a processor, is used to implement the methods described above. A "readable storage medium" can be any means capable of containing, storing, communicating, propagating, or transmitting a program for use by or in conjunction with an instruction execution system, apparatus, or device. More specific examples of a readable storage medium include: an electrical connection with one or more wires (electronic device), a portable computer disk drive (magnetic device), random access memory (RAM), read-only memory (ROM), erasable and programmable read-only memory (EPROM or flash memory), fiber optic devices, and portable read-only memory (CDROM), etc.

[0118] This disclosure also provides a computer program product, the methods of which can be implemented wholly or partially through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented wholly or partially as a computer program product. The computer program product includes one or more computer programs or instructions. When the computer program or instructions are loaded and executed, all or part of the processes or functions of this disclosure are performed.

[0119] Computer programs or instructions can be stored in a readable storage medium or transferred from one readable storage medium to another. For example, the computer program or instructions can be transferred from one website, computer, server, or data center to another website, computer, server, or data center via wired or wireless means. The readable storage medium can be any available medium capable of access, or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium, such as a floppy disk, hard disk, or magnetic tape; an optical medium, such as a digital video optical disc; or a semiconductor medium, such as a solid-state drive. The computer-readable storage medium can be a volatile or non-volatile storage medium, or it can include both volatile and non-volatile types of storage media.

[0120] Those skilled in the art will understand that embodiments of this disclosure can be provided as methods, systems, or computer program products. Therefore, this disclosure can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this disclosure can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0121] This disclosure is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to this disclosure. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0122] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0123] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0124] In the description of this specification, the references to terms such as "one embodiment / mode," "some embodiments / modes," "example," "specific example," or "some examples," etc., refer to specific features, structures, or characteristics described in connection with that embodiment / mode or example, which are included in at least one embodiment / mode or example of this disclosure. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment / mode or example. Moreover, the specific features, structures, or characteristics described may be combined in any suitable manner in one or more embodiments / modes or examples. Furthermore, without contradiction, those skilled in the art can combine and integrate the different embodiments / modes or examples described in this specification, as well as the features of different embodiments / modes or examples.

[0125] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this disclosure, "a plurality of" means at least two, such as two, three, etc., unless otherwise explicitly specified.

[0126] Those skilled in the art should understand that the above embodiments are merely for illustrating the present disclosure and are not intended to limit the scope of the disclosure. Those skilled in the art can make other changes or modifications based on the above disclosure, and these changes or modifications still fall within the scope of the present disclosure.

Claims

1. A pedestrian re-identification method, characterized in that, Person re-identification is achieved based on a student model and a teacher model, wherein the student model and the teacher model are multimodal large language models, and the feature space of the student model is aligned with the feature space of the teacher model through knowledge distillation. The person re-identification method includes: The user-inputted query data of the target pedestrian is input into the student model for feature extraction to determine the feature vector of the query data; the query data can be an RGB image, an infrared image, a hand-drawn sketch, or a text description. The feature vectors of the query data are matched with the feature vectors of pedestrian images in the database to obtain candidate images; and Based on the similarity of the candidate images, the candidate images are filtered to determine the target image whose similarity is greater than or equal to the similarity threshold.

2. The pedestrian re-identification method as described in claim 1, characterized in that, Also includes: The target image with a similarity greater than or equal to the similarity threshold is input into the teacher model for secondary matching to determine the final target image.

3. The pedestrian re-identification method as described in claim 1, characterized in that, The training process of the teacher model includes: Construct a paired dataset containing four modalities: RGB images, infrared images, hand-drawn sketches, and text descriptions. Each paired dataset contains different modalities of the same pedestrian. Based on the instruction prompts, feature extraction is performed on the modal data to obtain the feature vector of the modal data; Based on the feature vectors of the modality data, a loss function is constructed for the teacher model to optimize the training of the teacher model; the loss function includes identity recognition loss, triplet loss and / or similarity distribution matching loss.

4. The pedestrian re-identification method as described in claim 3, characterized in that, Based on command-based prompts, feature extraction is performed on the modal data to obtain the feature vector of the modal data, including: Based on the instruction prompts, the modal data is forward-propagated to obtain the final hidden state of the modal data; Pooling is performed on the final hidden state of the modality data to obtain the feature vector of the modality data.

5. The pedestrian re-identification method as described in claim 1, characterized in that, A trainable low-rank adapter is added to the linear transformation matrix of the target level in the teacher model to extract features from the modal data.

6. The pedestrian re-identification method as described in claim 1, characterized in that, The training process of the student model includes: Construct a paired dataset containing four modalities: RGB images, infrared images, hand-drawn sketches, and text descriptions. Each paired dataset contains different modalities of the same pedestrian. Feature extraction is performed on the modal data to obtain the feature vector of the modal data; Using the feature vectors of the modal data generated by the teacher model as supervision signals, principal component mapping loss, feature relation loss, and cosine similarity matching loss are constructed to optimize the training of the student model.

7. The pedestrian re-identification method as described in claim 6, characterized in that, Determining the principal component mapping loss includes: Singular value decomposition is performed on the feature matrix composed of the feature vectors generated by the teacher model to obtain the principal components of the target dimension. Project the feature vectors generated by the student model and the feature vectors generated by the teacher model onto the vector space formed by the principal components of the target dimension; Calculate the similarity loss between the feature vectors generated by the projected student model and the feature vectors generated by the projected teacher model, and determine the principal component mapping loss.

8. The pedestrian re-identification method as described in claim 1, characterized in that, The student model is used to extract features from the user-inputted query data of the target pedestrian to determine the feature vector of the query data, including: The query data is preprocessed, including size normalization, color space conversion, and / or text segmentation. Feature extraction is performed on the preprocessed query data to determine the feature vector of the preprocessed query data.

9. An electronic device, characterized in that, include: The memory stores execution instructions; as well as A processor that executes the execution instructions stored in the memory, causing the processor to perform the pedestrian re-identification method according to any one of claims 1 to 8.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the pedestrian re-identification method according to any one of claims 1 to 8.

Images