A lightweight prompt learning pedestrian re-identification method fusing side information embedding

Abstract

The application discloses a kind of fusion side information embedding light-weight hint learning pedestrian re-identification method, belong to computer vision and intelligent transportation technical field.The purpose is to solve the semantic difference, feature noise, structural redundancy and scene robustness insufficient when general vision-language pre-training model adapts fine-grained pedestrian re-identification.The method uses two-branch architecture and two-stage training paradigm, through normalization entropy analysis, the dimension of CLIP visual encoder FFN hidden layer is reduced from 4D to 2D, camera ID side information is only injected into [CLS] token, and double-type visual features are extracted;Identity-specific learnable prompt sequence is generated, cross-modal features are extracted by freezing text encoder;After double-branch feature customization, multi-loss joint optimization is carried out, pre-training knowledge reservation and task adaptation are cooperated, recognition accuracy and deployment efficiency are considered, feature noise is suppressed, complex scene robustness is improved, and is suitable for cross-camera pedestrian retrieval tasks such as intelligent security.

Description

Technical Field

[0001] This invention belongs to the field of computer vision, and in particular relates to a lightweight cue-based pedestrian re-identification method that integrates side information embedding. It is applicable to pedestrian identification tasks in scenarios such as cross-camera pedestrian retrieval, intelligent security, and smart cities, and has excellent robustness, especially in complex scenarios such as occlusion, multiple viewpoints, and changes in lighting. Background Technology

[0002] Person re-identification (ReID), a core technology for cross-camera target retrieval, aims to identify the same pedestrian from non-overlapping images or videos captured by different cameras. It is a key supporting technology for intelligent security and smart city construction. In recent years, person re-identification methods have evolved from hand-designed features to deep convolutional networks, and then to visual Transformer architectures. Cross-modal person re-identification based on large-scale vision-language pre-trained models (such as CLIP) has become a cutting-edge research direction in this field.

[0003] The CLIP model, pre-trained on massive image-text datasets, possesses powerful cross-modal semantic alignment capabilities and general visual feature extraction capabilities. Transferring it to person re-identification tasks can bridge the semantic understanding gap of single visual modalities, potentially achieving zero-shot generalization. However, adapting the general CLIP model to fine-grained person re-identification tasks still faces several key technical bottlenecks: Transferring general vision-language pre-trained models (such as CLIP) to fine-grained pedestrian re-identification tasks still faces several key technical bottlenecks: CLIP pre-trained knowledge is geared towards general visual recognition and lacks the ability to model fine-grained features of pedestrian identity; fixed text descriptions cannot adapt to the identity-specific semantics of different pedestrians, and direct transfer will lead to insufficient feature discriminative power; its visual encoder's residual connections and attention outputs have different normalized entropy distributions. As the network depth increases, the residual connection entropy value approaches 0, and feature extrema are concentrated in a few channels, introducing a large amount of feature noise and reducing the spatial resolution and discriminative power of the features; at the same time... The visual encoder feedforward network (FFN) is expanded by 4 times, resulting in a large number of parameters and high computational complexity, making it difficult to adapt to resource-constrained deployment scenarios such as edge devices and embedded devices. Existing methods do not fully integrate information from camera perspective and shooting scene, and feature matching is prone to failure due to missing scene context information in scenarios such as cross-camera, occlusion, and perspective changes. In addition, fine-tuning all parameters of the CLIP encoder can easily destroy its pre-trained cross-modal semantic alignment knowledge, leading to a decrease in the model's generalization ability, while a simple fixed encoder strategy cannot fully adapt to the fine-grained task requirements of pedestrian re-identification.

[0004] To overcome the shortcomings of existing technologies, a lightweight cue-based pedestrian re-identification method (Lite_SIE_CLIPReID) with fused side information embedding is proposed. This method addresses issues such as semantic differences, feature noise, structural redundancy, and missing scene information when transferring the general CLIP model to pedestrian re-identification. It improves the model's discriminative power for fine-grained pedestrian features and robustness to complex scenes while preserving the cross-modal knowledge of CLIP pre-training, while simultaneously achieving model lightweighting to meet the efficiency requirements of practical deployment. Summary of the Invention

[0005] To achieve the above-mentioned objectives, the technical solution adopted by this invention is as follows: a lightweight cue-learning pedestrian re-identification method that integrates side information embedding. The overall process is input → feature extraction → feature processing → loss optimization → inference output. It adopts a dual-branch architecture that processes image streams and text streams in parallel. The model training is completed through a two-stage training paradigm (Stage 1: cue learning stage, Stage 2: end-to-end fine-tuning stage). Finally, cross-modal feature alignment is achieved in the loss optimization stage, and features are flexibly selected in the inference stage to complete cross-camera pedestrian retrieval. The steps of a lightweight cue-based pedestrian re-identification method that incorporates side information embedding are as follows: Step 1: Pedestrian image preprocessing and lightweight visual encoder construction; First, normalization and data augmentation operations are performed on the input pedestrian RGB image. Based on the CLIP-ViT-B / 16 model architecture, the positional encoding length is adjusted by bicubic interpolation resampling to match the sequence length after image segmentation. The core innovation of this step is the lightweight transformation of the feedforward network FFN: the dimension of the MLP hidden layer is halved from 4D to 2D, where D=768, the weights are cropped and scaled to maintain variance consistency, and a lightweight image / text encoder Lite_Encoder is constructed to reduce the number of model parameters and inference latency. Specifically, this step is described in detail below: a1. Pedestrian image preprocessing: First, perform normalization and data augmentation operations on the input raw pedestrian RGB image: first, normalize the pixel values ​​according to the formula... pixel norm = pixel / 255 scales to the [0,1] range. pixel It takes the pixel values ​​of the input image and eliminates differences in pixel value magnitude; then it performs enhancement operations such as random horizontal flipping, edge padding, random cropping, and random erasing to improve the model's generalization ability. a2. Position-encoded resampling: ViT-B / 16 divides the image into image blocks and encodes the corresponding positions. After dividing the blocks into non-overlapping blocks according to size, the image blocks are obtained. The position codes pre-trained by CLIP are subjected to bicubic interpolation resampling operation to adjust the position code length so that the position code matches the length of the image block sequence. The position code dimension remains unchanged after resampling. a3. Lightweight modification of feedforward network: CLIP uses ViT-B / 16 as its backbone, and its final output is the superposition of residual connections and attention outputs. However, the residual connections of ViT-B / 16 suffer from the problem of feature extrema concentrated in a few channels, causing most feature vectors to point in similar directions in the latent space. This makes it difficult to effectively distinguish features at different spatial locations using cosine similarity, thus introducing noise interference and reducing feature discriminative power. To overcome these shortcomings, a lightweight modification is made to the feedforward network FFN of ViT-B / 16. All pre-trained parameters of the residual connections, layer normalization module, and multi-head attention module are retained, and the hidden layer dimension of the feedforward network FFN is halved from the original 4D to 2D. a4. Encoder initialization: The modified feedforward network FFN is integrated with the resampled positional encoder and other retained ViT-B / 16 modules to construct a lightweight visual encoder. When loading the CLIP pre-trained weights, the weights of FFN are cropped and scaled to complete the encoder initialization. The pre-processed pedestrian RGB image is then output and fed into the subsequent dual-type visual feature extraction network for further pedestrian re-identification. Step 2: Extraction of dual-type visual features from fused side information; An embedding network is constructed to generate a bias vector of camera viewpoint side information. The precise side information embedding only applies to the [cls] token and does not interfere with the image patch features. The image is divided into blocks and converted into feature vectors. The [cls] token containing side information is concatenated to form an input sequence. After forward propagation by the encoder, the global semantic feature F_cls corresponding to the [cls] token and the local detail feature F_visual obtained by the mean pooling of the image patches are extracted to achieve dual-type visual feature extraction. Specifically, this step is described in detail below: b1. Generation of side information bias vector: A single-hidden-layer embedding network is constructed. The weight matrix and bias vector of the embedding network are initialized using a normal distribution. The camera ID is converted to one-hot encoding and then input into the embedding network. After linear transformation, the camera view-side information bias vector SIE is output. cam The parameters embedded in the network are trainable parameters; b2. Precise side information embedding: Side information bias vector SIE camIn the form of element-wise addition, only the original [cls] token is embedded into the lightweight visual encoder. cls ] base In the process, the [cls] token with fused side information is obtained; only the [cls] token is used for side information embedding, without involving any image block tokens, to avoid side information interfering with image detail features; b3. Image patch sequence generation: The preprocessed pedestrian RGB image is divided into non-overlapping blocks of a certain size, along the height and width directions; a linear projection operation is performed on each image block to convert it into a one-dimensional feature vector, resulting in an image block sequence. b4. ViT standard input sequence construction: Add a [cls] token containing fusion side information at the beginning of the image patch sequence. cls ] SIE This forms the ViT standard input sequence S. input The first line is [ cls ] SIE Rows 2-129 are image patch feature sequences; the resampled position codes are then compared with the input sequence S. input Element-wise addition is performed to superimpose spatial location information, resulting in the final encoder input X. in ; b5. Dual-type visual feature extraction: The final input will be X. in Input a lightweight visual encoder, perform forward propagation processing, and obtain the encoder output X. out X output from encoder out Extract two types of visual features: (a) Visual-like feature F_cls: The first row of the encoder output sequence contains [...]. cls ] SIE Features are obtained by performing a linear transformation on the projection layer at the top of the encoder, capturing the global semantic information of pedestrians; (b) Main visual feature F_visual: Perform global mean pooling operation on the 128 image block token features in rows 2 to 129 of the encoder output sequence, aggregate local detail information of pedestrians, and enhance robustness to occluded scenes; The output visual features F_cls and the main visual features F_visual flow to the dual-branch visual feature lightweight processing stage. As the core features of the visual flow, they will undergo customized lightweight processing separately and will not directly participate in the loss calculation. Step 3: Input ID tags and generate identity-specific learnable text prompt sequences; Input pedestrian identity ID tags as supervised training tags for pedestrian re-identification tasks; pre-trained word embedding matrix of CLIP model; convert static pedestrian identity ID tags into dynamic, identity-specific learnable text prompt sequences to bridge the semantic differences between general CLIP model and fine-grained pedestrian re-identification tasks. Specifically, this step is described in detail below: c1. Learnable context vector initialization: For each pedestrian identity category in the training set, construct a learnable context vector. E learn The learnable context vectors of all identity categories are integrated into a learnable matrix, initialized with a normal distribution, and are trainable parameters; c2. Fixed text word embedding generation: Select text with a fixed prefix and text with a fixed suffix, and segment both texts according to CLIP's text segmentation rules to obtain prefix and suffix sequences. Then, using the CLIP pre-trained word embedding matrix, convert the token sequences of the two texts into fixed word embeddings. E prefix and E suffix ; c3. Concatenation of identity-specific hint sequences: Based on the pedestrian's identity ID label, the learnable context vector of the corresponding category is selected from the learnable matrix. E learn ;Will E prefix , E learn , E suffix By concatenating the tokens sequentially, a learnable text prompt sequence P, unique to the identity, is generated. The text prompt sequence P flows to frozen text feature extraction, serving as the core input of the text stream for extracting cross-modal text features; Step 4: Frozen text feature extraction and cross-modal dimension alignment; Load the pre-trained weights of CLIP text encoder and fix the parameters. Extract features from the input text prompt sequence. After linear transformation and L2 normalization, obtain cross-modal text features that are aligned with the visual feature dimensions, while fully preserving the pre-trained knowledge of CLIP text encoder. The input identity-specific learnable text prompt sequence P and the complete pre-trained weights of the pre-trained CLIP text encoder, including a multi-head attention module, a layer normalization module, a feedforward network, and a text projection layer; Specifically, this step is described in detail below: d1. CLIP text encoder frozen: Load all the pre-trained weights of the CLIP text encoder, set all the encoder parameters to be untrainable, and keep the encoder parameters unchanged during subsequent model training, thus fully preserving its pre-trained cross-modal semantic knowledge; d2. Forward extraction of text features: Input the text prompt sequence P into the frozen CLIP text encoder, perform forward propagation processing, and add the CLIP text pre-trained positional encoding to P element by element; d3. Cross-modal dimension alignment: Take the last line of the CLIP text encoder output sequence TextOutput, corresponding to <sep>The token's features are input into the CLIP pre-trained text projection layer, where a linear transformation is performed, followed by L2 normalization, ultimately yielding text features T that are perfectly aligned with the visual feature dimensions of the image stream. Text features T flow to multi-loss joint optimization training, serving as the computational input for cross-modal alignment loss, and participate in model optimization in conjunction with visual features; Step 5: Lightweight and customized processing of dual-branch visual features; To address the different attributes of F_cls and F_visual, a dual-branch lightweight customized processing path is designed: F_cls is dimensionality-reduced and lightweighted through a projection bottleneck layer, while F_visual undergoes a dimension-preserving nonlinear transformation through the projection bottleneck layer. The parameters of the two branches are independent of each other. Combined with one-dimensional batch normalization BatchNorm1d, F_cls_norm and F_visual_norm are obtained respectively, which improves the model inference efficiency and enhances the discriminative power of features. Specifically, this step is described in detail below: e1. Class feature branch processing: F_cls is dimensionality reduced and lightweighted by passing it through the projection bottleneck layer and the BatchNorm1d layer in sequence, and then the normalized visual feature F_cls_norm is obtained. e2. Main visual feature branch processing: F_visual is subjected to a nonlinear transformation process that preserves its dimension, passing through a projection bottleneck layer and a one-dimensional batch normalization BatchNorm1d layer in sequence, and the normalized main visual feature F_visual_norm is obtained after processing. The standardized class visual features F_cls_norm and standardized main visual features F_visual_norm are output and flow together to the multi-loss joint optimization training stage, serving as the input for calculating the identity classification loss and triplet loss; at the same time, they flow to the re-identification inference output stage, serving as the basis for the final re-identification features; Step Six: End-to-end fine-tuning and training of the multi-loss joint optimization model. By jointly optimizing multiple losses such as identity classification loss, triplet loss, and cross-modal alignment loss, the trainable parameters of the model are fine-tuned end-to-end, achieving a synergistic improvement in visual feature discriminative power and cross-modal feature consistency. Specifically, this step is described in detail below: f1. Construction of the projection classifier: Two independent projection classifiers are constructed, adapted to F_cls_norm and F_visual_norm respectively. Each classifier consists of a single linear transformation layer, and the output dimension is the total number of pedestrian identity categories C in the training set. The parameters of both classifiers are trainable parameters, and the initial values ​​are initialized using a normal distribution. f2. Calculation of identity classification loss: Input F_cls_norm into classifier 1, denoted as Cla1, to obtain the identity prediction score Logits1; input F_visual_norm into classifier 2, denoted as Cla2, to obtain the identity prediction score Logits2; use the cross-entropy loss function to calculate the identity classification losses Lid1 and Lid2 of the two branches respectively, and then take the average to obtain the total identity classification loss L_id, thereby enhancing the class discrimination power of visual features and realizing intra-class feature aggregation; f3. Triplet loss calculation: In the batch samples, one positive sample and four negative samples with the same identity are selected for each anchor sample to construct a triplet sample set; the triplet loss L is calculated based on F_cls_norm and F_visual_norm respectively. tri1 and L tri2 Then, the average value is taken to obtain the total triplet loss L_tri, which increases the inter-class feature distance and achieves the dual goals of intra-class feature aggregation and inter-class feature separation. f4. Calculation of cross-modal alignment loss: First, perform L2 normalization on F_visual_norm, and then calculate the cosine similarity loss with the normalized text feature T to obtain the cross-modal alignment loss L_i2t, which improves the cross-modal alignment accuracy between visual features and text features and makes full use of CLIP's cross-modal semantic knowledge. f5. Construction of the total loss function: A total loss function is constructed based on a weight configuration that prioritizes visual discrimination and secondarily considers cross-modal alignment. L total = λ1 L id +λ2L tri +λ3L i2t λ1 is the identity classification loss weight, λ2 is the triplet loss weight, and λ3 is the cross-modal alignment loss weight; ensuring the model focuses on pedestrian identity discrimination while fully utilizing CLIP's cross-modal semantic knowledge to improve feature generalization ability; all loss calculations are based on batch samples. f6. End-to-end model fine-tuning: The SGD stochastic gradient descent optimizer is used, with the total loss function... L total To optimize the objective, the trainable parameters of the model are fine-tuned end-to-end; f7. Parameter Update: Backpropagation computes the gradients of all trainable parameters, and the SGD optimizer performs parameter updates. The parameters are updated once after each batch of training is completed. After each epoch of training, the model's mAP and Rank-1 metrics are evaluated on the validation set, and the parameter set of the model with the best performance on the validation set is saved for subsequent inference. The optimized model parameter set is output and then fed back to the lightweight visual encoder, learnable context vectors, and projection bottleneck layer to complete the training loop of the entire model. Step 7: Cross-camera pedestrian re-identification reasoning and result output; Re-identify features, perform similarity retrieval, and output accurate results.

[0006] Input standardized class visual features F_cls_norm and standardized main visual features F_visual_norm; Gallery set image feature library, which is a set of standardized features obtained from all pedestrian images to be retrieved and stored in the database, supports fast similarity retrieval; Inference configuration is based on the actual deployed hardware, accuracy requirements and scene characteristics, such as regular monitoring and heavily occluded scenes.

[0007] Specifically, this step is described in detail below: g1. Selection of final re-identification features: High-precision inference feature fusion involves concatenating features according to their dimensions to obtain fused features. F concat =F_cls_norm⊕F_visual_norm, which combines global semantic information with local detail information.

[0008] g2. Feature normalization: L2 normalization is performed on the selected final re-identification features. After normalization, the L2 norm of the features is 1, eliminating the influence of feature magnitude on similarity calculation and ensuring the accuracy of cosine similarity calculation; g3. Cross-camera similarity retrieval: Normalized query features F norm Input the pedestrian re-identification retrieval system and calculate its cosine similarity with all features in the gallery feature library; g4. Sorting and outputting search results: The gallery samples are sorted from highest to lowest based on their cosine similarity to the query features. A higher similarity score indicates a better match between the sample and the queried pedestrian. The final output of the retrieval system includes: the image path or unique identifier of the matched sample, the matching ranking, the similarity score, and the pedestrian identity label; it also outputs cross-camera pedestrian re-identification retrieval results, a structured matching list sorted by similarity, for use by practical business systems such as security monitoring and traffic management.

[0009] The lightweight cue-based pedestrian re-identification method proposed in this invention, which integrates Lite_SIE_CLIPReID side information embedding, has the following significant advantages compared to existing technologies: 1. Address the semantic discrepancy between general models and fine-grained tasks: Design an identity-specific learnable text prompt generation module. Using a three-part structure of "fixed prefix + learnable context vector + fixed suffix", it converts static pedestrian ID tags into dynamic identity-specific semantic descriptions without the need for manual text design. This achieves efficient adaptation of CLIP pre-trained knowledge to fine-grained pedestrian re-identification tasks and improves the identity discrimination power of features.

[0010] 2. Suppress deep feature noise and improve feature discriminativeness: Through normalized entropy difference analysis, the feature distribution difference between the residual connection and attention output of the CLIP visual encoder was found. The dimension of the FFN hidden layer was halved from 4D to 2D to filter feature noise, while retaining the pre-trained parameters of the residual connection and attention module to ensure the stability of feature transfer and spatial resolution.

[0011] 3. Achieving lightweight model while balancing accuracy and efficiency: The lightweight transformation of FFN based on entropy analysis halves the number of FFN parameters while maintaining consistent feature variance through weight pruning and scaling, without causing performance degradation; at the same time, a dual-branch feature lightweight processing path is designed to reduce the dimensionality of global features and preserve the maintenance of local features, which greatly reduces the computational complexity of the model and meets the deployment requirements of resource-constrained scenarios such as edge devices and embedded devices.

[0012] 4. Accurately integrate side information to improve robustness in complex scenes: Camera view side information is embedded only into the [cls] token to avoid interfering with the extraction of image patch detail features. This allows visual features to perceive shooting scene information and better model the association between occlusion patterns and identity features in scenarios such as cross-camera, occlusion, and viewpoint changes. This significantly improves the robustness of the model in complex scenes, with mAP and Rank-1 improvements of 0.9% and 1.2% respectively on the occlusion robustness dataset Occluded-Duke.

[0013] 5. Two-stage training paradigm to avoid forgetting pre-training knowledge: The two-stage training adopts "cue learning + end-to-end fine-tuning". Stage 1 freezes the core parameters of CLIP encoder and trains only the learnable cue and side information embedding network to complete cross-modal basic alignment. Stage 2 unfreezes the lightweight encoder and jointly optimizes multiple loss functions to achieve fine-grained task adaptation, effectively retaining CLIP's cross-modal semantic alignment knowledge and improving the model's generalization ability.

[0014] 6. Cross-modal feature alignment to enhance semantic understanding: Construct an image stream-text stream dual-branch architecture, jointly optimize cross-modal alignment loss during the loss optimization stage, achieve accurate alignment of visual features and text features, make full use of CLIP's cross-modal semantic knowledge, bridge the semantic understanding gap of a single visual modality, and improve the model's semantic understanding of pedestrian appearance.

[0015] 7. Strong modular versatility and easy migration and expansion: The lightweight visual encoder, learnable prompt generation module, and side information embedding module of this invention can all be inserted as independent modules into other pedestrian re-identification frameworks and used as a strong baseline. They have good versatility and scalability and can be migrated to other target re-identification tasks such as vehicle re-identification. Attached Figure Description

[0016] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention.

[0017] Figure 1: Overall flowchart of the method of the present invention; Figure 2: Overall architecture diagram of the Lite_SIE_CLIPReID model of this invention; Figure 3: Schematic diagram of the lightweight visual / text encoder of the present invention; Figure 4: Schematic diagram of the end-to-end fine-tuning stage; Figure 5 Figure: Experimental results of this invention on Market1501, MSMT17, DukeMTMC-reID, and Occluded-Duke datasets; Figure 6 The ablation experiment of this invention. Detailed Implementation

[0018] To make the above and other objects, features and advantages of the present invention more apparent, specific embodiments of the present invention are described below in detail with reference to the accompanying drawings: Figure 1 Figure 2 shows the overall architecture of the Lite_SIE_CLIPReID model, which can be divided into seven steps: (The flowchart of this method is shown in Figure 2.) Step 1: Pedestrian image preprocessing and lightweight visual encoder construction; Normalization and data augmentation are performed on the input pedestrian RGB image. Based on the CLIP-ViT-B / 16 model, the position encoding length is adjusted by bicubic interpolation resampling to match the sequence length after image segmentation. The core innovation of this step is the lightweight modification of the feedforward network FFN: the hidden layer dimension of the MLP is halved from 4D to 2D, where D=768. By cropping and scaling weights to maintain variance consistency, a lightweight image / text encoder Lite_Encoder is constructed to reduce the number of model parameters and inference latency. Specifically, this step is described in detail below: a1. Pedestrian image preprocessing: First, perform normalization and data augmentation operations on the input raw pedestrian RGB image: first, normalize the pixel values ​​according to the formula... pixel norm = pixel Scaling from / 255 to the [0,1] range eliminates differences in pixel values; then, enhancement operations such as random horizontal flipping, edge padding, random cropping, and random erasing are performed to improve the model's generalization ability. a2. Position-encoded resampling: ViT-B / 16 divides the image into image blocks and encodes the corresponding positions. After dividing the blocks into non-overlapping blocks according to size, the image blocks are obtained. The position codes pre-trained by CLIP are subjected to bicubic interpolation resampling operation to adjust the position code length so that the position code matches the length of the image block sequence. The position code dimension remains unchanged after resampling. a3. Lightweight modification of feedforward network: CLIP uses ViT-B / 16 as its backbone, and its final output is the superposition of residual connections and attention outputs. However, the residual connections of ViT-B / 16 suffer from the problem of feature extrema concentrated in a few channels, causing most feature vectors to point in similar directions in the latent space. This makes it difficult to effectively distinguish features at different spatial locations using cosine similarity, thus introducing noise interference and reducing feature discriminative power. To overcome these shortcomings, a lightweight modification is made to the feedforward network FFN of ViT-B / 16. All pre-trained parameters of the residual connections, layer normalization module, and multi-head attention module are retained, and the hidden layer dimension of the feedforward network FFN is halved from the original 4D to 2D. a4. Encoder initialization: The modified feedforward network FFN is integrated with the resampled positional encoder and other retained ViT-B / 16 modules to construct a lightweight visual encoder. When loading the CLIP pre-trained weights, the weights of FFN are cropped and scaled to complete the encoder initialization. The pre-processed pedestrian RGB image is then output and fed into the subsequent dual-type visual feature extraction network for further pedestrian re-identification. Step 2: Extraction of dual-type visual features from fused side information; An embedding network is constructed to generate a bias vector of camera viewpoint side information. The precise side information embedding only applies to the [cls] token and does not interfere with the image patch features. The image is divided into blocks and converted into feature vectors. The [cls] token containing side information is concatenated to form an input sequence. After forward propagation by the encoder, the global semantic feature F_cls corresponding to the [cls] token and the local detail feature F_visual obtained by the mean pooling of the image patches are extracted to achieve dual-type visual feature extraction. Specifically, this step is described in detail below: b1. Generation of side information bias vector: A single-hidden-layer embedding network is constructed. The weight matrix and bias vector of the embedding network are initialized using a normal distribution. The camera ID is converted to one-hot encoding and then input into the embedding network. After linear transformation, the camera viewpoint information bias vector is output. SIE cam The parameters embedded in the network are trainable parameters; b2. Precise side information embedding: Side information bias vector SIE cam In the form of element-wise addition, only the original [cls] token is embedded into the lightweight visual encoder. cls ] base In the process, the [cls] token with fused side information is obtained; only the [cls] token is used for side information embedding, without involving any image block tokens, to avoid side information interfering with image detail features; b3. Image patch sequence generation: The preprocessed pedestrian RGB image is divided into non-overlapping blocks of a certain size, along the height and width directions; a linear projection operation is performed on each image block to convert it into a one-dimensional feature vector, resulting in an image block sequence. b4. ViT standard input sequence construction: Add a [cls] token containing fusion side information at the beginning of the image patch sequence. cls ] SIE This forms the ViT standard input sequence S. input The first line is [ cls ] SIE Rows 2-129 are image patch feature sequences; the resampled position codes are then compared with the input sequence S. input Element-wise addition is performed to superimpose spatial location information, resulting in the final encoder input X. in ; b5. Dual-type visual feature extraction: The final input will be X. in Input the lightweight visual encoder shown in Figure 3, perform forward propagation processing, and obtain the encoder output X. out X output from encoder out Extract two types of visual features: (a) Visual-like feature F_cls: The first row of the encoder output sequence contains [...]. cls ] SIE Features are obtained by performing a linear transformation on the projection layer at the top of the encoder, capturing the global semantic information of pedestrians; (b) Main visual feature F_visual: Perform global mean pooling operation on the 128 image block token features in rows 2 to 129 of the encoder output sequence, aggregate local detail information of pedestrians, and enhance robustness to occluded scenes; The output visual features F_cls and the main visual features F_visual flow to the dual-branch visual feature lightweight processing stage. As the core features of the visual flow, they will undergo customized lightweight processing separately and will not directly participate in the loss calculation. Step 3: Input ID tags and generate identity-specific learnable text prompt sequences; Input pedestrian identity ID labels as supervised training labels for pedestrian re-identification tasks; embed the pre-trained words of the pre-trained CLIP model into the matrix; convert static pedestrian identity ID labels into dynamic, identity-specific learnable text prompt sequences to bridge the semantic differences between the general CLIP model and fine-grained pedestrian re-identification tasks. Specifically, this step is described in detail below: c1. Learnable context vector initialization: For each pedestrian identity category in the training set, construct a learnable context vector. E learn The learnable context vectors of all identity categories are integrated into a learnable matrix, initialized with a normal distribution, and are trainable parameters; c2. Fixed text word embedding generation: Select text with a fixed prefix and text with a fixed suffix, and segment both texts according to CLIP's text segmentation rules to obtain prefix and suffix sequences. Then, using the CLIP pre-trained word embedding matrix, convert the token sequences of the two texts into fixed word embeddings. E prefix and E suffix ; c3. Concatenation of identity-specific hint sequences: Based on the pedestrian's identity ID label, the learnable context vector of the corresponding category is selected from the learnable matrix. E learn ;Will E prefix , E learn , E suffix By concatenating the tokens sequentially, a learnable text prompt sequence P, unique to the identity, is generated. The text prompt sequence P flows to frozen text feature extraction, serving as the core input of the text stream for extracting cross-modal text features; Step 4: Frozen text feature extraction and cross-modal dimension alignment; Load the pre-trained weights of CLIP text encoder and fix the parameters. Extract features from the input text prompt sequence. After linear transformation and L2 normalization, obtain cross-modal text features that are aligned with the visual feature dimensions, while fully preserving the pre-trained knowledge of CLIP text encoder. The input identity-specific learnable text prompt sequence P and the complete pre-trained weights of the pre-trained CLIP text encoder, including a multi-head attention module, a layer normalization module, a feedforward network, and a text projection layer; Specifically, this step is described in detail below: d1. CLIP text encoder frozen: Load all the pre-trained weights of the CLIP text encoder, set all the encoder parameters to be untrainable, and keep the encoder parameters unchanged during subsequent model training, thus fully preserving its pre-trained cross-modal semantic knowledge; d2. Forward extraction of text features: Input the text prompt sequence P into the frozen CLIP text encoder and perform forward propagation: add the pre-trained positional encoding of the CLIP text to P element by element; d3. Cross-modal dimension alignment: Take the last line of the CLIP text encoder output sequence TextOutput, corresponding to <sep>The token's features are input into the CLIP pre-trained text projection layer, where a linear transformation is performed, followed by L2 normalization, ultimately yielding text features T that are perfectly aligned with the visual feature dimensions of the image stream. Text features T flow to multi-loss joint optimization training, serving as the computational input for cross-modal alignment loss, and participate in model optimization in conjunction with visual features; Step 5: Lightweight and customized processing of dual-branch visual features; To address the different attributes of F_cls and F_visual, a dual-branch lightweight customized processing path is designed: F_cls is dimensionality-reduced and lightweighted through a projection bottleneck layer, while F_visual undergoes a dimension-preserving nonlinear transformation through the projection bottleneck layer. The parameters of the two branches are independent of each other and are both paired with one-dimensional batch normalization BatchNorm1d to obtain F_cls_norm and F_visual_norm respectively, thereby improving the model's inference efficiency and enhancing the discriminative power of features. Specifically, this step is described in detail below: e1. Class feature branch processing: F_cls is dimensionality reduced and lightweighted by passing it through the projection bottleneck layer and the BatchNorm1d layer in sequence, and then the normalized visual feature F_cls_norm is obtained. e2. Main visual feature branch processing: F_visual is subjected to a nonlinear transformation process that preserves its dimension, passing through a projection bottleneck layer and a one-dimensional batch normalization BatchNorm1d layer in sequence, and the normalized main visual feature F_visual_norm is obtained after processing. The standardized class visual features F_cls_norm and standardized main visual features F_visual_norm are output and flow together to the multi-loss joint optimization training stage, serving as the input for calculating the identity classification loss and triplet loss; at the same time, they flow to the re-identification inference output stage, serving as the basis for the final re-identification features; Step Six: End-to-end fine-tuning and training of the multi-loss joint optimization model. Figure 4 is a schematic diagram of the end-to-end lightweight fine-tuning stage of this method. Through joint optimization of multiple losses such as identity classification loss, triplet loss and cross-modal alignment loss, the trainable parameters of the model are fine-tuned end-to-end, so as to achieve a synergistic improvement in visual feature discriminative power and cross-modal feature consistency. Specifically, this step is described in detail below: f1. Construction of the projection classifier: Two independent projection classifiers are constructed, adapted to F_cls_norm and F_visual_norm respectively. Each classifier consists of a single linear transformation layer, and the output dimension is the total number of pedestrian identity categories C in the training set. The parameters of both classifiers are trainable parameters, and the initial values ​​are initialized using a normal distribution. f2. Calculation of identity classification loss: Input F_cls_norm into classifier 1, denoted as Cla1, to obtain the identity prediction score Logits1; input F_visual_norm into classifier 2, denoted as Cla2, to obtain the identity prediction score Logits2; use the cross-entropy loss function to calculate the identity classification losses Lid1 and Lid2 of the two branches respectively, and then take the average to obtain the total identity classification loss L_id, thereby enhancing the class discrimination power of visual features and realizing intra-class feature aggregation; f3. Triplet loss calculation: In the batch samples, one positive sample and four negative samples with the same identity are selected for each anchor sample to construct a triplet sample set; the triplet loss L is calculated based on F_cls_norm and F_visual_norm respectively. tri1 and L tri2 Then, the average value is taken to obtain the total triplet loss L_tri, which increases the inter-class feature distance and achieves the dual goals of intra-class feature aggregation and inter-class feature separation. f4. Calculation of cross-modal alignment loss: First, perform L2 normalization on F_visual_norm, and then calculate the cosine similarity loss with the normalized text feature T to obtain the cross-modal alignment loss L_i2t, which improves the cross-modal alignment accuracy between visual features and text features and makes full use of CLIP's cross-modal semantic knowledge. f5. Construction of the total loss function: The total loss function is constructed based on a weight configuration that prioritizes visual discrimination and secondarily considers cross-modal alignment: L total = λ1 L id +λ2L tri +λ3L i2t λ1 is the identity classification loss weight, λ2 is the triplet loss weight, and λ3 is the cross-modal alignment loss weight; ensuring the model focuses on pedestrian identity discrimination while fully utilizing CLIP's cross-modal semantic knowledge to improve feature generalization ability; all loss calculations are based on batch samples. f6. End-to-end model fine-tuning: The SGD stochastic gradient descent optimizer is used, with a total loss function L total To optimize the objective, the trainable parameters of the model are fine-tuned end-to-end. Optimizer parameter configuration: batch size 64, momentum coefficient 0.9, weight decay rate 1e−4 (only applies to the weight matrix), bias term learning rate is twice the base learning rate (base learning rate 0.008, bias term learning rate 0.016); the learning rate adopts a cosine annealing decay strategy, smoothly decaying from 0.008 to 0 over 360 epochs; the maximum gradient clipping norm is set to 5.0 to prevent gradient explosion.

[0019] f7. Parameter Update: Backpropagation computes the gradients of all trainable parameters, and the SGD optimizer performs parameter updates. The parameters are updated once after each batch of training is completed. After each epoch of training, the model's mAP and Rank-1 metrics are evaluated on the validation set, and the parameter set of the model with the best performance on the validation set is saved for subsequent inference. Fine-tuning is terminated when there is no improvement in validation set performance for 10 consecutive epochs, or when 360 epochs of training are completed.

[0020] The optimized model parameter set is output and then fed back to the lightweight visual encoder, learnable context vectors, and projection bottleneck layer to complete the training loop of the entire model. Step 7: Cross-camera pedestrian re-identification reasoning and result output; Re-identify features, perform similarity retrieval, and output accurate results.

[0021] Input standardized class visual features F_cls_norm and standardized main visual features F_visual_norm; Gallery set image feature library, which is a set of standardized features obtained from all pedestrian images to be retrieved and stored in the database, supports fast similarity retrieval; Inference configuration is based on the actual deployed hardware, accuracy requirements and scene characteristics, such as regular monitoring and heavily occluded scenes.

[0022] Specifically, this step is described in detail below: g1. Selection of final re-identification features: High-precision inference feature fusion involves concatenating features according to their dimensions to obtain fused features. F concat =F_cls_norm⊕F_visual_norm, which combines global semantic information with local detail information.

[0023] g2. Feature normalization: L2 normalization is performed on the selected final re-identification features. After normalization, the L2 norm of the features is 1, eliminating the influence of feature magnitude on similarity calculation and ensuring the accuracy of cosine similarity calculation; g3. Cross-camera similarity retrieval: Normalized query features F norm Input the pedestrian re-identification retrieval system and calculate its cosine similarity with all features in the gallery feature library; g4. Sorting and outputting search results: The gallery samples are sorted from highest to lowest based on their cosine similarity to the query features. A higher similarity score indicates a better match between the sample and the queried pedestrian. The final output of the retrieval system includes: the image path or unique identifier of the matched sample, the matching ranking, the similarity score, and the pedestrian identity label; it also outputs cross-camera pedestrian re-identification retrieval results, a structured matching list sorted by similarity, for use by practical business systems such as security monitoring and traffic management.

[0024] The invention will be further described below with reference to simulation examples.

[0025] Simulation example: Our model is implemented using the PyTorch framework and trained on an NVIDIA TITAN V GPU. During training, we employed data augmentation strategies including random horizontal flipping, edge padding, random cropping, and random erasure. The input pedestrian RGB image has a resolution of 256×128, with pixels normalized to [0,1].

[0026] Based on CLIP-ViT-B / 16, the lightweight visual encoder has an FFN hidden layer dimension of 1536 and a positional encoding length of 129; the learnable text cue sequence dimension is 13×512; the batch size is 64, and the total number of training epochs is 360; the optimizer uses SGD with a momentum coefficient of 0.9, a weight decay rate of 1e-4, a base learning rate of 0.008, and a bias term learning rate of 0.016.

[0027] To verify the effectiveness of this invention, we selected four representative public person re-identification datasets for experimental validation, including Market1501, MSMT17, DukeMTMC-reID, and the Occluded-Duke dataset specifically designed for occluded scenes. The Occluded-Duke dataset contains 9%, 100%, and 10% occluded images in its training, query, and gallery sets, respectively, to verify the model's occlusion robustness. Using mean accuracy (mAP), Rank-1, Rank-5, and Rank-10 as evaluation metrics, we compared the model with existing state-of-the-art CLIP-based person re-identification models (such as CLIPReID, AlphaCLIP, and VIPReID). The results are as follows: Figure 5 Experimental results are shown on the Market1501, MSMT17, DukeMTMC-reID, and Occluded-Duke datasets. The experimental results demonstrate the superiority and innovation of the method presented in this invention. 1. Excellent performance on standard datasets: On the Market1501 dataset, the mAP reaches 90.2% and the Rank-1 reaches 95.7%, which is an improvement of 0.6% in mAP and 0.7% in Rank-1 compared to the baseline CLIPReID model; On the MSMT17 dataset, the mAP reaches 74.8% and the Rank-1 reaches 89.2%, which is an improvement of 1.5% and 0.5% respectively compared to the baseline model, demonstrating the model's good adaptability to multiple cameras and large scenes.

[0028] 2. Significantly improved robustness in occluded scenarios: On the occlusion robustness dataset Occluded-Duke, the mAP reached 60.4% and the Rank-1 reached 68.3%. Compared with the baseline CLIPReID model, the mAP improved by 0.9% and the Rank-1 improved by 1.2%, which verifies the effectiveness of side information embedding and dual-type visual feature extraction in occluded scenarios.

[0029] 3. Lightweight design without performance loss: This invention halves the number of FFN parameters in the CLIP visual encoder and reduces the model computation by about 40%, but the performance on all datasets does not degrade. On the contrary, it achieves varying degrees of improvement, proving the effectiveness of the lightweight approach of "simplification rather than removal based on entropy analysis".

[0030] 4. Cross-modal alignment enhances generalization ability: Compared with the model without cross-modal alignment loss, the model of this invention performs better in cross-dataset generalization experiments, proving that cross-modal feature alignment can effectively utilize CLIP's pre-trained semantic knowledge and improve the model's generalization ability.

[0031] To validate the proposed module, we conducted several independent ablation experiments to verify the importance of the experimental parameters and components.

[0032] All ablation experiments were performed on the Market1501, MSMT17, DukeMTMC-reID, and Occluded-Duke datasets. In each independent ablation experiment, only one parameter was changed, while all other parameters remained the same, to test the impact of a single parameter or component on the final result. Figure 6 The ablation experiment results of this invention are shown.

[0033] Starting with the baseline CLIPReID model, the proposed modules were gradually added, and their performance was evaluated using mAP and Rank-1 as metrics, with higher values ​​being better. The overall results clearly demonstrate that the proposed components made a positive and quantifiable contribution to the model's performance. With the gradual integration of each module, the model accuracy continuously improved, and the final complete model achieved the highest accuracy on both metrics. This fully demonstrates the effectiveness and advanced design of the overall architecture.

[0034] In summary, the Lite_SIE_CLIPReID method proposed in this invention effectively solves the key technical bottlenecks in migrating the general CLIP model to person re-identification through lightweight modification, side information embedding, learnable hints, two-stage training, and joint optimization of multiple losses. While improving the robustness and feature discrimination power of the model in complex scenarios, it also achieves model lightweighting, providing a solution that balances accuracy and efficiency for the practical deployment of person re-identification.< / sep> < / sep>

Claims

1. A lightweight cue-based pedestrian re-identification method that integrates side information embedding, comprising the following steps: Step 1: Pedestrian image preprocessing and lightweight visual encoder construction; Normalization and data augmentation are performed on the input pedestrian RGB image; based on CLIP-ViT-B / 16 (abbreviated as ViT-B / 16), the position encoding length is adjusted by bicubic interpolation resampling. The core innovation of this step is the lightweight transformation of the feedforward network FFN: the dimension of the MLP hidden layer is halved from 4D to 2D, the weights are cropped and scaled to maintain variance consistency, and a lightweight image or text encoder Lite_Encoder is constructed to reduce the number of model parameters and inference latency. Specifically, this step is described in detail below: a1. Pedestrian image preprocessing: First, perform normalization and data augmentation operations on the input raw pedestrian RGB image: first, normalize the pixel values ​​according to the formula... pixel norm =pixel / 255 Scaled to the [0,1] range to eliminate differences in pixel value magnitudes, where, pixel These are the pixel values ​​of the input image; Then, enhancement operations such as random horizontal flipping, edge padding, random cropping, and random erasure are performed to improve the model's generalization ability. a2. Position-encoded resampling: ViT-B / 16 divides the image into image blocks and encodes the corresponding positions. After dividing the blocks into non-overlapping blocks according to size, the image blocks are obtained. The position codes pre-trained by CLIP are subjected to bicubic interpolation resampling operation to adjust the position code length so that the position code matches the length of the image block sequence. The position code dimension remains unchanged after resampling. a3. Lightweight modification of feedforward network: The residual connections of the ViT-B / 16 network have feature extrema concentrated in a few channels, causing most feature vectors to point in similar directions in the latent space, introducing feature noise and reducing feature discriminative power. To overcome the above defects, the feedforward network FFN of ViT-B / 16 is modified to be lightweight. All pre-trained parameters of the residual connections, layer normalization module and multi-head attention module are retained, and only the feedforward network FFN is modified to reduce its hidden layer dimension from the original 4D to 2D. a4. Encoder initialization: The modified feedforward network FFN is integrated with the resampled positional encoder and other retained ViT-B / 16 modules to construct a lightweight visual encoder. When loading the CLIP pre-trained weights, the weights of FFN are cropped and scaled to complete the encoder initialization. The pre-processed pedestrian RGB image is then output and fed into the subsequent dual-type visual feature extraction network for further pedestrian re-identification. Step 2: Extraction of dual-type visual features from fused side information; An embedding network is constructed to generate a camera viewpoint side information bias vector. The precise side information embedding only applies to the [cls] token and does not interfere with image patch features. The image is segmented and converted into feature vectors. The [cls] token containing side information is concatenated to form an input sequence. After forward propagation by the encoder, the global semantic feature F_cls corresponding to the [cls] token and the local detail feature F_visual obtained by average pooling of the image blocks are extracted to achieve dual-type visual feature extraction. Specifically, this step is described in detail below: b1. Generation of side information bias vector: A single-hidden-layer embedding network is constructed. The weight matrix and bias vector of the embedding network are initialized using a normal distribution. The camera ID is converted to one-hot encoding and then input into the embedding network. After linear transformation, the camera viewpoint bias vector is output. SIE cam The parameters embedded in the network are trainable parameters; b2. Precise side information embedding: Side information bias vector SIE cam In the form of element-wise addition, only the original [cls] token is embedded into the lightweight visual encoder. cls ] base In the process, the [cls] token with fused side information is obtained; only the [cls] token is used for side information embedding, without involving any image block tokens, to avoid side information interfering with image detail features; b3. Image patch sequence generation: The preprocessed pedestrian RGB image is divided into non-overlapping blocks of a certain size, along the height and width directions; a linear projection operation is performed on each image block to convert it into a one-dimensional feature vector, resulting in an image block sequence. b4. ViT standard input sequence construction: Add a [cls] token containing fusion side information at the beginning of the image patch sequence. cls ] SIE ViT standard input sequence S input The first line is [ cls ] SIE Rows 2 through 129 are image patch feature sequences; The resampled position code is combined with the input sequence S input Element-wise addition is performed to superimpose spatial location information, resulting in the final encoder input X. in ; b5. Dual-type visual feature extraction: The final input will be X. in Input a lightweight visual encoder, perform forward propagation processing, and obtain the encoder output X. out X output from encoder out Extract two types of visual features: (a) Visual-like feature F_cls: The first row of the encoder output sequence contains [...]. cls ] SIE Features are obtained by performing a linear transformation on the projection layer at the top of the encoder, capturing the global semantic information of pedestrians; (b) Main visual feature F_visual: Perform global mean pooling operation on the 128 image block token features in rows 2 to 129 of the encoder output sequence, aggregate local detail information of pedestrians, and enhance robustness to occluded scenes; The output visual features F_cls and the main visual features F_visual flow to the dual-branch visual feature lightweight processing stage. As the core features of the visual flow, they will undergo customized lightweight processing separately and will not directly participate in the loss calculation. Step 3: Input ID tags and generate identity-specific learnable text prompt sequences; Input pedestrian identity ID labels as supervised training labels for pedestrian re-identification tasks; embed the pre-trained words of the pre-trained CLIP model into the matrix; convert static pedestrian identity ID labels into dynamic, identity-specific learnable text prompt sequences to bridge the semantic differences between the general CLIP model and fine-grained pedestrian re-identification tasks. Specifically, this step is described in detail below: c1. Learnable context vector initialization: For each pedestrian identity category in the training set, construct a learnable context vector. E learn The learnable context vectors of all identity categories are integrated into a learnable matrix, initialized with a normal distribution, and are trainable parameters; c2. Fixed text word embedding generation: Select text with a fixed prefix and text with a fixed suffix, and segment both texts according to CLIP's text segmentation rules to obtain prefix and suffix sequences. Then, using the CLIP pre-trained word embedding matrix, convert the token sequences of the two texts into fixed word embeddings E. prefix and E suffix ; c3. Concatenation of identity-specific hint sequences: Based on the pedestrian's identity ID label, the learnable context vector of the corresponding category is selected from the learnable matrix. E learn ;Will E prefix , E learn , E suffix By concatenating the tokens sequentially, a learnable text prompt sequence P, unique to the identity, is generated. The text prompt sequence P flows to frozen text feature extraction, serving as the core input of the text stream for extracting cross-modal text features; Step 4: Frozen text feature extraction and cross-modal dimension alignment; Load the pre-trained weights of CLIP text encoder and fix the parameters. Extract features from the input text prompt sequence. After linear transformation and L2 normalization, obtain cross-modal text features that are aligned with the visual feature dimensions, while fully preserving the pre-trained knowledge of CLIP text encoder. The input identity-specific learnable text prompt sequence P and the complete pre-trained weights of the pre-trained CLIP text encoder, including a multi-head attention module, a layer normalization module, a feedforward network, and a text projection layer; Specifically, this step is described in detail below: d1. CLIP text encoder frozen: Load all the pre-trained weights of the CLIP text encoder, set all the encoder parameters to be untrainable, and keep the encoder parameters unchanged during subsequent model training, thus fully preserving its pre-trained cross-modal semantic knowledge; d2. Forward extraction of text features: Input the text prompt sequence P into the frozen CLIP text encoder, perform forward propagation processing, and add the CLIP text pre-trained positional encoding to P element by element; d3. Cross-modal dimension alignment: Take the last line of the CLIP text encoder output sequence TextOutput, corresponding to <sep> The token's features are input into the CLIP pre-trained text projection layer, where a linear transformation is performed, followed by L2 normalization, ultimately yielding text features T that are perfectly aligned with the visual feature dimensions of the image stream.< / sep> Text features T flow to multi-loss joint optimization training, serving as the computational input for cross-modal alignment loss, and participate in model optimization in conjunction with visual features; Step 5: Lightweight and customized processing of dual-branch visual features; To address the different attributes of F_cls and F_visual, a dual-branch lightweight customized processing path is designed: F_cls is dimensionality-reduced and lightweighted through a projection bottleneck layer, while F_visual undergoes a dimension-preserving nonlinear transformation through the projection bottleneck layer. The parameters of the two branches are independent of each other and are both paired with one-dimensional batch normalization BatchNorm1d to obtain F_cls_norm and F_visual_norm respectively, thereby improving the model's inference efficiency and enhancing the discriminative power of features. Specifically, this step is described in detail below: e1. Class feature branch processing: F_cls is dimensionality reduced and lightweighted by passing it through the projection bottleneck layer and the BatchNorm1d layer in sequence, and then the normalized visual feature F_cls_norm is obtained. e2. Main visual feature branch processing: F_visual is subjected to a nonlinear transformation process that preserves its dimension, passing through a projection bottleneck layer and a one-dimensional batch normalization BatchNorm1d layer in sequence, and the normalized main visual feature F_visual_norm is obtained after processing. The standardized class visual features F_cls_norm and standardized main visual features F_visual_norm are output and flow together to the multi-loss joint optimization training stage, serving as the input for calculating the identity classification loss and triplet loss; at the same time, they flow to the re-identification inference output stage, serving as the basis for the final re-identification features; Step Six: End-to-end fine-tuning and training of the model with multi-loss joint optimization; By jointly optimizing multiple losses such as identity classification loss, triplet loss, and cross-modal alignment loss, the trainable parameters of the model are fine-tuned end-to-end, achieving a synergistic improvement in visual feature discriminative power and cross-modal feature consistency. Specifically, this step is described in detail below: f1. Construction of the projection classifier: Two independent projection classifiers are constructed, adapted to F_cls_norm and F_visual_norm respectively. Each classifier consists of a single linear transformation layer, and the output dimension is the total number of pedestrian identity categories C in the training set. The parameters of both classifiers are trainable parameters, and the initial values ​​are initialized using a normal distribution. f2. Calculation of identity classification loss: Input F_cls_norm into classifier 1, denoted as Cla1, to obtain the identity prediction score Logits1; input F_visual_norm into classifier 2, denoted as Cla2, to obtain the identity prediction score Logits2; use the cross-entropy loss function to calculate the identity classification losses Lid1 and Lid2 of the two branches respectively, and then take the average to obtain the total identity classification loss L_id, thereby enhancing the class discrimination power of visual features and realizing intra-class feature aggregation; f3. Triplet loss calculation: In the batch samples, one positive sample and four negative samples with the same identity are selected for each anchor sample to construct a triplet sample set; the triplet loss L is calculated based on F_cls_norm and F_visual_norm respectively. tri1 and L tri2 Then, the average value is taken to obtain the total triplet loss L_tri, which increases the inter-class feature distance and achieves the dual goals of intra-class feature aggregation and inter-class feature separation. f4. Calculation of cross-modal alignment loss: First, perform L2 normalization on F_visual_norm, and then calculate the cosine similarity loss with the normalized text feature T to obtain the cross-modal alignment loss L_i2t, which improves the cross-modal alignment accuracy between visual features and text features and makes full use of CLIP's cross-modal semantic knowledge. f5. Construction of the total loss function: The total loss function is constructed based on a weight configuration that prioritizes visual discrimination and secondarily considers cross-modal alignment: L total = λ1 L id +λ2L tri +λ3L i2t λ1 is the identity classification loss weight, λ2 is the triplet loss weight, and λ3 is the cross-modal alignment loss weight; ensuring the model focuses on pedestrian identity discrimination while fully utilizing CLIP's cross-modal semantic knowledge to improve feature generalization ability; all loss calculations are based on batch samples. f6. End-to-end model fine-tuning: The SGD stochastic gradient descent optimizer is used, with a total loss function L total To optimize the objective, the trainable parameters of the model are fine-tuned end-to-end; f7. Parameter Update: Backpropagation computes the gradients of all trainable parameters, and the SGD optimizer performs parameter updates. The parameters are updated once after each batch of training is completed. After each epoch of training, the model's mAP and Rank-1 metrics are evaluated on the validation set, and the parameter set of the model with the best performance on the validation set is saved for subsequent inference. The optimized model parameter set is output and then fed back to the lightweight visual encoder, learnable context vectors, and projection bottleneck layer to complete the training loop of the entire model. Step 7: Cross-camera pedestrian re-identification reasoning and result output; Re-identify features, perform similarity retrieval, and output accurate results; Input the standardized class visual feature F_cls_norm and the standardized main visual feature F_visual_norm; the gallery set image feature library is a set of standardized features obtained from all pedestrian images to be retrieved and stored in the database to support fast similarity retrieval; Specifically, this step is described in detail below: g1. Selection of final re-identification features: High-precision inference feature fusion involves concatenating features according to their dimensions to obtain fused features. F concat =F_cls_norm⊕F_visual_norm, which merges global semantic information with local detail information; g2. Feature normalization: The selected final re-identification features are subjected to L2 normalization. After normalization, the L2 norm of the features is 1, which eliminates the influence of feature magnitude on similarity calculation and ensures the accuracy of cosine similarity calculation. g3. Cross-camera similarity retrieval: Normalized query features F norm Input the pedestrian re-identification retrieval system and calculate its cosine similarity with all features in the gallery feature library; g4. Sorting and outputting search results: The gallery samples are sorted from high to low based on their cosine similarity to the query features. The higher the similarity score, the better the match between the sample and the identity of the query pedestrian. The final output of the retrieval system includes: the image path or unique identifier of the matched sample, the matching ranking, the similarity score, and the pedestrian identity label. Output cross-camera pedestrian re-identification retrieval results, a structured matching list sorted by similarity, for use by actual business systems such as security monitoring and traffic management.

Images