A pedestrian re-identification system, method, storage medium and electronic device

Abstract

The application discloses a pedestrian re-identification system and method, a storage medium and an electronic device, and constructs a unified multi-task learning framework: on the one hand, a sketch information compensation module is designed to explicitly supplement the missing color and texture information of the sketch by using text semantics at the structure level, so that the complementary features between modes are promoted; on the other hand, a personalized supervised learning strategy and an adaptive loss adjustment mechanism are introduced to perform differentiated supervision and dynamic balance for different task characteristics at the optimization level, so that the multi-task collaborative convergence is coordinated; finally, a multi-modal pedestrian re-identification system with higher retrieval accuracy, stronger generalization ability and better suitability for actual complex scenes is realized, and three technical problems existing in the prior art pedestrian re-identification method based on the description type are overcome: the modal isolation training leads to insufficient generalization ability, the unified optimization strategy ignores the essential difference between the text and the sketch task, and an effective cross-modal information complementary mechanism is lacked.

Description

Technical Field

[0001] This invention relates to the field of computer vision technology, and specifically to a pedestrian re-identification system, method, storage medium, and electronic device. Background Technology

[0002] Pedestrian re-identification technology aims to retrieve specific pedestrian targets from images or videos across multiple cameras using computer vision methods, and has significant application value in the field of intelligent security and surveillance. With technological advancements, its research paradigm has expanded from traditional single-modal (e.g., image retrieval) to cross-modal retrieval, allowing queries for corresponding RGB images using non-image modalities (e.g., text descriptions, hand-drawn sketches, infrared images, etc.). Especially in practical scenarios such as criminal investigation, when the target only possesses verbal descriptions or simple hand-drawn features, description-based pedestrian re-identification (including text-to-image and sketch-to-image) demonstrates outstanding practical significance.

[0003] Currently, description-based person re-identification research mainly focuses on two query modalities: text and sketch. Text descriptions are easy to obtain but have coarse granularity, making it difficult to accurately depict visual details; sketches, on the other hand, retain more structural information but lack information on color and texture. Existing technical solutions often model and train independently for a single modality, such as building separate text-image or sketch-image retrieval models. However, this isolated approach makes it difficult for the model to generalize to unseen query modalities, limiting its practical deployment capability in multimodal flexible query scenarios.

[0004] The closest existing background technologies fall into two main categories. One category involves multimodal person re-identification frameworks that simultaneously fuse text and sketches as query conditions. This approach improves retrieval performance by jointly using two modalities for querying. However, during multimodal feature learning, it still uses independent pre-trained parameters for text and visual representations, leading to insufficient alignment between modal representations, limited generalization ability, and ultimately failing to achieve optimal recognition accuracy. The other category proposes a modality-independent unified architecture capable of simultaneously training on three retrieval tasks: text-photo, sketch-photo, and text + sketch-photo. This framework learns shared representations across modalities through a parameter-sharing mechanism to address the uncertainty of query modalities. However, this method treats different tasks as homogeneous problems, employing a uniform supervised learning strategy and failing to fully consider the fundamental differences between text and visual tasks in semantic alignment and feature learning mechanisms. Furthermore, it fails to deeply explore complementary information between modalities, resulting in weak discriminative power of the multimodal fusion features and limited information coupling effects.

[0005] In summary, existing background technologies mainly suffer from the following shortcomings: 1) The use of independent pre-training parameters leads to weak model generalization ability, making it difficult to adapt to the actual needs of multimodal mixed queries; 2) The unified optimization strategy ignores the inherent differences and characteristics of different retrieval tasks (such as text-based semantic matching and sketch-based structural matching), limiting the upper limit of model performance on each task; 3) There is a lack of effective mechanisms to explicitly utilize information from one modality (such as color descriptions in text) to compensate for the inherent deficiencies of another modality (such as the lack of color information in sketches), thereby restricting the richness and discriminative power of cross-modal representations.

[0006] Therefore, it is necessary to provide new pedestrian re-identification schemes to overcome the three major limitations of the existing technologies. Summary of the Invention

[0007] The purpose of this invention is to provide a pedestrian re-identification system, method, storage medium, and electronic device.

[0008] To achieve the above objectives, in one aspect, the present invention proposes a pedestrian re-identification system, comprising:

[0009] The encoder includes a visual encoder and a text encoder, which are used to process the input image and text respectively, and output visual features and text features respectively after processing. The image includes a photograph and a sketch, and the visual features include photograph features and first sketch features.

[0010] The sketch information compensation module, connected to the encoder, is used to mine semantically related text-guided visual features from the photo features, and to fuse the text-guided visual features into the first sketch features, and output the compensated, information-enhanced second sketch representation.

[0011] An adaptive confidence loss adjustment module, connected to the sketch information compensation module, is used to supervise the text retrieval task and the visual retrieval task respectively using different loss functions, and to dynamically adjust the loss weight of each task during the supervision process using an adaptive loss weight adjustment mechanism; the text retrieval task is a text-to-photo retrieval task, and the visual retrieval task includes a sketch-to-photo sketch retrieval task and a fusion retrieval task of text and sketch fusion features to photo fusion retrieval task.

[0012] In a preferred embodiment, the encoder uses a Transformer-based model, the visual encoder uses a Vision Transformer model, and the text encoder uses a standard Transformer model.

[0013] In a preferred embodiment, the sketch information compensation module includes a text-guided visual miner and an information compensation submodule. The visual miner is connected to the visual encoder and the text encoder, and is used to receive the photo features from the visual encoder and the text features from the text encoder as input to mine the text-guided visual features. The information compensation submodule is connected to the visual encoder and the visual miner, and is used to fuse the text-guided visual features output by the visual miner into the first sketch features from the visual encoder, thereby outputting the second sketch representation.

[0014] In a preferred embodiment, the processing of the visual mining tool includes: mapping the photo features and text features to a common subspace through a learnable first linear transformation layer and a second linear transformation layer, respectively; calculating the similarity between each spatial location of the transformed photo features and each word of the text features to form an association matrix; using the association matrix as attention weights, performing weighted summation on the original photo features to aggregate the semantically related visual region features of the text features, thereby obtaining the text-guided visual features; the compensation process of the information compensation submodule includes: performing adaptive instance normalization on the original sketch features, and generating affine parameters for the text-guided visual features through a multilayer perceptron; using the affine parameters to perform channel-by-channel scaling and biasing operations on the normalized sketch features, and finally outputting the second sketch features.

[0015] In a preferred embodiment, the supervised learning module uses a triplet marginal loss function to supervise the text retrieval task and a contrastive loss function to supervise the visual retrieval task.

[0016] In a preferred embodiment, the adaptive confidence loss adjustment module's weight adjustment process for each task includes: real-time monitoring of the loss value of each task and mapping the loss value of each task to the confidence level of each task; using the confidence level of the text task as an anchor benchmark, calculating its modulation factor for the sketching task and the fusion task respectively, and also calculating the feedback modulation factor of the sketching task and the fusion task for the text task respectively; dynamically adjusting the loss weight of the sketching task using the modulation factor of the sketching task, dynamically adjusting the loss weight of the fusion task using the modulation factor of the fusion task, and dynamically adjusting the loss weight of the text task using the feedback modulation factor of the sketching task and the fusion task for the text task.

[0017] On the other hand, the present invention proposes a pedestrian re-identification method, comprising:

[0018] S1, a visual encoder and a text encoder are used to process the image and text respectively, and the processed visual features and text features are output respectively. The image includes a photograph and a sketch, and the visual features include photograph features and first sketch features.

[0019] S2, extract semantically related text-guided visual features from the photo features, and integrate the text-guided visual features into the first sketch features to output a compensated, information-enhanced second sketch representation;

[0020] S3, different loss functions are used to supervise the text retrieval task and the visual retrieval task respectively, and an adaptive loss weight adjustment mechanism is used to dynamically adjust the loss weight of each task during the supervision process; the text retrieval task is a text-to-photo retrieval task, and the visual retrieval task includes a sketch-to-photo sketch retrieval task and a fusion retrieval task of text and sketch fusion features to photo fusion retrieval task.

[0021] In a preferred embodiment, S2 includes: mapping the photo features and text features to a common subspace through a learnable first linear transformation layer and a second linear transformation layer, respectively; calculating the similarity between each spatial location of the transformed photo features and each word of the text features to form an association matrix; using the association matrix as attention weights to perform weighted summation on the original photo features, thereby aggregating visual region features semantically related to the text features to obtain the text-guided visual features; performing adaptive instance normalization on the original sketch features; and generating affine parameters for the text-guided visual features through a multilayer perceptron, using the affine parameters to normalize... The sketch features are then scaled and biased channel by channel to finally output the second sketch feature; and / or, S3 includes: real-time monitoring of the loss value of each task and mapping the loss value of each task to the confidence level of each task; using the confidence level of the text task as the anchor benchmark, calculating the modulation factor of the text task and the fusion task respectively, and also calculating the feedback modulation factor of the text task and the fusion task to the text task respectively; dynamically adjusting the loss weight of the sketch task using the modulation factor of the sketch task, dynamically adjusting the loss weight of the fusion task using the modulation factor of the fusion task, and dynamically adjusting the loss weight of the text task using the feedback modulation factor of the sketch task and the fusion task to the text task.

[0022] In another aspect, the present invention proposes a readable storage medium storing a computer program, which, when run, performs the steps in the above-described pedestrian re-identification method.

[0023] In another aspect, the present invention proposes an electronic device, including a memory and a processor, wherein the memory stores a computer program, and the computer program, when run by the processor, performs the steps in the above-described pedestrian re-identification method.

[0024] Compared with the prior art, the present invention has the following beneficial effects:

[0025] This invention constructs a unified multi-task learning framework: on the one hand, by designing a sketch information compensation module, it explicitly supplements the missing color and texture information of the sketch at the structural level using text semantics, promoting feature complementarity between modalities; on the other hand, it introduces personalized supervised learning strategies and adaptive loss adjustment mechanisms, performing differentiated supervision and dynamic balancing for different task characteristics at the optimization level, coordinating multi-task collaborative convergence; ultimately, it achieves a multi-modal person re-identification system with higher retrieval accuracy, stronger generalization ability, and greater applicability to complex real-world scenarios. It overcomes three major technical problems of existing description-based person re-identification methods: insufficient generalization ability due to modal isolation training, the uniform optimization strategy ignoring the essential differences between text and sketch tasks, and the lack of an effective cross-modal information complementarity mechanism. Attached Figure Description

[0026] Figure 1 This is a schematic diagram of the model structure of the pedestrian re-identification system of the present invention;

[0027] Figure 2 This is a schematic diagram illustrating the principle of the visual excavator of the present invention;

[0028] Figure 3 A visual comparison diagram of attention feature maps;

[0029] Figures 4a-4c A visual comparison diagram of feature distributions: Figure 4a For the original feature distribution, Figure 4b For the feature distribution of the baseline model, and Figure 4c The feature distribution of TPL-Net proposed in this invention;

[0030] Figure 5 This is a flowchart illustrating the pedestrian re-identification method of the present invention. Detailed Implementation

[0031] The specific embodiments of the present invention will be described in detail below, but it should be understood that the scope of protection of the present invention is not limited to the specific embodiments.

[0032] Unless otherwise expressly stated, throughout the specification and claims, the term "comprising" or its variations such as "including" or "comprising" shall be understood to include the stated elements or components without excluding other elements or other components.

[0033] This invention proposes a Task-Promoted Learning Network (TPL-Net) to address the multi-task person re-identification problem based on descriptive modalities (text, sketch, and their fusion modalities). The overall technical solution of this invention aims to achieve this through a unified framework: on the one hand, by designing a sketch information compensation module, it explicitly supplements the missing color and texture information of the sketch using text semantics at the structural level, promoting feature complementarity between modalities; on the other hand, it introduces personalized supervised learning strategies and adaptive loss adjustment mechanisms, performing differentiated supervision and dynamic balancing for different task characteristics at the optimization level, coordinating and optimizing the collaborative convergence of three retrieval tasks: text-to-image, sketch-to-image, and text + sketch fusion to image. Ultimately, this results in a multi-modal person re-identification system with higher retrieval accuracy, stronger generalization ability, and greater applicability to complex real-world scenarios. The core of this invention lies in designing a learning mechanism that promotes knowledge complementarity and collaborative convergence among multiple tasks, overcoming the three major limitations of existing description-based person re-identification methods: insufficient generalization ability due to modal isolation training, the uniform optimization strategy ignoring the essential differences between text and sketch tasks, and the lack of an effective cross-modal information complementarity mechanism.

[0034] like Figure 1 As shown, the pedestrian re-identification system disclosed in this invention is a Task Facilitating Learning Network (TPL-Net) built on a dual-encoder architecture. Specifically, it includes an encoder, a sketch information compensation module, and an adaptive confidence loss adjustment module. The encoder comprises a visual encoder and a text encoder, which process the input image (including photographs and sketches) and text modalities respectively, outputting visual features and text features respectively. The photograph and sketch modalities share the same parameters of the visual encoder to ensure consistency in the visual feature space. In implementation, the encoder preferably uses a Transformer-based model (a deep learning architecture based on self-attention mechanism). For example, the visual encoder specifically uses the Vision Transformer (ViT, a computer vision model based on the Transformer architecture), while the text encoder uses the standard Transformer model (the original Transformer architecture first proposed by Vaswani et al. in the paper "Attention is All You Need" in 2017). The encoder can be initialized using the parameters of a large-scale vision-language pre-trained model (such as CLIP) to obtain a good cross-modal representation foundation.

[0035] The system of this invention processes input data as follows: for the input image (photo X) p Or sketch X s First, the image is segmented into fixed-size blocks, then linearly projected and positionally encoded before being fed into a visual encoder to output a global visual feature representation (i.e., photographic features). or sketch features For the input text description X T First, word segmentation is performed, then the data is fed into a text encoder to output a global text feature representation (i.e., text features). Subsequently, features from the sketch modality Features of text modality The data is fed into the core sketch information compensation module for enhancement and fusion. Finally, the system utilizes a personalized supervised learning strategy and an adaptive confidence loss adjustment mechanism to jointly optimize and train the three tasks: text retrieval, sketch retrieval, and multimodal fusion retrieval.

[0036] Preferably, the sketch information compensation module is one of the key innovations of this invention. It aims to utilize the rich semantic information of the text modality to compensate for the inherent deficiencies in details such as color and texture in the sketch, thereby generating a more informative and discriminative sketch representation. The sketch information compensation module is connected to the encoder and includes two sub-modules: a text-guided visual miner and an information compensation sub-module. The visual miner is connected to the aforementioned visual encoder and text encoder and is used to receive photo features from the visual encoder. and text features from the text encoder As input, such as Figure 2 As shown, its goal is to uncover visual details that are semantically related to the text description. Here, relevance means that the features in the text description actually exist in the features of the photograph. For example, the text description might state "black top, wearing shorts," and the pedestrian in the photograph might possess these clothing characteristics. In practice, this can be represented by the parameter "similarity"; higher similarity indicates higher relevance.

[0037] The specific processing steps of the visual mining tool include:

[0038] 1) Feature Transformation and Similarity Calculation: First, through two learnable first linear transformation layers... Second linear transformation layer The features of the photos were respectively and text features Mapping to a common subspace; then, calculating the transformed image features. Each spatial location and text feature The similarity between each word forms an association matrix. Among them, the correlation matrix Each element in Indicates the first The image region and the first The degree of correlation between text words is specifically represented as follows:

[0039] .

[0040] In the formula, Indicates the first Feature vectors of image regions Indicates the first Feature vectors of text words This represents the similarity calculation function.

[0041] 2) Attention-weighted aggregation: using the calculated association matrix As attention weights, the original photo features By performing weighted summation, visual region features closely related to the text description are aggregated to obtain the visual representation guided by the text. ,in, A text-guided visual mining tool. This represents a learnable linear transformation layer. This process forces the model to learn the correspondence between textual semantics and local regions of the image.

[0042] The information compensation submodule is connected to the aforementioned visual encoder and visual miner, and is used to guide visual features based on the text output by the visual miner. Fused into the first sketch features from the visual encoder In this way, the second sketch representation is output. ,like Figure 1 As shown. The compensation process specifically includes:

[0043] 1) Sketch Feature Normalization: Normalization of the original sketch features Adaptive Instance Normalization (AdaIN) is performed to normalize its feature distribution. Specifically, it is expressed as follows:

[0044] ;

[0045] in, and Represents a multilayer sensing function. It is a hyperparameter to prevent the denominator from being zero. N stands for normalization operation. This represents the normalized sketch features.

[0046] 2) Affine parameter generation: This involves guiding the visual features derived from the above text. A set of affine parameters (specifically scaling factors) is generated using a multilayer perceptron (MLP). and bias factor These parameters are used as modulation weights for feature calibration.

[0047] 3) Feature Modulation: The generated affine parameters are used to modulate the normalized sketch features. Channel-by-channel scaling and biasing operations are performed to finally output a compensated, information-enhanced second sketch representation. This process integrates detailed information such as color and texture corresponding to the semantic meaning of the text into an abstract, structured sketch representation through feature modulation.

[0048] The adaptive confidence loss adjustment module is connected to the sketch information compensation module mentioned above. It is used to supervise text retrieval and visual retrieval tasks using different loss functions, and to dynamically adjust the loss weights of each task during the supervision process using an adaptive loss weight adjustment mechanism. Preferably, considering the differences in learning characteristics of different query tasks, this invention abandons a uniform loss function and designs customized supervision functions for text retrieval and visual retrieval tasks (including sketch retrieval and fusion retrieval tasks). For text retrieval tasks (i.e., text-to-image retrieval tasks), the core is fine-grained semantic alignment. This invention uses the Triplet Margin Loss (TML) function for supervision. This loss function constructs a (text anchor, corresponding positive image, non-corresponding negative image) triple, and in the feature embedding space, it uses a marginal value... Apply constraints that force matching text-photo pairs to be closer together, and non-matching pairs to be farther apart. Marginal value The preferred range is [0.05, 0.3], and in the best implementation, it is set to 0.1. This loss function based on local comparison can effectively learn subtle differences in semantic attributes. In this embodiment, the triplet marginal loss function for text retrieval tasks is defined as:

[0049] ;

[0050] in, This represents the marginal loss function for a text retrieval task. Represents the distance metric function. Indicates a positive sample pair. This represents a negative sample pair.

[0051] For the aforementioned visual retrieval tasks (including sketch-to-photo sketch retrieval and text + sketch fusion feature-to-photo fusion retrieval), challenges arise from large modal differences and high intra-class variance. This invention employs a contrastive loss function for supervision. Within a batch, this loss function treats matched query-library pairs as positive samples and all other combinations as negative samples. Discriminative features are learned by optimizing the similarity of positive sample pairs to be significantly higher than that of negative sample pairs. The temperature parameter in the contrastive loss function... The preferred range is [0.05, 0.2], and in the best implementation, it is set to 0.07. This loss based on global comparison provides rich discriminative context and enhances the model's robustness to abstract queries.

[0052] For a containing For each batch of samples, assuming we have global embedding features of the query modality (sketching or fused multimodal query). and global embedding features of candidate photo modalities The present invention will These are considered positive sample pairs, while all other combinations within the batch are considered positive. These are considered negative sample pairs. The contrastive loss function for sketch retrieval tasks and text + sketch fusion tasks is defined as follows:

[0053] ;

[0054] In the formula, This represents the contrastive loss function for the sketch retrieval task. This represents the contrastive loss function for the fusion task, where M is the number of samples. This represents the similarity calculation function. For positive sample pairs, For negative sample pairs, This represents hyperparameters.

[0055] Preferably, to dynamically balance the inconsistency in convergence speed and difficulty among tasks in multi-task learning, this invention proposes a hierarchical adaptive loss weight adjustment mechanism. The adaptive confidence loss adjustment module of this invention specifically includes the following weight adjustment process for each task:

[0056] 1) Calculation of task confidence

[0057] During training, the loss value for each task is monitored in real time, and each task's loss value is mapped to the prediction confidence level for that task using a negative exponential function. For example, the confidence level for the text task is... The confidence level for the sketch task is The confidence level of the fusion task is The lower the loss value, the higher the confidence level.

[0058] 2) Hierarchical weight modulation

[0059] To train relatively stable text task confidence As an anchoring reference, its modulation factor for the sketching task and fusion task is calculated respectively. and This calculation utilizes the harmonic mean of confidence levels, for example... , Simultaneously, the feedback modulation factors of the sketching and fusion tasks on the text task are also calculated. and This design constitutes a hierarchical regulation network.

[0060] 3) Dynamic reweighting of losses

[0061] The contribution weights of each task's loss are dynamically adjusted using the calculated modulation factor: specifically, using the modulation factor of the sketch task. The loss weights of the sketching task are dynamically adjusted, utilizing the modulation factor of the fusion task. The loss weights for the fusion task are dynamically adjusted, and the feedback modulation factor for the text task is dynamically applied using the sketching and fusion tasks. and Adjusting the loss weights for text tasks will adaptively update the corresponding modal loss:

[0062] ;

[0063] ;

[0064] ;

[0065] in, This represents the loss function for the sketch retrieval task after the loss weights have been updated. and These are hyperparameters that control the scaling and decay characteristics of the loss adjustment curve. This represents the loss function for the fusion retrieval task after the loss weights have been updated. This represents the loss function for the text retrieval task after the loss weights have been updated.

[0066] Finally, the total loss function is:

[0067] .

[0068] This invention systematically addresses key challenges in multi-task, multi-modal pedestrian re-identification through three core innovative modules: 1. An innovative cross-modal information compensation mechanism: A Sketch Information Compensation (SICM) module is designed. Instead of simply concatenating or weighting text and sketch features, this module uses identity-aligned text descriptions as semantic guidance to actively mine and reconstruct detailed information such as color and texture from corresponding pedestrian photos. Then, affine transformations are used to modulate these details into an abstract sketch representation. This "guide-mining-modulation" compensation mechanism achieves targeted and accurate information transfer from rich modalities (text / photo) to poor modalities (sketches), effectively compensating for the inherent deficiencies of sketches in visual detail and generating enhanced sketch representations that are richer in information and more discriminative. 2. A personalized optimization strategy that respects task differences: A personalized supervised learning (ISL) strategy is innovatively adopted. This strategy abandons the "one-size-fits-all" approach of using the same loss function for all tasks. For the fine-grained semantic matching characteristics of text retrieval, it employs triplet loss for constraint; for the structural matching and high intra-class variance challenges of sketch and multimodal retrieval, it uses contrastive loss for optimization. This strategy respects the essential differences in optimization objectives and learning difficulty among different modal query tasks, providing the most suitable optimization gradient for each task, avoiding optimization conflicts between tasks caused by a single loss function, and ensuring that each task converges towards its optimal direction. 3. A dynamically balanced hierarchical multi-task coordination mechanism innovatively introduces an adaptive confidence loss adjustment (ACLR) mechanism. This mechanism works by monitoring the prediction confidence of each task in real time during training, and using the relatively stable text task as an anchor point to construct a hierarchical confidence interaction network, dynamically calculating and adjusting the loss weights of the sketch task, fusion task, and the text task itself. This mechanism achieves unbalanced dynamic weight allocation based on the real-time learning state of the tasks. When a task is difficult to learn (low confidence), its loss weight is adaptively increased to receive more attention; conversely, it is decreased. This effectively alleviates training imbalance caused by different task convergence speeds and ensures that shared parameters are optimized in a balanced manner to adapt to all query modalities.

[0069] Through the synergistic effect of the above three core innovative modules, this invention achieves the following outstanding technical effects:

[0070] Technical Advantage 1: Comprehensive Performance Leadership

[0071] Table 1 below shows a comparison of state-of-the-art methods on the Tri-Market-Sketch-1K and TriReID (PKU-Sketch) datasets. In the table, 'S' represents sketch-only query and 'T+S' represents sketch + text fusion query. Table 2 shows a comparison of state-of-the-art methods on the Tri-CUHK-PEDES and Tri-RSTPReid datasets. In the table, 'T' represents text-only query. In the four authoritative benchmark tests of Tri-Market-Sketch-1K, TriReID, Tri-CUHK-PEDES, and Tri-RSTPReid, the TPL-Net proposed in this invention achieved state-of-the-art retrieval accuracy (such as Rank-1 accuracy and mAP) in all three query modalities: text (T), sketch (S), and text + sketch fusion (T+S). This demonstrates the unity and efficiency of the framework of this invention, enabling it to master all subtasks simultaneously.

[0072]

[0073] Table 1

[0074]

[0075] Table 2

[0076] Technical effect 2: Each innovative module makes a clear and complementary contribution.

[0077] Table 3 below shows the ablation studies of the proposed components (ISL, ACLR, SICM) on the Tri-Market-Sketch-1K and Tri-CUHK-PEDES datasets. B represents the baseline model using a dual-stream Transformer encoder and supervised by contrastive loss. The ablation experiments shown in Table 3 quantitatively demonstrate the indispensability of each module: the introduction of ISL brings a comprehensive performance improvement in the first stage, validating the effectiveness of differential supervision. The addition of ACLR further optimizes the multi-task balance, especially bringing significant gains in text tasks, demonstrating the necessity of the dynamic adjustment mechanism. SICM ultimately fills the gap in the sketch modality, not only achieving outstanding results in sketch retrieval but also positively promoting text retrieval tasks by improving the quality of shared features, reflecting the synergistic enhancement effect between modules.

[0078]

[0079] Table 3

[0080] Technical effect 3: Powerful cross-domain generalization capability

[0081] Table 4 below shows the cross-domain retrieval performance on the PKU-Sketch dataset, reporting the top k accuracy rates (%). Without any fine-tuning, TPL-Net trained on the Tri-Market-Sketch-1K dataset was directly applied to the cross-domain PKU-Sketch dataset, significantly outperforming existing methods in both sketch query and fusion query. This demonstrates that the pedestrian representation learned in this invention possesses high transferability and robustness, reduces dependence on target domain data, and enhances its practical application value.

[0082]

[0083] Table 4

[0084] Technical Effect 4: Improved Interpretable Feature Learning

[0085] Figure 3 A visualization and comparative analysis chart of attention feature maps. Figures 4a-4c This is a visualization and comparative analysis chart of the feature distribution. In the chart, different shapes represent different modalities, and colors represent different pedestrian IDs. Figure 4a This is the original feature distribution map. Figure 4b The feature distribution map of the baseline model. Figure 4c This is a feature distribution map of the TPL-Net proposed in this invention. These two visualizations intuitively demonstrate the internal working mechanism and effects of this invention: 1) More precise attention: After adding SICM, the model focuses more intently on key body structures of pedestrians (such as torso and limb contours), reducing background interference, indicating that the compensation mechanism effectively improves the discriminative power of the features. 2) Better feature distribution: In the feature space learned by TPL-Net, samples of different modalities of the same pedestrian are tightly clustered, while samples of different pedestrians are clearly separated, forming an ideal clustering structure "centered on the photo and surrounded by multiple modalities". This directly confirms that the ISL strategy enhances intra-class consistency, the ACLR mechanism improves inter-class discriminative power, and the SICM module narrows the modal gap between the sketch and the photo.

[0086] On the other hand, such as Figure 5 As shown, the present invention also provides a pedestrian re-identification method, which specifically includes the following steps:

[0087] S1, a visual encoder and a text encoder are used to process the image and text respectively, and the processed visual features and text features are output respectively. The image includes a photograph and a sketch, and the visual features include photograph features and first sketch features.

[0088] S2, extract semantically related text-guided visual features from the photo features, and integrate the text-guided visual features into the first sketch features to output a compensated, information-enhanced second sketch representation;

[0089] Specifically, S2 includes: mapping the photo features and text features to a common subspace through a learnable first linear transformation layer and a second linear transformation layer, respectively; calculating the similarity between each spatial location of the transformed photo features and each word of the text features to form an association matrix; using the association matrix as attention weights to perform weighted summation on the original photo features, thereby aggregating the semantically related visual region features of the text features to obtain the text-guided visual features; performing adaptive instance normalization on the original sketch features; and generating affine parameters for the text-guided visual features through a multilayer perceptron; using the affine parameters to perform channel-wise scaling and biasing operations on the normalized sketch features, and finally outputting the second sketch feature.

[0090] S3, different loss functions are used to supervise the text retrieval task and the visual retrieval task respectively, and an adaptive loss weight adjustment mechanism is used to dynamically adjust the loss weight of each task during the supervision process; the text retrieval task is a text-to-photo retrieval task, and the visual retrieval task includes a sketch-to-photo sketch retrieval task and a fusion retrieval task of text and sketch fusion features to photo fusion retrieval task.

[0091] Specifically, S3 includes: real-time monitoring of the loss value of each task, and mapping the loss value of each task to the confidence level of each task; using the confidence level of the text task as an anchor benchmark, calculating its modulation factor for the sketching task and the fusion task respectively, and also calculating the feedback modulation factor of the sketching task and the fusion task for the text task respectively; dynamically adjusting the loss weight of the sketching task using the modulation factor of the sketching task, dynamically adjusting the loss weight of the fusion task using the modulation factor of the fusion task, and dynamically adjusting the loss weight of the text task using the feedback modulation factor of the sketching task and the fusion task for the text task.

[0092] In addition, the specific workflow and working principle of each step mentioned above can be found in the description of the pedestrian re-identification system, which will not be repeated here.

[0093] In another aspect, the present invention also provides a readable storage medium having a computer program stored thereon, which, when run, implements the steps in the pedestrian re-identification method provided in the above embodiments.

[0094] In another aspect, the present invention also provides an electronic device, including a memory and a processor, wherein the memory stores a computer program, and the computer program, when executed by the processor, performs the steps in the above-described pedestrian re-identification method.

[0095] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a processor-including system, or other system that can fetch and execute instructions from, an instruction execution system, apparatus, or device). For the purposes of this specification, "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transmit programs for use by, or in conjunction with, an instruction execution system, apparatus, or device.

[0096] More specific examples (a non-exhaustive list) of readable storage media include: electrical connections (electronic devices) with one or more wires, portable computer disk drives (magnetic devices), random access memory (RAM), read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic devices, and portable optical disc read-only memory (CDROM). Furthermore, readable storage media can even be paper or other suitable media on which the program can be printed, since the program can be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, interpreting, or otherwise processing as necessary, and then stored in computer memory.

[0097] It should be understood that various parts of the present invention can be implemented in hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented in software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0098] The advantages of this invention lie in its construction of a unified multi-task learning framework: on the one hand, by designing a sketch information compensation module, it explicitly supplements the missing color and texture information of the sketch at the structural level using textual semantics, promoting feature complementarity between modalities; on the other hand, it introduces personalized supervised learning strategies and adaptive loss adjustment mechanisms, performing differentiated supervision and dynamic balancing for different task characteristics at the optimization level, coordinating multi-task collaborative convergence; ultimately, it achieves a multi-modal person re-identification system with higher retrieval accuracy, stronger generalization ability, and greater applicability to complex real-world scenarios. This overcomes three major technical problems of existing description-based person re-identification methods: insufficient generalization ability due to modal isolation training, the uniform optimization strategy ignoring the essential differences between text and sketch tasks, and the lack of an effective cross-modal information complementarity mechanism.

[0099] The foregoing description of specific exemplary embodiments of the invention is for illustrative and explanatory purposes. These descriptions are not intended to limit the invention to the precise forms disclosed, and it will be apparent that many changes and variations can be made in accordance with the foregoing teachings. The exemplary embodiments were chosen and described in order to explain the specific principles of the invention and its practical application, thereby enabling those skilled in the art to implement and utilize various different exemplary embodiments of the invention, as well as various different choices and variations. The scope of the invention is intended to be defined by the claims and their equivalents.

[0100] The foregoing description of specific exemplary embodiments of the invention is for illustrative and explanatory purposes. These descriptions are not intended to limit the invention to the precise forms disclosed, and it will be apparent that many changes and variations can be made in accordance with the foregoing teachings. The exemplary embodiments were chosen and described in order to explain the specific principles of the invention and its practical application, thereby enabling those skilled in the art to implement and utilize various different exemplary embodiments of the invention, as well as various different choices and variations. The scope of the invention is intended to be defined by the claims and their equivalents.

Claims

1. A pedestrian re-identification system, characterized in that, The system includes: The encoder includes a visual encoder and a text encoder, which are used to process the input image and text respectively, and output visual features and text features respectively after processing. The image includes a photograph and a sketch, and the visual features include photograph features and first sketch features. The sketch information compensation module, connected to the encoder, is used to mine semantically related text-guided visual features from the photo features, and to fuse the text-guided visual features into the first sketch features, and output the compensated, information-enhanced second sketch representation. An adaptive confidence loss adjustment module, connected to the sketch information compensation module, is used to supervise the text retrieval task and the visual retrieval task respectively using different loss functions, and to dynamically adjust the loss weight of each task during the supervision process using an adaptive loss weight adjustment mechanism; the text retrieval task is a text-to-photo retrieval task, and the visual retrieval task includes a sketch-to-photo sketch retrieval task and a fusion retrieval task of text and sketch fusion features to photo fusion retrieval task.

2. The system as described in claim 1, characterized in that, The encoder uses a Transformer-based model, the visual encoder uses the Vision Transformer model, and the text encoder uses the standard Transformer model.

3. The system as described in claim 1, characterized in that, The sketch information compensation module includes a text-guided visual miner and an information compensation submodule. The visual miner is connected to the visual encoder and the text encoder, and is used to receive the photo features from the visual encoder and the text features from the text encoder as input to mine the text-guided visual features. The information compensation submodule is connected to the visual encoder and the visual miner, and is used to fuse the text-guided visual features output by the visual miner into the first sketch features from the visual encoder, thereby outputting the second sketch representation.

4. The system as described in claim 3, characterized in that, The processing steps of the visual mining tool include: mapping the photo features and text features to a common subspace through a learnable first linear transformation layer and a second linear transformation layer, respectively; calculating the similarity between each spatial location of the transformed photo features and each word of the text features to form an association matrix; using the association matrix as attention weights, performing weighted summation on the original photo features to aggregate the semantically related visual region features of the text features, thereby obtaining the text-guided visual features; the compensation process of the information compensation submodule includes: performing adaptive instance normalization on the original sketch features, and generating affine parameters for the text-guided visual features through a multilayer perceptron; using the affine parameters to perform channel-wise scaling and biasing operations on the normalized sketch features, and finally outputting the second sketch features.

5. The system as described in claim 2, characterized in that, The supervised learning module uses a triplet marginal loss function to supervise text retrieval tasks and a contrastive loss function to supervise visual retrieval tasks.

6. The system as described in claim 1, characterized in that, The adaptive confidence loss adjustment module's weight adjustment process for each task includes: real-time monitoring of the loss value of each task and mapping the loss value of each task to the confidence level of each task; using the confidence level of the text task as an anchor benchmark, calculating its modulation factor for the sketching task and the fusion task respectively, and also calculating the feedback modulation factor of the sketching task and the fusion task for the text task respectively; dynamically adjusting the loss weight of the sketching task using the modulation factor of the sketching task, dynamically adjusting the loss weight of the fusion task using the modulation factor of the fusion task, and dynamically adjusting the loss weight of the text task using the feedback modulation factor of the sketching task and the fusion task for the text task.

7. A pedestrian re-identification method, characterized in that, The method includes: S1, a visual encoder and a text encoder are used to process the image and text respectively, and the processed visual features and text features are output respectively. The image includes a photograph and a sketch, and the visual features include photograph features and first sketch features. S2, extract semantically related text-guided visual features from the photo features, and integrate the text-guided visual features into the first sketch features to output a compensated, information-enhanced second sketch representation; S3, different loss functions are used to supervise the text retrieval task and the visual retrieval task respectively, and an adaptive loss weight adjustment mechanism is used to dynamically adjust the loss weight of each task during the supervision process; the text retrieval task is a text-to-photo retrieval task, and the visual retrieval task includes a sketch-to-photo sketch retrieval task and a fusion retrieval task of text and sketch fusion features to photo fusion retrieval task.

8. The method as described in claim 7, characterized in that, S2 includes: mapping the photo features and text features to a common subspace through learnable first and second linear transformation layers respectively; calculating the similarity between each spatial location of the transformed photo features and each word of the text features to form an association matrix; using the association matrix as attention weights to perform weighted summation on the original photo features, thereby aggregating visual region features semantically related to the text features to obtain the text-guided visual features; performing adaptive instance normalization on the original sketch features; and generating affine parameters for the text-guided visual features through a multilayer perceptron, using the affine parameters to adjust the normalized sketch features. The feature is scaled and biased channel by channel to finally output the second sketch feature; and / or, S3 includes: real-time monitoring of the loss value of each task and mapping the loss value of each task to the confidence level of each task; using the confidence level of the text task as the anchor benchmark, calculating its modulation factor for the sketch task and the fusion task respectively, and also calculating the feedback modulation factor of the sketch task and the fusion task for the text task respectively; dynamically adjusting the loss weight of the sketch task using the modulation factor of the sketch task, dynamically adjusting the loss weight of the fusion task using the modulation factor of the fusion task, and dynamically adjusting the loss weight of the text task using the feedback modulation factor of the sketch task and the fusion task for the text task.

9. A readable storage medium, characterized in that: The readable storage medium stores a computer program, which, when executed, performs the steps of the pedestrian re-identification method according to claim 7 or 8.

10. An electronic device, characterized in that: The electronic device includes a memory and a processor. The memory stores a computer program, which, when executed by the processor, performs the steps of the pedestrian re-identification method according to claim 7 or 8.

Images