Domain generalization pedestrian re-identification method based on self-prompt learning

Through the domain generalization method of self-prompt learning, high-quality segmentation masks and self-prompt sets are generated, which solves the problem of insufficient generalization ability of the pedestrian re-identification model under the changes in cross-camera airport scenes, and improves the recognition accuracy and generalization ability , suitable for privacy protection and high labeling cost scenarios.

CN118918610APending Publication Date: 2024-11-08UESTC (SHENZHEN) ADVANCED RES INST

Patent Information

Application Number
CN202411020428.2
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-29
Publication Date
2024-11-08

AI Technical Summary

Technical Problem

The existing pedestrian re-identification model lacks generalization capabilities when changes in cross-camera airport scenes, especially when the interference of background and occlusion, resulting in a decrease in recognition accuracy, especially when the cost of privacy protection and labeling is high. Domain data.

Method used

Using the domain generalization method based on self-censoring learning, by constructing an image segmentation model and a pedestrian re-identification model based on self-censoring, using attention guidance to generate a self-censoring set, perform pixel-level target-scene separation, and generate high-quality segmentation masks code, extract pedestrian features twice to improve identification accuracy.

Benefits of technology

Improve the accuracy and generalization ability of pedestrian re-identification, maintain high application value under high privacy protection or labeling costs, automatically locate pedestrian identity areas, screen out identity-independent information, ensure the minimum loss of edge information, and reliable generation. segmentation result.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118918610A_ABST
    Figure CN118918610A_ABST
Patent Text Reader

Abstract

The invention discloses a self-prompt learning-based domain generalization pedestrian re-identification method, which comprises the following steps of: firstly, constructing an image segmentation model for segmenting an input image according to a prompt to obtain a foreground image, pre-training by adopting a data set of a large-scale image segmentation task, and then, on the basis of the pre-trained image segmentation model, carrying out self-prompt learning on the basis of the pre-trained image segmentation model; according to the method, a pedestrian re-identification model based on self-prompt is constructed, a self-prompt set is generated in the model through attention guidance, so that segmentation prompt is generated, a feature map is processed through a soft mask technology based on an obtained segmentation result, then pedestrian features are extracted for the second time, and pedestrian re-identification is performed according to the secondary pedestrian features. According to the method, the high-quality prompt set is automatically generated, and pixel-level target-scene separation is carried out, so that the poor generalization ability of the model caused by interference of background, shielding and the like in an unseen camera scene is relieved, and the accuracy of pedestrian re-identification is improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to the technical field of pedestrian re-identification, and more specifically, relates to a domain generalization pedestrian re-identification method based on self-prompt learning. Background Art

[0002] Person re-identification (RE-identification) aims to retrieve people with the same identity from multiple disjoint camera views. It has a wide range of applications, such as video security and human-centric understanding analysis, and is mainly dedicated to solving the identification and retrieval of pedestrians across cameras and scenes. When it is difficult to obtain clear face video clips, person re-identification can fill the technical gap of face recognition and enhance the spatiotemporal continuity of cross-camera data.

[0003] Person re-identification has great potential value in both public safety and commercial economy. In terms of public safety, person re-identification technology can replace manual path finding for suspects and missing persons, greatly preventing the occurrence of safety accidents. In terms of commercial economy, by converting collected customer trajectories into benefit statistics through person re-identification, market managers can obtain more valuable insights and in-depth analysis of customer behavior. A major challenge of person re-identification is the changes in cross-camera scenes, such as background, occlusion, lighting, shooting angle, etc. Relying on the progress of deep learning, person re-identification under the independent and identically distributed assumption has been able to achieve excellent results.

[0004] However, when deploying person re-identification models to real-world safety scenarios, target domain data is often unavailable due to privacy protection and annotation costs. Therefore, how to improve the generalization ability of person re-identification models is a key technical issue. Since the target domain is invisible, the model will be disturbed by background and occlusion in unseen scenes, which affects its discriminability. To solve this problem, researchers have proposed two types of methods. The first type focuses on feature disentanglement, aiming to separate identity-related information from identity-irrelevant information; the second type is based on semantic segmentation, which separates pedestrian parts from interference factors at the image level. The segmentation models used in person re-identification in the past are usually pre-trained on human parsing datasets, but the segmentation performance under traditional segmentation models is poor. With the emergence of large-scale pre-trained visual models, segmentation-based person re-identification has gradually become valuable. However, these large-scale pre-trained semantic segmentation models rely heavily on high-resolution images and high-quality cues to achieve stable segmentation results. They can only achieve satisfactory results on images for person re-identification with high-quality manual cues. Summary of the invention

[0005] The purpose of the present invention is to overcome the shortcomings of the prior art and provide a domain generalization pedestrian re-identification method based on self-cue learning. By automatically generating a high-quality cue set, pixel-level object-scene separation is performed to alleviate the poor generalization ability of the model caused by interference such as background and occlusion in unseen camera scenes, thereby improving the accuracy of pedestrian re-identification.

[0006] In order to achieve the above object, the domain generalization pedestrian re-identification method based on self-prompt learning of the present invention comprises the following steps:

[0007] S1: Construct an image segmentation model to generate a segmentation mask image from an input image according to a prompt. Each pixel value in the segmentation mask image represents the probability that the corresponding pixel in the input image belongs to the foreground or background. Set a dataset for a large-scale image segmentation task according to actual needs and pre-train the image segmentation model. The image segmentation model includes an embedding module, a Transformer encoder, a pooling module, a prompt encoder, and a mask decoder, where:

[0008] The embedding module is used to divide the pedestrian image X of size C×H×W into N non-overlapping image patches, where C represents the number of channels of the input pedestrian image and H×W represents the size of the input pedestrian image; each patch is then mapped to a vector as an image token, thereby obtaining N image tokens x n , n = 1, 2, ..., N; and get the position embedding p of each image patch at the same time n , then N image tokens x n and position embedding p n The image token sequence Y = [x 1 +p 1 ,x 2 +p 2 ,…,x N +p N ] and output to the Transformer encoder;

[0009] The Transformer encoder consists of a stack of L layers of Transformer encoding blocks, which are used to encode the received image token sequence Y. Each layer of Transformer encoding blocks extracts a hidden representation Z for the input features. j , j = 1, 2, ..., L, the hidden representation Z of the last layer output L The hidden representation f(Z) of the image token sequence Y is sent to the pooling module;

[0010] The pooling module is used to perform a pooling operation on the hidden representation f(Z) to obtain a feature vector z=pooling(f(Z)) and output it to the decoder;

[0011] The prompt encoder is used to encode the prompt information to obtain a prompt vector t, and output it to the mask decoder;

[0012] The mask decoder is used to decode the feature vector z and the prompt vector t to obtain the segmentation mask mask;

[0013] S2: Based on the image segmentation model pre-trained in step S1, a self-cue-based person re-identification model is constructed, including the image segmentation model pre-trained in step S1 and an attention guidance cue generation module, a scene-independent feature extraction module, and a classifier, wherein:

[0014] The embedding module, Transformer encoder, and pooling module in the image segmentation model are used to extract features from the input image X and obtain pedestrian features z;

[0015] The attention-guided cue generation module is used to obtain the attention of each image token in the input image token and the output hidden representation from the L-layer Transformer encoding block of the Transformer encoder in the image segmentation model, and then generate a set of self-cues and output them to the cue encoder of the image segmentation model. The specific method is:

[0016] Remember the hidden representation Z of each layer of Transformer encoding block j The N image tokens are z j,n , for each input image token x in the input image token sequence Y n , each image token z of each layer of Transformer encoding block j,n The feature vector z output by the pooling module is used as a node, and the attention value between each input image token and the output image token extracted by each layer of the Transformer encoding block is used as the weight of the edge between the corresponding image tokens. The image token Z output by the last layer is L,n The weight of the edge between the feature vector z is set to 1 / N to form a weighted directed acyclic graph; then the maximum flow algorithm is used to obtain each input image token x n The maximum flow path r to the eigenvector z n , and its flow rate is w n ; Select the input image token corresponding to the top M largest flows m=1,2,…,M,n m Represents the sequence number of the input image token corresponding to the mth maximum flow; then from the M input image tokens Select several as prompt image tokens, and use their corresponding position information as self-prompts to form a self-prompt set. The screening method of prompt image tokens is:

[0017] 1) Input image token As the first hint image token, record its position information as S 1 ;

[0018] 2) Let sequence number k=2, m=2;

[0019] 3) Determine whether Represents the input image token The position of , d() represents the distance to be obtained, α represents the preset distance threshold, if yes, go to step 4), otherwise go to step 6);

[0020] 4) Input image token as the kth hint image token;

[0021] 5) Let k=k+1, and go to step 6);

[0022] 6) Determine whether m<M, if yes, go to step 7), otherwise the screening ends;

[0023] 7) Let m=m+1, and return to step 3);

[0024] The hint decoder in the image segmentation model is used to encode the hint information in the self-hint set to obtain the hint vector t;

[0025] The mask decoder in the image segmentation model is used to decode the segmentation mask mask according to the feature vector z and the prompt vector t;

[0026] The scene-independent feature extraction module is used to extract the jth segmentation mask of the Transformer encoder according to the segmentation mask output by the mask decoder. * The hidden representation of the output of the layer Transformer encoding block is processed, j * Set according to actual needs, and then re-enter the jth * The +1 layer Transformer encoding block performs secondary feature extraction, and finally the secondary pedestrian feature z′ is obtained by the pooling module and sent to the classifier. The specific method of hidden representation processing is:

[0027] Perform Gaussian convolution on the segmentation mask to map the classification result of the segmentation edge into a floating point value from 0 to 1 to obtain the pixel set segmentation soft mask mask′; select the jth * Hidden representation of the output of the layer Transformer encoding block To hide Deconvolution is performed to obtain features of the same size as the segmentation soft mask mask′ Use segmentation soft mask mask′ to feature Perform filtering operations to obtain features Then the feature Perform convolution to restore hidden representation The size of the feature Then hide the representation and Features Fusion to obtain a new hidden representation

[0028]

[0029] The classifier is used to classify according to the secondary pedestrian feature z′ to obtain the pedestrian ID;

[0030] S3: Obtain a training sample set based on the actual deployment scenario. Each training sample image is represented as i=1,2,…,D, where D represents the number of training sample images and the label f ​​of the training sample. i Indicates the ID of the pedestrian in the pedestrian image;

[0031] S4: fix the parameters of other modules in the self-cue-based person re-identification model, input each training sample in step S3 into the self-cue-based person re-identification model, optimize and train the Transformer encoder, and obtain a trained self-cue-based person re-identification model;

[0032] S5: When pedestrian re-identification is required, the pedestrian image is input into the self-suggestion-based pedestrian re-identification model trained in step S4 to obtain the pedestrian recognition result.

[0033] The present invention discloses a domain generalized person re-identification method based on self-cue learning. Firstly, an image segmentation model is constructed for obtaining a foreground image by segmenting an input image according to a cue, and a data set of a large-scale image segmentation task is used for pre-training. Then, based on the pre-trained image segmentation model, a person re-identification model based on self-cue is constructed. In the model, a self-cue set is generated by attention-guided generation to generate segmentation cues. Based on the segmentation result obtained in this way, the feature map is processed by a soft masking technology. Then, pedestrian features are extracted for a second time, and pedestrian re-identification is performed based on the secondary pedestrian features.

[0034] The present invention has the following beneficial effects:

[0035] 1) By introducing an attention-guided prompt generation method, the present invention can automatically locate the area containing pedestrian identity discrimination information, generate more refined segmentation prompts, and improve the ability to filter out identity-irrelevant information, thereby improving the accuracy of pedestrian re-identification;

[0036] 2) The present invention adopts scene-independent feature extraction, which can generate reliable segmentation results while minimizing edge information loss;

[0037] 3) The present invention has high practical application value in situations where there is a lack of target domain data due to privacy protection or high labeling costs. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 It is a flowchart of a specific implementation of the domain generalization pedestrian re-identification method based on self-prompt learning of the present invention;

[0039] Figure 2 It is a structural diagram of the image segmentation model in the present invention;

[0040] Figure 3 is a structural diagram of a self-cue-based person re-identification model in the present invention;

[0041] Figure 4 This is the comparison result between the segmentation mask generated by the attention-based prompt of the present invention and the artificial single prompt in this embodiment. DETAILED DESCRIPTION

[0042] The specific implementation of the present invention is described below in conjunction with the accompanying drawings so that those skilled in the art can better understand the present invention. It should be noted that in the following description, when the detailed description of known functions and designs may dilute the main content of the present invention, these descriptions will be omitted here.

[0043] Example

[0044] Figure 1 FIG. 1 is a flowchart of a specific implementation of the domain generalization person re-identification method based on self-prompt learning of the present invention. Figure 1 As shown, the specific steps of the domain generalization pedestrian re-identification method based on self-prompt learning of the present invention include:

[0045] S101: Build and pre-train an image segmentation model:

[0046] Build an image segmentation model to generate a segmentation mask image from the input image according to the prompt. Each pixel value in the segmentation mask image represents the probability that the corresponding pixel in the input image belongs to the foreground or background. Set a dataset for large-scale image segmentation tasks according to actual needs and pre-train the image segmentation model.

[0047] Figure 2 It is a structural diagram of the image segmentation model in the present invention. Figure 2 As shown, the image segmentation model in the present invention includes an embedding module, a Transformer encoder, a pooling module, a hint encoder and a mask decoder, wherein:

[0048] The embedding module is used to divide the pedestrian image X of size C×H×W into N non-overlapping image patches, where C represents the number of channels of the input pedestrian image and H×W represents the size of the input pedestrian image. Each patch is then mapped to a vector as an image token, thereby obtaining N image tokens x n , n = 1, 2, ..., N. At the same time, get the position embedding p of each image patch n , then N image tokens x n and position embedding p n Overlay to form image token sequence Y = [x 1 +p 1 ,x 2 +p 2 ,…,x N +p N ] and output to the Transformer encoder.

[0049] The Transformer encoder consists of a stack of L layers of Transformer encoding blocks, which are used to encode the received image token sequence Y. Each layer of Transformer encoding blocks extracts a hidden representation Z for the input features. j , the hidden representation Z of the last layer output L The hidden representation f(Z) of the image token sequence Y is sent to the pooling module.

[0050] The Transformer coding block is a commonly used neural network structure, and its specific structure and working process are not described in detail here. In this embodiment, the L-layer Transformer coding block alternately uses the local attention (window attention) mechanism and the global attention (global attention) mechanism to process the input features of the current Transformer coding block. The local attention mechanism and the global attention mechanism are two commonly used attention mechanisms. The specific process is not described here. In short, when the Transformer coding block adopts local attention, the input features will be divided into different windows on the two-dimensional plane, and then the attention operation will be performed within the window. When the Transformer coding block adopts global attention, it will be performed on all image tokens in the input features.

[0051] The pooling module is used to perform a pooling operation on the hidden representation f(Z) to obtain a feature vector z=pooling(f(Z)) and output it to the decoder. This is because the image token output by the Transformer encoder provided in the present invention does not contain a classification token, so a pooling operation needs to be added.

[0052] The prompt encoder is used to encode the prompt information (prompt) to obtain a prompt vector t, and output it to the mask decoder.

[0053] The mask decoder is used to decode the segmentation mask mask according to the feature vector z and the prompt vector t.

[0054] S102: Building a self-hinting based person re-identification model:

[0055] Based on the image segmentation model pre-trained in step S101, a self-cue-based person re-identification model is constructed. Figure 3 is a structural diagram of the pedestrian re-identification model based on self-prompt in the present invention. Figure 3 As shown, the self-cue-based person re-identification model in the present invention includes the pre-trained image segmentation model and attention guidance cue generation module, scene-independent feature extraction module, and classifier in step S101, wherein:

[0056] The embedding module, Transformer encoder, and pooling module in the image segmentation model are used to extract features from the input image X and obtain pedestrian features z.

[0057] The attention-guided cue generation module is used to obtain the attention of each image token in the input image token and the output hidden representation from the L-layer Transformer encoding block of the Transformer encoder in the image segmentation model, and then generate a set of self-cues and output them to the cue encoder of the image segmentation model. The specific method is:

[0058] Remember the hidden representation Z of each layer of Transformer encoding block j The N image tokens are z j,n , for each input image token x in the input image token sequence Y n , each image token z of each layer of Transformer encoding block j,n The feature vector z output by the pooling module is used as a node, and the attention value between each input image token and the output image token extracted by each layer of the Transformer encoding block is used as the weight of the edge between the corresponding image tokens. The image token Z output by the last layer is L,n The weight of the edge between the input image token xn and the feature vector z is set to 1 / N, forming a weighted directed acyclic graph. Then the maximum flow algorithm is used to obtain the maximum flow path r from each input image token xn to the feature vector z. n , and its flow rate is w n . Select the input image tokens corresponding to the top M largest flows m=1,2,…,M,n mRepresents the sequence number of the input image token corresponding to the mth maximum flow. Then from the M input image tokens Select several as prompt image tokens, and use their corresponding position information as self-prompts to form a self-prompt set. The screening method of prompt image tokens is:

[0059] 1) Select the first hint image token:

[0060] The input image token As the first hint image token, record its position information as S 1 .

[0061] 2) Let sequence number k=2, m=2.

[0062] 3) Determine whether Represents the input image token The position of , d() represents the distance to be calculated, α represents the preset distance threshold, if yes, go to step 4), otherwise go to step 6).

[0063] 4) Determine the kth prompt image token:

[0064] The input image token as the kth hint image token.

[0065] 5) Let k=k+1 and go to step 6).

[0066] 6) Determine whether m<M, if so, go to step 7), otherwise the screening ends.

[0067] 7) Let m=m+1, and return to step 3).

[0068] The hint decoder in the image segmentation model is used to encode the hint information in the self-hint set to obtain the hint vector t.

[0069] The mask decoder in the image segmentation model is used to decode the segmentation mask mask according to the feature vector z and the prompt vector t.

[0070] The scene-independent feature extraction module is used to extract the jth segmentation mask of the Transformer encoder according to the segmentation mask output by the mask decoder. * The hidden representation of the output of the layer Transformer encoding block is processed, j * Set according to actual needs, and then re-enter the jth * The +1 layer Transformer encoding block performs secondary feature extraction, and finally the secondary pedestrian feature z′ is obtained by the pooling module and sent to the classifier. The specific method of hidden representation processing is:

[0071] Perform Gaussian convolution on the segmentation mask to map the classification results of the segmentation edge into floating point values ​​from 0 to 1 to obtain the pixel set segmentation soft mask mask′. * Hidden representation of the output of the layer Transformer encoding block To hide Deconvolution is performed to obtain features of the same size as the segmentation soft mask mask′ Use segmentation soft mask mask′ to feature Perform filtering operations to obtain features Then the feature Perform convolution to restore hidden representation The size of the feature Then hide the representation and Features Fusion to obtain a new hidden representation

[0072]

[0073] According to the above description, in the self-prompted image segmentation model, an input image will generate two feature vectors. The first one is generated based on the input image, and the second one is generated based on a hidden representation of a layer after processing based on the segmentation mask. It can be seen that the feature vector generated the second time can better filter the background and thus strengthen the foreground features. Pedestrian re-identification based on the feature vector generated the second time can effectively improve the recognition accuracy. Experiments have found that j * ≤0.5L, that is, it is better to use shallow hidden representation for segmentation soft mask processing.

[0074] The classifier is used to classify according to the secondary pedestrian feature z′ to obtain the pedestrian ID.

[0075] According to the above description, in the process of self-prompt learning, the present invention guides prompt generation through attention, that is, uses the attention flow to locate the area of ​​an instance and generates sparse prompts. The sparse prompts and image embedding are jointly input to the mask decoder, and the mask related to the pedestrian is output; and the scene-independent feature extraction is adopted, and the mask is used to guide the model to filter background information, thereby effectively improving the accuracy of pedestrian re-identification.

[0076] S103: Obtain training sample set:

[0077] The training sample set is obtained according to the actual deployment scenario, and each training sample pedestrian image is represented as i=1,2,…,D, where D represents the number of training sample images and the label f ​​of the training sample. i Indicates the ID of the pedestrian in the pedestrian image.

[0078] S104: Training a self-hinting based person re-identification model:

[0079] The parameters of other modules in the self-cue-based person re-identification model are fixed, each training sample in step S103 is input into the self-cue-based person re-identification model, and the Transformer encoder is optimized and trained to obtain a trained self-cue-based person re-identification model.

[0080] The setting of the loss function is very important for the optimization training of the Transformer encoder. In this embodiment, in order to improve the training effect, the present invention proposes a comprehensive loss function calculation method, and the specific method is as follows:

[0081] The ID loss L of the training sample is calculated using the following formula: ce :

[0082]

[0083] Among them, B represents the number of training samples in a training batch, f b represents the label of the bth training sample in the current batch, z b ′ represents the secondary pedestrian features obtained from the bth training sample in the current batch, and C() represents the classification result of the classifier.

[0084] The triplet loss L of the training sample is calculated using the following formula: tri :

[0085]

[0086] Among them, d p represents the Euclidean distance between the bth training sample in the current batch and the secondary pedestrian features of the same pedestrian sample in the current batch of training samples, d n It represents the Euclidean distance between the bth training sample in the current batch and the secondary pedestrian features of different pedestrian samples in the current batch of training samples, and ε represents the difference boundary between positive and negative samples.

[0087] The consistency loss L of the training sample is calculated using the following formula: con :

[0088]

[0089] in, represents the new hidden representation obtained by the b-th training sample in the current batch, Z b,L represents the hidden representation of the b-th training sample in the current batch output by the Transformer encoder, |||| 2 represents the calculation of the two-norm, Representation and Hidden Representation The matching classifier, C L () indicates and hides Z b,L Matching classifiers, KL() means to obtain the KL divergence of the two classification results, and λ represents the preset hyperparameter.

[0090] Two goals can be achieved through consistency loss: one is to improve the model's deep perception of pedestrians and reduce the interference of background and occlusion; the other is to increase the ID discriminability of the hidden representation after segmentation soft mask processing.

[0091] Then the final loss LOSS is calculated using the following formula:

[0092] LOSS=αL ce +βL tri +γL con

[0093] Among them, α, β, and γ represent preset weights.

[0094] S105: Pedestrian Re-identification:

[0095] When pedestrian re-identification is required, the pedestrian image is input into the self-prompting based pedestrian re-identification model trained in step S104 to obtain the pedestrian recognition result.

[0096] In order to better illustrate the technical effect of the present invention, two specific examples are used to experimentally verify the present invention.

[0097] Example 1

[0098] In this embodiment, the experimental conditions are set as follows: system: Ubuntu 20.04, software: Python 3.6, graphics card: NVIDIA GeForce RTX 3090×4, video memory: 24576MB.

[0099] In order to comprehensively evaluate the reliability and effectiveness of attention-based cue generation in the segmentation mask of pedestrian re-identification tasks, the present invention conducts a series of visualization experiments to compare and analyze the segmentation results obtained by the traditional method of artificial single cue (SP) and the attention-based cue generation (APG) of the present invention. In the traditional artificial single cue method, due to the lack of in-depth understanding of the identity discrimination information in pedestrian images, it is often difficult to adapt to the diversity and complexity of pedestrian re-identification images, resulting in difficulty in accurately locating pedestrians and their related accessories in practical applications. For example, artificial single cues may not be able to distinguish between decorative items and key identity discrimination items on pedestrians, and may not be able to accurately identify pedestrian posture changes and occlusions.

[0100] To overcome these challenges, this paper proposes a self-hinting learning method based on attention flow. The method first uses a visual basic model to analyze the attention distribution in the image and identify the areas related to pedestrian identity discrimination. Then, guided by the attention flow, a set of sparse hints is automatically generated, which can accurately cover the entire body of the pedestrian and accessories related to his identity, such as backpacks, handbags, handheld items, etc. Figure 4 is a comparison result diagram of the segmentation mask generated by the attention-based prompt of the present invention and the artificial single prompt in this embodiment. Figure 4 It can be seen that the automatic prompt set generated by the self-prompt learning of the present invention not only comprehensively covers the entire pedestrian, but also accurately locates the pedestrian accessories with identity recognition capabilities. Figure 4 As shown in the first row, self-hint learning can accurately identify the backpacks and handbags carried by pedestrians, even if these accessories are small in size or partially occluded in the image. Similarly, in the second row, self-hint learning can also identify the objects in the pedestrians' hands, which is usually a challenge for manual single hints. In addition, the hints generated by self-hint learning can effectively filter out the interference information caused by occlusion. Figure 4 As shown in the third column, in complex scenes, when the target pedestrian is partially occluded by other pedestrians, self-hint learning can still accurately locate the target pedestrian and filter out the interference information of the occluded pedestrian, which is crucial to improving the accuracy of pedestrian re-identification.

[0101] In general, the experimental results of the present invention fully demonstrate the reliability and effectiveness of self-prompt learning in generating high-quality segmentation masks, and provide an innovative solution for the pedestrian re-identification task. Through self-prompt learning, the present invention can significantly improve the generalization ability and accuracy of the pedestrian re-identification system in complex scenarios, and has broad application prospects.

[0102] Example 2

[0103] In this embodiment, the experimental conditions are set as follows: system: Ubuntu 20.04, software: Python 3.6, graphics card: NVIDIA GeForce RTX 3090×4, video memory: 24576MB.

[0104] In order to more accurately evaluate the effectiveness of the present invention in pedestrian re-identification data, this embodiment conducts a series of generalization experiments on pedestrian re-identification. Market1501 (M), MSMT17 (MS), CUHK02 (C2), CUHK03 (C3), CUHK-SYSU (CS), Duke-MTMC (D) are used as source domains (training sets) and M+CS+C2+C3 is used as source domain training. The present invention is compared with 12 existing comparison methods on four target domains PRID, GRID, VIPeR, and iLIDs. The comparison methods include:

[0105] MLDG, for details, please see the paper "D.Li, Y.Yang, Y.-Z.Song, and T.Hospedales, "Learning togeneralize: Meta-learning for domain generalization," in Proceedings of the AAAI conference on artificial intelligence, vol.32, no.1, 2018.";

[0106] PPA, for details, see the paper "S.Qiao,C.Liu,W.Shen,and ALYuille,“Few-shot image recognition by predicting parameters from activations,”in Proceedings of theIEEE conference on computer vision and pattern recognition,2018,pp.7229–7238.";

[0107] DIMN, for details, please see the paper "J.Song, Y.Yang, Y.-Z.Song, T.Xiang, and TMHospedales, "Generalizable person re-identification by domain-invariant mapping network," in IEEE Conference on Computer Vision and Pattern Recognition, 2019.";

[0108] SNR, for details, see the paper "X. Jin, C. Lan, W. Zeng, Z. Chen, and L. Zhang, "Style normalization and restitution for generalizable person re-identification," in IEEE Conference on Computer Vision and Pattern Recognition, 2020.";

[0109] RaMoE, for details, see the paper "Y. Dai, X. Li, J. Liu, Z. Tong, and L.-Y. Duan, "Generalizable person reidentification with relevance-aware mixture of experts," in Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition, 2021.";

[0110] DMG-Net, for details, see the paper "Y. Bai, J. Jiao, W. Ce, J. Liu, Y. Lou, X. Feng, and L.-Y. Duan, "Person30k: A dual-meta generalization network for person reidentification," in Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition, 2021, pp. 2123–2132.";

[0111] DIR-ReID, for details, see the paper "Y.-F. Zhang, Z. Zhang, D. Li, Z. Jia, L. Wang, and T. Tan, "Learning domain invariant representations for generalizable person re-identification," IEEE Transactions on Image Processing, vol. 32, pp. 509–523, 2022.";

[0112] QAConv. For details, see the paper "S. Liao and L. Shao, 'Interpretable and generalizable person reidentification with query-adaptive convolution and temporal lifting,' in European Conference on Computer Vision. Springer, 2020, pp. 456–474.";

[0113] M3L. For details, see the paper "Y. Zhao, Z. Zhong, F. Yang, Z. Luo, Y. Lin, S. Li, and N. Sebe, 'Learning to generalize unseen domains via memory-based multi-source metalearning for person re-identification,' in Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition, 2021, pp. 6277–6286.";

[0114] MetaBIN. For details, see the paper "S. Choi, T. Kim, M. Jeong, H. Park, and C. Kim, 'Meta batch instance normalization for generalizable person re-identification,' in Proceedings of the IEEE / CVF conference on Computer Vision and Pattern Recognition, 2021, pp. 3425–3435.";

[0115] META. For details, see the paper "B. Xu, J. Liang, L. He, and Z. Sun, 'Mimic embedding via adaptive aggregation: learning generalizable person re-identification,' in European Conference on Computer Vision. Springer, 2022, pp. 372–388.";

[0116] ACL. For details, see the paper “P. Zhang, H. Dou, Y. Yu, and X. Li, “Adaptive cross-domain learning for generalizable person re-identification,” in European Conference on Computer Vision. Springer, 2022, pp. 215–232.”

[0117] Table 1 is a comparison table of the pedestrian re-identification performance of the present invention and the comparative method in this embodiment, which tests the two indicators of mean average precision mAP (%) and first correct rate R1 (%) respectively. The present invention is divided into two models, B and L, which respectively indicate that the pre-training models used are standard (ViT-B) and large (ViT-L).

[0118]

[0119] Table 1

[0120] As shown in Table 1, our method outperforms the state-of-the-art methods by a large margin. We achieve higher mAP and R1 on PRID with ViT-B as the backbone, 12.2% and 16.9% higher than ACL, respectively. On average, across the four datasets, our method outperforms the state-of-the-art methods by 5.7% in mAP and 7.4% in R1. Notably, our method outperforms previous methods, even those that use Duke-MTMC as part of the training set, e.g., on the GRID dataset, our method achieves 12.7% higher mAP than DMG-Net.

[0121] Then, in order to further verify the present invention in the scenario where the test domain data is larger and more difficult, this embodiment conducted a leave-one-out experiment on four datasets: CUHK03, MSMT17, Market1501, and CUHK-SYSU, that is, three of the datasets were selected as source domains, and the remaining dataset was used as the target domain for the test. Table 2 is a comparison table of image classification performance of the present invention and the comparative method in this embodiment under two settings: protocol 1: only the source domain training set, and protocol 2: both the training set and the test set are used.

[0122]

[0123] Table 2

[0124] For Protocol 1, only the training set of the dataset is used for training. As shown in Table 2, due to the existence of a large number of evaluation datasets for domain generalization person re-identification, no method can currently achieve the state-of-the-art methods across all datasets. For example, META and ACL perform best on the MSMT17 and Market1501 datasets, respectively. When using ViT-B as the backbone, the present invention exceeds ACL or META. In addition, when using ViT-L as the backbone, the performance of the present invention is consistently significantly better than all state-of-the-art methods. For example, compared with the current state-of-the-art CUHK03 dataset, the present invention has an advantage of 4.6% and 3.1% in mAP, respectively.

[0125] Compared with Protocol 1, Protocol 2 involves the integration of training and test sets during training, resulting in a substantial increase in training data. As shown in Table 2, SPL maintains a significant lead with more training data. Specifically, the present invention outperforms ACL by 6.7% in mAP and 5.9% in R1. Compared with the recently proposed method SuA-SpML, the present invention outperforms it by 20.1% and 19.2% in mAP and Rank 1, respectively.

[0126] Through the above data analysis and comparison, it can be seen that the pedestrian re-identification generalization ability of the present invention is stronger than that of other methods, and has broader application prospects.

[0127] Although the above describes the illustrative specific embodiments of the present invention to facilitate those skilled in the art to understand the present invention, it should be clear that the present invention is not limited to the scope of the specific embodiments. For those of ordinary skill in the art, as long as various changes are within the spirit and scope of the present invention as defined and determined by the attached claims, these changes are obvious, and all inventions and creations using the concept of the present invention are protected.

Claims

1. A domain generalization pedestrian re-identification method based on self-hint learning, comprising the following steps: S1: Build an image segmentation model to generate a segmentation mask image from the input image according to the prompt. Each pixel value in the segmentation mask image represents the probability that the corresponding pixel in the input image belongs to the foreground or background. According to actual needs, a dataset for large-scale image segmentation tasks is set up to pre-train the image segmentation model; the image segmentation model includes an embedding module, a Transformer encoder, a pooling module, a hint encoder, and a mask decoder, where: The embedding module is used to divide the pedestrian image X of size C×H×W into N non-overlapping image patches, where C represents the number of channels of the input pedestrian image and H×W represents the size of the input pedestrian image; each patch is then mapped to a vector as an image token, thereby obtaining N image tokens x n , n = 1, 2, ..., N; and get the position embedding p of each image patch at the same time n , then N image tokens x n and position embedding p n The image token sequence Y = [x1+p1, x2+p2,…, x N +p N ] and output to the Transformer encoder; The Transformer encoder consists of a stack of L layers of Transformer encoding blocks, which are used to encode the received image token sequence Y. Each layer of Transformer encoding blocks extracts a hidden representation Z for the input features. j , j = 1, 2, ..., L, the hidden representation Z of the last layer output L The hidden representation f(Z) of the image token sequence Y is sent to the pooling module; The pooling module is used to perform a pooling operation on the hidden representation f(Z) to obtain a feature vector z=pooling(f(Z)) and output it to the decoder; The prompt encoder is used to encode the prompt information to obtain a prompt vector t, and output it to the mask decoder; The mask decoder is used to decode the feature vector z and the prompt vector t to obtain the segmentation mask mask; S2: Based on the image segmentation model pre-trained in step S1, a self-cue-based person re-identification model is constructed, including the image segmentation model pre-trained in step S1 and an attention guidance cue generation module, a scene-independent feature extraction module, and a classifier, wherein: The embedding module, Transformer encoder, and pooling module in the image segmentation model are used to extract features from the input image X and obtain pedestrian features z; The attention-guided cue generation module is used to obtain the attention of each image token in the input image token and the output hidden representation from the L-layer Transformer encoding block of the Transformer encoder in the image segmentation model, and then generate a set of self-cues and output them to the cue encoder of the image segmentation model. The specific method is: Remember the hidden representation Z of each layer of Transformer encoding block j The N image tokens are z j,n , for each input image token x in the input image token sequence Y n , each image token z of each layer of Transformer encoding block j,n The feature vector z output by the pooling module is used as a node, and the attention value between each input image token and the output image token extracted by each layer of the Transformer encoding block is used as the weight of the edge between the corresponding image tokens. The image token Z output by the last layer is L,n The weight of the edge between the feature vector z is set to 1 / N to form a weighted directed acyclic graph; then the maximum flow algorithm is used to obtain each input image token x n The maximum flow path r to the eigenvector z n , and its flow rate is w n ; Select the input image token corresponding to the top M largest flows m=1,2,…,M,n m Represents the sequence number of the input image token corresponding to the mth maximum flow; then from the M input image tokens Select several as prompt image tokens, and use their corresponding position information as self-prompts to form a self-prompt set. The screening method of prompt image tokens is: 1) Input image token As the first hint image token, its position information is recorded as S1; 2) Let sequence number k=2, m=2; 3) Determine whether Represents the input image token The position of , d() represents the distance to be obtained, α represents the preset distance threshold, if yes, go to step 4), otherwise go to step 6); 4) Input image token as the kth hint image token; 5) Let k=k+1, and go to step 6); 6) Determine whether m<M, if yes, go to step 7), otherwise the screening ends; 7) Let m=m+1, and return to step 3); The hint decoder in the image segmentation model is used to encode the hint information in the self-hint set to obtain the hint vector t; The mask decoder in the image segmentation model is used to decode the segmentation mask mask according to the feature vector z and the prompt vector t; The scene-independent feature extraction module is used to extract the jth segmentation mask of the Transformer encoder according to the segmentation mask output by the mask decoder. * The hidden representation of the output of the layer Transformer encoding block is processed, j * Set according to actual needs, and then re-enter the jth * The +1 layer Transformer encoding block performs secondary feature extraction, and finally the secondary pedestrian feature z′ is obtained by the pooling module and sent to the classifier. The specific method of hidden representation processing is: Perform Gaussian convolution on the segmentation mask mask to map the classification result of the segmentation edge into a floating point value from 0 to 1 to obtain the pixel set segmentation soft mask mask′; select the jth * Hidden representation of the output of the layer Transformer encoding block To hide Deconvolution is performed to obtain features of the same size as the segmentation soft mask mask′ Use segmentation soft mask mask′ to feature Perform filtering operations to obtain features Then the feature Perform convolution to restore hidden representation The size of the feature Then hide the representation and Features Fusion to obtain a new hidden representation The classifier is used to classify according to the secondary pedestrian feature z′ to obtain the pedestrian ID; S3: Obtain a training sample set based on the actual deployment scenario. Each training sample pedestrian image is represented as i=1,2,…,D, where D represents the number of training sample images and the label f ​​of the training sample. i Indicates the ID of the pedestrian in the pedestrian image; S4: fix the parameters of other modules in the self-cue-based person re-identification model, input each training sample in step S3 into the self-cue-based person re-identification model, optimize and train the Transformer encoder, and obtain a trained self-cue-based person re-identification model; S5: When pedestrian re-identification is required, the pedestrian image is input into the self-suggestion-based pedestrian re-identification model trained in step S4 to obtain the pedestrian recognition result.

2. The domain generalization person re-identification method according to claim 1, characterized in that: In the step S1, the L-layer Transformer encoding block in the Transformer encoder alternately uses the local attention mechanism and the global attention mechanism to process the input features of the current Transformer encoding block.

3. The domain generalization person re-identification method according to claim 1, characterized in that: The Transformer coding block number j selected when hiding the representation processing in step S2 * ≤0.5L.

4. The domain generalization person re-identification method according to claim 1, characterized in that: The calculation method of the loss function during the Transformer encoder optimization training in step S4 is as follows: The ID loss L of the training sample is calculated using the following formula: ce : Among them, B represents the number of training samples in a training batch, f b represents the label of the bth training sample in the current batch, z b ′ represents the secondary pedestrian features obtained from the b-th training sample in the current batch, and C() represents the classification result of the classifier; The triplet loss L of the training sample is calculated using the following formula: tri : Among them, d p represents the Euclidean distance between the bth training sample in the current batch and the secondary pedestrian features of the same pedestrian sample in the current batch of training samples, d n represents the Euclidean distance between the bth training sample in the current batch and the secondary pedestrian features of different pedestrian samples in the current batch of training samples, and ε represents the difference boundary between positive and negative samples; The consistency loss L of the training sample is calculated using the following formula: con : in, represents the new hidden representation obtained by the b-th training sample in the current batch, Z b,L represents the hidden representation of the b-th training sample in the current batch output by the Transformer encoder, || ||2 represents the calculation of the two-norm, Representation and Hidden Representation The matching classifier, C L () indicates and hides Z b,L Matching classifiers, KL() means to obtain the KL divergence of the two classification results, and λ represents the preset hyperparameter; Then the final loss LOSS is calculated using the following formula: LOSS=αL ce +βL tri +γL con Among them, α, β, and γ represent preset weights.

Citation Information

Patent Citations

  • Pedestrian re-identification method and device based on attention guidance and storage medium

    CN112906614A

  • Domain adaptive pedestrian re-identification method

    CN113158901A

  • Domain generalization pedestrian re-identification method and system based on element-graph perception

    CN116386079A

  • Domain generalization pedestrian re-identification method based on multi-level data disturbance strategy

    CN117173477A

Cited By

  • Training method of pedestrian re-identification model supporting multiple scenes and re-identification method

    CN119763149A