Clothing-changing pedestrian re-identification method based on clothing semantic progressive guide learning

Abstract

The invention discloses a clothing-changing pedestrian re-identification method based on clothing semantic progressive guide learning, and belongs to the technical field of computer vision. Enabling the mask image to be gradually close to an image containing rich clothing information in a feature space through a progressive guide learning module, and gradually learning and capturing clothing hierarchical features; and through a clothing feature reinforcement learning module, fusing the original image of each type of pedestrians and the category mean value of the type of pedestrians, and inputting fused features into a clothing feature reinforcement learning loss function to calculate loss and update model parameters. According to the invention, through the progressive guide learning module, the model can dynamically learn richer clothing characterization information; and through a clothing feature enhancement learning module, the clothing characterization capability of each type of pedestrian image is enhanced from the perspective of a feature space. According to the invention, under a long-term security and protection monitoring scene in which pedestrian clothes are varied, more robust pedestrian discriminative features can be learned.

Description

Technical Field

[0001] The present invention belongs to the technical field of computer vision, and in particular relates to a method for re-identifying pedestrians who have changed clothes based on progressive guided learning of clothing semantics. Background Art

[0002] Pedestrian re-identification is an important task in computer vision. It is mainly used to identify the same pedestrian in multi-camera scenarios by comparing images or videos taken at different times and from different perspectives. With the growing demand for intelligent monitoring and public safety, pedestrian re-identification technology plays a key role in practical applications. In recent years, with the continuous evolution of deep learning algorithms, the accuracy and robustness of pedestrian re-identification have been significantly improved, allowing the technology to maintain a high recognition rate in complex scenarios such as cross-perspective, lighting changes, and occlusion. This has further promoted its widespread application in real life, such as intelligent transportation, smart security, and social security management.

[0003] Person re-identification matches and identifies the same pedestrian in images or videos taken by different cameras. It is affected by changes in camera perspective, fluctuations in lighting conditions, diversity in pedestrian postures, occlusions, and changes in clothing, making person re-identification a challenging task. These factors often exist simultaneously in real scenes, increasing the difficulty of extracting robust features. Early methods relied on manually designed features and focused on underlying information such as color and texture, but their reliability dropped significantly when lighting changes or pedestrians change clothes. In addition, long-distance shooting, low-pixel images, and changes in complex backgrounds further reduce the accuracy of recognition, so how to improve the performance of person re-identification in these situations remains a key research challenge.

[0004] In recent years, with the rapid development of deep learning technology, remarkable results have been achieved in the field of person re-identification. In particular, the release of large-scale datasets such as CUHK03 and Market1501, as well as the widespread application of deep convolutional neural networks, have made pedestrian re-identification technology receive widespread attention. Methods based on deep learning, such as Deep ReID proposed by Wei Li et al., which automatically learns identity-sensitive and viewpoint-insensitive features through deep convolutional networks, and the effective application of PRW proposed by Liang Zheng et al. in complex scenes, have significantly improved the accuracy and robustness of pedestrian re-identification. In addition, methods such as the full-scale network OSNet and the fine-grained feature mining PCB method further optimize the recognition effect by introducing multi-scale feature learning and fine division of local features.

[0005] In addition, recent developments include the use of human skeleton points and posture prediction to calibrate feature extraction, which significantly improves the performance of pedestrian re-identification under occlusion and complex backgrounds. For example, the PGFA method uses key point information to generate an attention map, guides the model to focus on unoccluded areas, and removes irrelevant information during feature extraction, significantly improving the performance of the model under occlusion.

[0006] Although methods based on deep neural convolutional networks have made significant progress, some key issues have not been fully addressed and hinder the further promotion and productization of pedestrian re-identification technology.

[0007] First, there is a significant gap between existing person re-identification datasets and real-world application scenarios. Current datasets are usually based on images collected in a short period of time, and may only cover a few days of surveillance video, so pedestrians in the sample often do not experience significant changes in clothing. However, in practical applications, person re-identification systems are usually deployed in long-running camera networks, and the time interval between the same pedestrian being photographed is long, and changes in clothing and accessories are very common. In addition, changes in environmental factors such as lighting and weather further exacerbate the diversity of pedestrian appearance representations, thereby increasing the difficulty of recognition.

[0008] Secondly, although some methods such as the PUL method proposed by Hehe Fan et al. attempt to solve the data labeling problem through unsupervised learning, these methods do not fully consider the challenges brought by clothing changes. In addition, most existing pedestrian re-identification methods rely on the features of clothing and accessories for discrimination, which is not effective when dealing with drastic changes in clothing. This limits the performance of the model in practical applications, especially when dealing with complex clothing change scenes.

[0009] In summary, the key issues currently hindering the commercialization of pedestrian re-identification methods include: 1. The lack of labeled training data with a large number of pedestrian representation changes; 2. The lack of feature learning methods that are robust to pedestrian representation changes. Summary of the invention

[0010] In order to overcome the above shortcomings, the present invention provides a method for re-identifying pedestrians with changing clothes based on progressive guided learning of clothing semantics, so that more robust discriminative features can be learned in scenarios with rich changes in pedestrian appearance.

[0011] To achieve the above object, the present invention provides a method for re-identifying pedestrians who have changed clothes based on clothing semantics progressive guided learning, the method comprising:

[0012] (1) Obtain pedestrian images to be processed and divide them into a training set and a test set; use a human key point parsing tool to process the training set and generate a mask image for each pedestrian image to construct a mask image training set;

[0013] (2) Use the pre-trained ResNet50 model to extract pedestrian features from the original image and the training set mask in the training set;

[0014] (3) Initialize the category mean of pedestrians and calculate the arithmetic mean of all image features under each category as the category mean of the category;

[0015] (4) Through the progressive guided learning module, the pedestrian features extracted by the ResNet50 network are fused with the category mean of this type of pedestrians as query features. The similarity matrix between the query features and the masked pedestrian features of the training set is calculated as the positive sample features. At the same time, the similarity matrix between the query features and the category mean of this type of pedestrians is calculated to obtain the similarity difference matrix as the negative sample features. The positive and negative sample features are input into the contrastive learning paradigm for progressive guided learning of clothing semantics, and the progressive guided learning loss is calculated and the parameters of the model are updated;

[0016] (5) Through the clothing feature enhancement learning module, the pedestrian features extracted by the ResNet50 network are fused with the category mean of the pedestrians in each category to enhance the clothing feature representation ability of the pedestrian image in each category. The fused features are input into the clothing feature enhancement learning loss function, the clothing feature enhancement learning loss is calculated, and the model parameters are updated;

[0017] (6) Finally, the pedestrian features in the query dataset and the test dataset are extracted, the distance between the query image and the test image in the feature space is calculated, matching is performed to complete pedestrian re-identification, and the recognition performance is evaluated.

[0018] The beneficial effects of the present invention are:

[0019] 1. The present invention proposes a clothing semantics-driven progressive guided learning module, which allows the mask image to continuously approach the clothing-rich feature image in the feature space. In this process, the model learns more clothing-rich semantic features, thereby obtaining a more robust pedestrian feature description.

[0020] 2. The present invention proposes a clothing feature enhancement learning module, which enhances the clothing representation ability of pedestrian images in each category from the perspective of feature space, so that more robust discriminative features can be learned in scenarios with rich changes in pedestrian clothing.

[0021] 3. The clothing semantics-based progressive guided learning method for pedestrian re-identification after changing clothes of the present invention has achieved excellent results in relevant clothing-changing pedestrian re-identification datasets, with higher recognition performance and stability. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0023] Figure 1 It is a flow chart for realizing the present invention;

[0024] Figure 2 is a flow chart of training steps in a specific embodiment of the present invention;

[0025] Figure 3 The figure is a flow chart of the test steps of a specific embodiment of the present invention. DETAILED DESCRIPTION

[0026] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0027] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0028] The clothing semantics progressive guided learning network model proposed in the present invention has specific implementation steps as shown in the attached Figure 1 shown.

[0029] Step 1: Get an image and generate a mask image

[0030] Obtain pedestrian images to be processed and divide them into a training set and a test set; use a human key point parsing tool to process the training set and generate a mask image for each pedestrian image to construct a mask image training set.

[0031] Step 2: Extract pedestrian features

[0032] Use the pre-trained ResNet50 model to extract the pedestrian features x of the original image in the training set and the pedestrian features y of the mask in the training set.

[0033] Step 3: Initialize the category mean

[0034] Calculate the arithmetic mean of all image features in each category and initialize the category mean of the category. Category mean V i The formula is:

[0035]

[0036] Among them, |P i | is the number of all instance features in the i-th category extracted by the ResNet50 model in the training set, F(x i ) is the feature of the original graph under the i-th category in the training set extracted by the ResNet50 model, P i The ResNet50 model extracts all the features of the i-th category in the training set.

[0037] Step 4: Progressive guided learning

[0038] The pedestrian features extracted by the ResNet50 network are merged with the category mean of the pedestrians to form the query feature. i ) is expressed as:

[0039] F′(x i )=λF(x i )+(1-λ)V i (2)

[0040] Among them, λ is a fusion hyperparameter used to balance pedestrian features and category means.

[0041] Calculate the query feature F′(x i ) and the training set mask pedestrian feature F(y i ), and obtain the similarity matrix as the positive sample feature. The formula is expressed as:

[0042] R p =Sim(F′(x i ), F(y i )) (3)

[0043] Among them, R p is the similarity matrix between the query feature and the masked pedestrian feature of the training set, Sim(F′(x i ), F(y i )) is the query feature F′(x i ) and the training set mask pedestrian feature F(y i ) can be used to measure the similarity.

[0044] At the same time, the query feature F′(x i ) and the mean value V of this pedestrian category i The similarity matrix of is obtained, and the similarity difference matrix is ​​obtained as the negative sample feature. The formula is expressed as:

[0045] R n =Sim(F′(x i ), V i ) (4)

[0046] R d =|R n -R p | (5)

[0047] Among them, R n is the similarity matrix between the query feature and the mean of the pedestrian category, R d is the similarity difference matrix.

[0048] Finally, the positive and negative sample features are input into the contrastive learning paradigm for clothing semantics progressive guided learning, the clothing semantics driven progressive guided learning loss is calculated and the model parameters are updated. d The formula is:

[0049]

[0050] Among them, clothing diversification gradually guides learning L d The similarity difference matrix R can be made d Reduce and improve the query feature F′(x i ) and the training set mask pedestrian feature F(y i ) p , so that in the learning process R p will gradually approach R n , so that the mask pedestrian feature F(y i ) In the learning process, it keeps approaching the rich features of clothes V i , allowing the model to learn more clothing-rich semantic features in the feature space.

[0051] Step 5: Clothing feature enhancement learning

[0052] The pedestrian features extracted by the ResNet50 network and the category mean of this type of pedestrian are the query features F′(x i ), enhance the clothing feature representation ability of pedestrian images in each category, input the fused features into the clothing feature enhancement learning loss function, calculate the clothing feature enhancement learning loss and update the model parameters. The formula is expressed as:

[0053] F′(x i )=λF(x i )+(1-λ)V i (7)

[0054]

[0055] Among them, F′(x i ) represents the query feature, represents the clothing feature enhancement learning loss, yi represents the label of the i-th category, D represents the total number of categories, and η represents the classifier.

[0056] Step 6: Clothing-changing pedestrian re-identification and evaluation

[0057] Calculate the distance between the query image and the test image, perform matching for re-ID, and evaluate the model performance.

[0058] The simulation effect of the present invention is further described below in conjunction with simulation experiments:

[0059] 1. Simulation environment

[0060] The hardware test platform of the embodiment of the present invention is: NVIDIA GeForce RTX2080T GPU, memory is 12GB, and the software platform is: Linux system, Pytorch framework and Python3.6 platform.

[0061] 2. Simulation content

[0062] Taking the effect of the present invention as an example, three convincing public datasets PRCC, VC-Clothes and LTCC are used to compare with other cutting-edge pedestrian re-identification methods. The experiment uses ground-truth samples showing clothing changes (Clothes-changing) to calculate the accuracy. At the same time, in order to illustrate the experimental results more objectively, the present invention uses cumulative matching features (CMC) and mean average precision (mAP) to evaluate the performance. The present invention is also compared with ten most advanced clothing-changing pedestrian re-identification methods from different references (Reference) (such as RCSANet, FSAM, GI-ReID, M2Net, CAL, AIM, AFL, 3DInvarReID, MCSC and LIFTCAP) on public datasets to verify its performance.

[0063] The first dataset selected in this embodiment is the PRCC dataset, which is a popular real-life clothing-changing pedestrian re-identification (CC-ReID) dataset containing 33,698 images of 221 identities. PRCC is one of the pioneers of clothing-changing Re-ID and is widely used to evaluate state-of-the-art algorithms. However, the dataset is limited by the diversity of scenes (captured by only 3 cameras) and the diversity of clothes (each person only changes clothes twice). The second dataset is the VC-Clothes dataset, which is a synthetic CC-ReID dataset rendered by the GTA5 game engine, containing 19,060 images of 512 identities captured by 4 cameras, with each identity having 1 to 3 sets of clothes. The third dataset is the LTCC dataset, which is one of the latest CC-ReID datasets and contains 17,119 images of 152 identities. Although the number of images in the LTCC dataset is less than that of the PRCC dataset, its multiple scenes (captured by 12 cameras) and varied clothes (each person can change clothes up to 14 times) make it one of the most challenging CC-ReID datasets in recent years.

[0064] 3. Simulation results and analysis

[0065] Table 1 shows the comparison results of the PRCC dataset under the clothing-changing mode, among which the GI-ReID model is not compared with the PRCC dataset. Compared with other clothing-changing pedestrian re-identification methods, the CSPG model (Ours) proposed in this paper has a significantly higher recognition accuracy in the clothing-changing mode (Clothes-changing) than other advanced clothing-changing pedestrian re-identification methods. Its performance is 5.7% higher than the most advanced model, reaching 65% in Rank-1 and 62.5% in mAP.

[0066] Table 2 shows the comparison results of the VC-Clothes dataset under the clothing-changing mode. Among them, RCSANet, M2Net, CAL, AIM, 3DInvarReID, MCSC and LIFTCAP are not compared on the VC-Clothes dataset. Compared with other clothing-changing pedestrian re-identification methods, the CSPG model (Ours) proposed in this paper has a significantly higher recognition accuracy in the clothing-changing mode (Clothes-changing) than other advanced clothing-changing pedestrian re-identification methods.

[0067] Table 3 shows the comparison results of the LTCC dataset under the clothing-changing mode. RCSANet and M2Net have not been compared on the LTCC dataset. Compared with other clothing-changing pedestrian re-identification methods, the CSPG model (Ours) proposed in this paper has the highest recognition accuracy under the clothing-changing mode.

[0068] Table 1 Comparison results of PRCC dataset in clothing variation mode

[0069]

[0070] Table 2 Comparison results of VC-Clothes dataset in clothing variation mode

[0071]

[0072] Table 3 Comparison results of LTCC dataset under clothing change mode

[0073]

[0074] From the data results in the above three tables, the clothing semantics-based progressive guided learning method for pedestrian re-identification after changing clothes proposed in this invention outperforms the other ten methods in recognition effects on three public datasets. The method proposed in this invention effectively enhances the clothing representation capability of pedestrian images in each category, and can effectively learn progressive hierarchical clothing information.

[0075] The embodiments described above are only descriptions of the preferred embodiments of the present invention and are not intended to limit the scope of the present invention. Without departing from the design spirit of the present invention, various modifications and improvements made to the technical solutions of the present invention by ordinary technicians in this field should all fall within the protection scope determined by the claims of the present invention.

Claims

1. A method for re-identifying people who change clothes based on clothing semantics progressive guided learning, characterized in that: The following steps are involved: (1) Obtain pedestrian images to be processed and divide them into a training set and a test set; use a human key point parsing tool to process the training set and generate a mask image for each pedestrian image to construct a mask image training set; (2) Use the pre-trained ResNet50 model to extract pedestrian features from the original image and the training set mask in the training set; (3) Initialize the category mean of pedestrians and calculate the arithmetic mean of all image features under each category as the category mean of the category; (4) Through the progressive guided learning module, the pedestrian features extracted by the ResNet50 network are fused with the category mean of the pedestrian class as the query feature, and the similarity matrix between the query feature and the masked pedestrian features of the training set is calculated as the positive sample feature; At the same time, the similarity matrix between the query feature and the mean of the pedestrian category is calculated, and the similarity difference matrix is ​​obtained as the negative sample feature. The positive and negative sample features are input into the contrastive learning paradigm for clothing semantic progressive guided learning, and the progressive guided learning loss is calculated and the parameters of the model are updated. (5) Through the clothing feature enhancement learning module, the pedestrian features extracted by the ResNet50 network are fused with the category mean of the pedestrians in each category to enhance the clothing feature representation ability of the pedestrian image in each category. The fused features are input into the clothing feature enhancement learning loss function, the clothing feature enhancement learning loss is calculated, and the model parameters are updated; (6) Finally, the pedestrian features in the query dataset and the test dataset are extracted, the distance between the query image and the test image in the feature space is calculated, matching is performed to complete pedestrian re-identification, and the recognition performance is evaluated.

2. The method according to claim 1, characterized in that: Step (3) The method for initializing the pedestrian category mean is as follows: Calculate the arithmetic mean of all image features in each category and use it as the category mean of the category. The category mean V i The formula is expressed as: Among them, |P i | is the number of all instance features in the i-th category extracted by the ResNet50 model in the training set, F(x i ) is the feature of the original graph under the i-th category in the training set extracted by the ResNet50 model, P i The ResNet50 model extracts all the features of the i-th category in the training set.

3. The method according to claim 1, characterized in that The training method of the progressive guided learning module in step (4) is as follows: The pedestrian features extracted by the ResNet50 network are fused with the category mean of the pedestrians to form the query feature. i ) is expressed as: F′(x i )=λF(x i )+(1-λ)V i (2) where λ is a fusion hyperparameter used to balance pedestrian features and category means; Calculate the query feature F′(x i ) and the training set mask pedestrian feature F(y i ), and the similarity matrix is ​​obtained as the positive sample feature, and the formula is expressed as: R p =Sim(F′(x i ),F(y i )) (3) Among them, R p is the similarity matrix between the query feature and the masked pedestrian feature of the training set, Sim(F′(x i ), F(y i )) is the query feature F′(x i ) and the training set mask pedestrian feature F(y i ) is used to calculate the inner product of , which is used to measure the similarity; At the same time, the query feature F′(x i ) and the mean value V of this pedestrian category i The similarity matrix of , the similarity difference matrix is ​​obtained as the negative sample feature, and the formula is expressed as: R n =Sim(F′(x i ),V i ) (4) R d =|R n -R p | (5) Among them, R n is the similarity matrix between the query feature and the mean of the pedestrian category, R d is the similarity difference matrix; Finally, the positive and negative sample features are input into the contrastive learning paradigm for clothing semantics progressive guided learning, the clothing semantics driven progressive guided learning loss is calculated and the parameters of the model are updated.

4. The method according to claim 1, characterized in that: The training method of the clothing feature enhancement learning module in step (5) is as follows: The pedestrian features extracted by the ResNet50 network are combined with the category mean of the pedestrian to form the query feature F′(x i ), enhance the clothing feature representation ability of pedestrian images in each category, input the fused features into the clothing feature enhancement learning loss function, calculate the clothing feature enhancement learning loss and update the model parameters, clothing feature enhancement learning loss The formula is: Among them, y i represents the label of the i-th category, D represents the total number of categories, and η represents the classifier.

Images