A method for re-identification of people with changed clothes based on clothing-aware hybridization and foreground enhancement

Abstract

The present invention belongs to the technical field of computer vision and pattern recognition, and in particular, relates to a method for re-identifying a pedestrian who is changing clothes based on clothing perception mixing and foreground enhancement. The steps are as follows: collect image data to construct a data set, and perform text description on the collected images, and divide the image data in the data set into a training set and a test set after pre-processing; construct a re-identification model for pedestrians who are changing clothes, which includes a text feature extraction unit, an image feature extraction unit, and a feature fusion unit, and input the pedestrian images in the training set into the re-identification model for pedestrians who are changing clothes for training, and obtain the loss function of each unit; summarize the losses to obtain the total loss, and train and optimize the re-identification model for pedestrians who are changing clothes by the total loss; input the data in the test set into the optimized and trained re-identification model for pedestrians who are changing clothes for similarity comparison, and select the image with the highest similarity as the final retrieval result; the present invention can improve the accuracy of re-identification of pedestrians who are changing clothes.

Description

Technical Field

[0001] The invention belongs to the technical field of computer vision and pattern recognition, and relates to a method for re-identifying pedestrians who change clothes based on clothing perception mixing and foreground enhancement. Background Art

[0002] Person Re-identification (Re-ID) refers to identifying pedestrians of the same identity in surveillance videos at different times and locations through specific pedestrian images. This process faces challenges such as perspective differences, lighting changes, and occlusions, making it an important topic in the field of computer vision. In recent years, with the popularization of surveillance videos, the demand for pedestrian re-identification technology in security, human-computer interaction, and intelligent transportation has increased, significantly improving the quality of life.

[0003] Pedestrian re-identification after changing clothes takes into account the factors of pedestrian clothing changes based on traditional pedestrian re-identification. Since people's appearance in reality changes with weather, occasions, etc., their clothing will also change accordingly. This is particularly important in terms of social security. Especially in crime scenes, suspects may evade surveillance by changing clothes and obscuring information, increasing the difficulty of arrest. Although pedestrian re-identification technology has made progress in recent years, it is not effective when dealing with clothing changes. This problem has attracted the attention of researchers and promoted the research on pedestrian re-identification after changing clothes to improve its application value.

[0004] At present, existing methods mostly focus on increasing clothing diversity, decoupling identity features from clothing features, or extracting stable biometric features of the human body (such as head, body contour and gait, etc.). However, these methods are still easily affected by clothing changes, and the biometric feature extraction is relatively single. In order to solve the above problems, the present invention proposes a clothing-changing pedestrian re-identification method based on clothing-aware mixing and foreground enhancement, aiming to improve the recognition performance under clothing changes. Summary of the invention

[0005] The present invention proposes a method for pedestrian re-identification in clothing-changing situations based on clothing-aware mixing and foreground enhancement. The purpose is to solve the problem of pedestrian re-identification in clothing-changing situations. By deepening the understanding of visual information and optimizing the extraction strategy, the method can achieve more reliable pedestrian re-identification in complex practical scenarios.

[0006] A clothing-changing pedestrian re-identification method based on clothing-aware mixing and foreground enhancement is as follows:

[0007] S1, collect image data to build a data set, and give a text description of the collected images, then preprocess the image data in the data set, and finally divide the data set into a training set and a test set;

[0008] S2. Construct a clothing-changing pedestrian re-identification model, which includes a text feature extraction unit, an image feature extraction unit, and a feature fusion unit. Input pedestrian images in the training set into the clothing-changing pedestrian re-identification model for training to obtain the loss function of each unit.

[0009] Text feature extraction unit: introduces learnable text tags with specific IDs to learn ambiguous text descriptions. The text extraction unit includes an image encoder and a text encoder.

[0010] Image feature extraction unit: including locating the center area of ​​the top to expand the cloth content of the clothes, randomly sampling clothes to fill the pedestrian's top part, and backbone network Vision Transformer feature extraction. The center area of ​​the top is located to expand the cloth content of the clothes. Specifically, the information of the clothing style is obtained from the images in the input training set. Multiple key points in the human body are extracted through the auxiliary tool HRNet human posture estimation network. The clothes are randomly sampled to fill the pedestrian's top part. Specifically, the SCHP human body parsing network is used to decompose the images in the input data set to generate pseudo labels belonging to different attributes. After the backbone network Vision Transformer extracts the features, the cross entropy loss between the text and the image is calculated based on the extracted features;

[0011] Feature fusion unit: The feature fusion unit extracts foreground features by fusing the ResNet network with the backbone network;

[0012] S3, sum up the losses to get the total loss, and use the total loss to train and optimize the clothing-changing pedestrian re-identification model;

[0013] S4. Input the data in the test set into the optimized and trained clothing-changing pedestrian re-identification model. The optimized clothing-changing pedestrian re-identification model is used to extract the features of the input pedestrian image. The extracted features are searched and compared with the data in the cloud database. The image with the highest similarity is the final search result.

[0014] S1 is as follows:

[0015] Capture pedestrian images in different scenes and from different perspectives through the camera, and provide text descriptions for the collected images;

[0016] Preprocess the image data in the dataset, crop, scale and normalize the collected images to make the images in the dataset have a uniform size and format;

[0017] Finally, the dataset is divided into training set and test set according to the ratio.

[0018] S2 is as follows:

[0019] The data in the training set is input into the clothing-changing pedestrian re-identification model, the text description of the image is input into the text feature extraction unit to obtain text features, the image is input into the image feature extraction unit to obtain image features, and then the text features and image features are input into the feature fusion unit to obtain enhanced image features. The specific process is as follows:

[0020] S2.1, text feature extraction unit:

[0021] Learnable text tags for specific IDs are introduced to learn ambiguous text descriptions. Each ID is independent. The text extraction unit includes an image encoder and a text encoder. The text description of the image is input into the text editor to obtain the following description: "A picture Photos of people”, Indicates learnable text tags, represents the number of learnable text tags, ;

[0022] Freeze the parameters of the image encoder and text encoder, and use contrastive losses from image to text and text to image to learn text tags. For optimization, the calculation formula is as follows:

[0023] ,

[0024] ,

[0025] ,

[0026] ,

[0027] in, represents any image in the input training set, Represents the text description corresponding to any image in the input training set, represents the operation of linearly projecting the image into the cross-modal embedding space, express The image features obtained after the operation are represents the linear projection embedding of text representation into the cross-modal embedding space, express The text features obtained after the operation, Represents the similarity between image and text features in feature space; Indicates that the identity tag is The image features, Indicates that the identity tag is The text features of Indicates that the identity tag is The similarity between image features and text features in feature space, represents the batch size, Indicates batch indexing, Indicates that the identity tag is The image-to-text contrast loss is express Batch image features, express Batch text features, , , express Batch The similarity between image features and text features in feature space; Indicates that the identity tag is The contrastive loss of text to image is Indicated in batches and target text features The set of all matching positive samples, Representing a collection The elements in image features representing the matched elements, express and The similarity in feature space is express and Similarity in feature space; Represents the text feature loss function.

[0028] S2.2, Image feature extraction unit:

[0029] S2.2.1. Locate the center area of ​​the top and expand the clothing fabric content:

[0030] The information of clothing styles is obtained from the images in the input training set. Multiple key points of the human body are extracted through the auxiliary tool HRNet human posture estimation network. Two key points, the left shoulder and the right crotch, are selected from the extracted key points. The midpoint of the line connecting the two key points is used as the center point of the clothing to intercept the cloth with the same size. The size is set to 32 pixels in width and 64 pixels in height. Then, the intercepted cloth image is pixel-copied and expanded to retain the original texture information of the clothing, and an image set of pure clothing fabric is obtained.

[0031] S2.2.2. Randomly sample clothes to fill the pedestrian's top:

[0032] Use the SCHP human body parsing network to decompose the images in the input dataset, generate pseudo labels belonging to different attributes, set the top label, and randomly sample the fabric content in the image set of pure clothing fabrics to fill the top label area, thereby realizing the exchange of clothes between pedestrians in the input image and obtaining the image after the clothing exchange process;

[0033] S2.2.3. Backbone network Vision Transformer feature extraction:

[0034] The pre-trained VisionTransformer visual converter is used to extract features from the images in the input dataset and the images after the clothing change process. The original features extracted are and changing clothes feature , the two extracted features are respectively combined with the text features extracted by the text feature extraction unit The cross entropy loss function from image to text is calculated to constrain the original features and clothing-changing features to be close to the corresponding text features. The calculation formula is as follows:

[0035] ,

[0036] ,

[0037] ,

[0038] in, represents the number of identity categories, The first level index representing the number of identity categories, The second level index representing the number of identity categories, Indicates The target distribution value for identity categories, Indicates The text features of identity categories, Indicates The text features of identity categories, express and The similarity in feature space is express and The similarity in feature space is Represents the original feature image-to-text cross entropy loss, Indicates the characteristics of changing clothes image-to-text cross entropy loss, express and The similarity in feature space is express and The similarity in feature space is represents the batch size, Indicates batch indexing, represents the consistency loss, Indicates batches of original features, Indicates The changing characteristics of each batch.

[0039] S2.3, feature fusion unit:

[0040] The ResNet50 residual network pre-trained on the ImageNet network is introduced to supplement the details ignored by the VisionTransformer visual converter in the feature extraction process. First, the images in the input training set are preprocessed to shield the interference factors in the clothes and background in the image. Specifically, the pixels belonging to the background label in the SCHP human body analysis results are set to 255, and the pixels of the top label are set to 0; then, feature information is exchanged between the ResNet50 residual network and the VisionTransformer visual converter, and the outputs of the Vision Transformer visual converter are selected and transmitted to the ResNet network respectively. Specifically, the output of the Vision Transformer visual converter is deformed first, and then adjusted to the same 4D shape as the output format of the ResNet network through convolution, batch normalization, relu activation and linear interpolation. At the same time, channel attention and spatial attention are introduced in each layer of the ResNet network to focus on the key information area, and then the deformed Vision The output of the Transformer visual converter is concatenated with the features extracted by ResNet on the channel, and finally the final foreground features are obtained through convolution, batch normalization and relu activation. The calculation process is as follows:

[0041] ,

[0042] ,

[0043] ,

[0044] ,

[0045] ,

[0046] ,

[0047] in, Represents the characteristics of the ResNet network output, represents the height of the feature map, represents the width of the feature map, The index representing the height, The index representing the width, represents the operation of the fully connected layer, Indicates the number of channels of the feature map, represents the index of the channel, represents the convolution operation, Represents the operation of the relu activation function, represents the channel attention weight, represents the spatial attention weight, represents multiplication, Represents the output of the ResNet network through the attention mechanism, represents a linear interpolation operation, represents the batch normalization operation, Represents a splicing operation, represents the features after splicing, Represents the foreground features, represents the fusion feature loss, Indicates batches of foreground features, Indicates The original features of the batches.

[0048] S3 is as follows:

[0049] Calculate total loss , the clothing-changing pedestrian re-identification model is trained and optimized according to the total loss. The calculation process is as follows:

[0050] ,

[0051] in represents the cross entropy loss, represents triple loss;

[0052] The loss function is as follows:

[0053] ,

[0054] in, Indicates The original features of the batches, Indicates The characteristics of the changing clothes of each batch, Indicates batches of foreground features, Indicates batches of identity tags;

[0055] The loss function is as follows:

[0056] ,

[0057] Where B is the size of the training batch, represents the boundary parameter of the triplet loss, and Indicates The positive sample features and negative sample features of batches of original features, and Indicates The positive and negative sample features of batches of clothing change features, and Indicates The positive and negative sample features of batches of Qianjin features, Indicates taking the maximum value, Represents the calculation of Euclidean distance.

[0058] Step S4 is specifically as follows:

[0059] The data in the test set is input into the trained and optimized clothing-changing pedestrian re-identification model. The features of the input pedestrian image are extracted through the optimized clothing-changing pedestrian re-identification model. The similarity between the extracted features and the features in the cloud database is calculated one by one, and the scores are given according to the similarity scores. The image with the highest similarity score is selected as the final retrieval result.

[0060] The effects provided in the content of the invention are only the effects of the embodiments, rather than all the effects of the invention. The above technical solution has the following advantages or beneficial effects:

[0061] The present invention proposes a method for re-identifying pedestrians who are changing clothes based on clothing perception mixing and foreground enhancement. The present invention simulates the clothing-changing behavior of pedestrians on the basis of retaining the clothing style and texture to confuse the model's learning of clothes, and gets rid of the traditional method of erasing or randomly sampling clothing pixels; a single network processing is difficult to achieve the goal of fine feature extraction, and the present invention extracts the foreground features of the clothing-erased image by fusing the Vision Transformer visual converter and the ResNet network, which can complement the defects between networks and enrich the semantic expression of the features; the present invention extracts pure clothing fabrics for exchange to simulate the clothing-changing behavior, and uses a multi-network fusion method to enhance the foreground features, enriching the semantic expression of the features and improving the recognizability of the features for pedestrian identity recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0062] The accompanying drawings are used to provide further understanding of the present invention and constitute a part of the specification. They are used to explain the present invention together with the embodiments of the present invention and do not constitute a limitation of the present invention.

[0063] Figure 1 The present invention is a flow chart of the method. DETAILED DESCRIPTION

[0064] In order to clearly illustrate the technical features of the present invention, the present invention is described in detail below through specific implementation methods and in conjunction with the accompanying drawings.

[0065] Example 1

[0066] like Figure 1 As shown in the figure, a method for re-identification of pedestrians with changed clothes based on clothing-aware mixing and foreground enhancement is as follows:

[0067] S1, collect image data to build a data set, and give a text description of the collected images, then preprocess the image data in the data set, and finally divide the data set into a training set and a test set;

[0068] S2. Construct a clothing-changing pedestrian re-identification model, which includes a text feature extraction unit, an image feature extraction unit, and a feature fusion unit. Input pedestrian images in the training set into the clothing-changing pedestrian re-identification model for training to obtain the loss function of each unit.

[0069] Text feature extraction unit: introduces learnable text tags with specific IDs to learn ambiguous text descriptions. The text extraction unit includes an image encoder and a text encoder.

[0070] Image feature extraction unit: including locating the center area of ​​the top to expand the cloth content of the clothes, randomly sampling clothes to fill the pedestrian's top part, and backbone network Vision Transformer feature extraction. The center area of ​​the top is located to expand the cloth content of the clothes. Specifically, the information of the clothing style is obtained from the images in the input training set. Multiple key points in the human body are extracted through the auxiliary tool HRNet human posture estimation network. The clothes are randomly sampled to fill the pedestrian's top part. Specifically, the SCHP human body parsing network is used to decompose the images in the input data set to generate pseudo labels belonging to different attributes. After the backbone network Vision Transformer extracts the features, the cross entropy loss between the text and the image is calculated based on the extracted features;

[0071] Feature fusion unit: The feature fusion unit extracts foreground features by fusing the ResNet network with the backbone network;

[0072] S3, sum up the losses to get the total loss, and use the total loss to train and optimize the clothing-changing pedestrian re-identification model;

[0073] S4. Input the data in the test set into the optimized and trained clothing-changing pedestrian re-identification model. The optimized clothing-changing pedestrian re-identification model is used to extract the features of the input pedestrian image. The extracted features are searched and compared with the data in the cloud database. The image with the highest similarity is the final search result.

[0074] S1 is as follows:

[0075] Capture pedestrian images in different scenes and from different perspectives through the camera, and provide text descriptions for the collected images;

[0076] Preprocess the image data in the dataset, crop, scale and normalize the collected images to make the images in the dataset have a uniform size and format;

[0077] Finally, the dataset is divided into training set and test set according to the ratio.

[0078] S2 is as follows:

[0079] The data in the training set is input into the clothing-changing pedestrian re-identification model, the text description of the image is input into the text feature extraction unit to obtain text features, the image is input into the image feature extraction unit to obtain image features, and then the text features and image features are input into the feature fusion unit to obtain enhanced image features. The specific process is as follows:

[0080] S2.1, text feature extraction unit:

[0081] Learnable text tags for specific IDs are introduced to learn ambiguous text descriptions. Each ID is independent. The text extraction unit includes an image encoder and a text encoder. The text description of the image is input into the text editor to obtain the following description: "A picture Photos of people”, Indicates learnable text tags, represents the number of learnable text tags, ;

[0082] Freeze the parameters of the image encoder and text encoder, and use contrastive losses from image to text and text to image to learn text tags. For optimization, the calculation formula is as follows:

[0083] ,

[0084] ,

[0085] ,

[0086] ,

[0087] in, represents any image in the input training set, Represents the text description corresponding to any image in the input training set, represents the operation of linearly projecting the image into the cross-modal embedding space, express The image features obtained after the operation are represents the linear projection embedding of text representation into the cross-modal embedding space, express The text features obtained after the operation, Represents the similarity between image and text features in feature space; Indicates that the identity tag is The image features, Indicates that the identity tag is The text features of Indicates that the identity tag is The similarity between image features and text features in feature space, represents the batch size, Indicates batch indexing, Indicates that the identity tag is The image-to-text contrast loss is express Batch image features, express Batch text features, , , express Batch The similarity between image features and text features in feature space; Indicates that the identity tag is The contrastive loss of text to image is Indicated in batches and target text features The set of all matching positive samples, Representing a collection The elements in image features representing the matched elements, express and The similarity in feature space is express and Similarity in feature space; Represents the text feature loss function.

[0088] S2.2, Image feature extraction unit:

[0089] S2.2.1. Locate the center area of ​​the top and expand the clothing fabric content:

[0090] The information of clothing styles is obtained from the images in the input training set. Multiple key points of the human body are extracted through the auxiliary tool HRNet human posture estimation network. Two key points, the left shoulder and the right crotch, are selected from the extracted key points. The midpoint of the line connecting the two key points is used as the center point of the clothing to intercept the cloth with the same size. The size is set to 32 pixels in width and 64 pixels in height. Then, the intercepted cloth image is pixel-copied and expanded to retain the original texture information of the clothing, and an image set of pure clothing fabric is obtained.

[0091] S2.2.2. Randomly sample clothes to fill the pedestrian's top:

[0092] Use the SCHP human body parsing network to decompose the images in the input dataset, generate pseudo labels belonging to different attributes, set the top label, and randomly sample the fabric content in the image set of pure clothing fabrics to fill the top label area, thereby realizing the exchange of clothes between pedestrians in the input image and obtaining the image after the clothing exchange process;

[0093] S2.2.3. Backbone network Vision Transformer feature extraction:

[0094] The pre-trained VisionTransformer visual converter is used to extract features from the images in the input dataset and the images after the clothing change process. The original features extracted are and changing clothes feature , the two extracted features are respectively combined with the text features extracted by the text feature extraction unit The cross entropy loss function from image to text is calculated to constrain the original features and clothing-changing features to be close to the corresponding text features. The calculation formula is as follows:

[0095] ,

[0096] ,

[0097] ,

[0098] in, represents the number of identity categories, The first level index representing the number of identity categories, The second level index representing the number of identity categories, Indicates The target distribution value for identity categories, Indicates The text features of identity categories, Indicates The text features of identity categories, express and The similarity in feature space is express and The similarity in feature space is Represents the original feature image-to-text cross entropy loss, Indicates the characteristics of changing clothes image-to-text cross entropy loss, express and The similarity in feature space is express and The similarity in feature space is represents the batch size, Indicates batch indexing, represents the consistency loss, Indicates batches of original features, Indicates The changing characteristics of each batch.

[0099] S2.3, feature fusion unit:

[0100] The ResNet50 residual network pre-trained on the ImageNet network is introduced to supplement the details ignored by the VisionTransformer visual converter in the feature extraction process. First, the images in the input training set are preprocessed to shield the interference factors in the clothes and background in the image. Specifically, the pixels belonging to the background label in the SCHP human body analysis results are set to 255, and the pixels of the top label are set to 0; then, feature information is exchanged between the ResNet50 residual network and the VisionTransformer visual converter, and the outputs of the Vision Transformer visual converter are selected and transmitted to the ResNet network respectively. Specifically, the output of the Vision Transformer visual converter is deformed first, and then adjusted to the same 4D shape as the output format of the ResNet network through convolution, batch normalization, relu activation and linear interpolation. At the same time, channel attention and spatial attention are introduced in each layer of the ResNet network to focus on the key information area, and then the deformed Vision The output of the Transformer visual converter is concatenated with the features extracted by ResNet on the channel, and finally the final foreground features are obtained through convolution, batch normalization and relu activation. The calculation process is as follows:

[0101] ,

[0102] ,

[0103] ,

[0104] ,

[0105] ,

[0106] ,

[0107] in, Represents the characteristics of the ResNet network output, represents the height of the feature map, represents the width of the feature map, The index representing the height, The index representing the width, represents the operation of the fully connected layer, Indicates the number of channels of the feature map, represents the index of the channel, represents the convolution operation, Represents the operation of the relu activation function, represents the channel attention weight, represents the spatial attention weight, represents multiplication, Represents the output of the ResNet network through the attention mechanism, represents a linear interpolation operation, represents the batch normalization operation, Represents a splicing operation, represents the features after splicing, Represents the foreground features, represents the fusion feature loss, Indicates batches of foreground features, Indicates The original features of the batches.

[0108] S3 is as follows:

[0109] Calculate total loss , the clothing-changing pedestrian re-identification model is trained and optimized according to the total loss. The calculation process is as follows:

[0110] ,

[0111] in represents the cross entropy loss, represents triple loss;

[0112] The loss function is as follows:

[0113] ,

[0114] in, Indicates The original features of the batches, Indicates The characteristics of the changing clothes of each batch, Indicates batches of foreground features, Indicates batches of identity tags;

[0115] The loss function is as follows:

[0116] ,

[0117] Where B is the size of the training batch, represents the boundary parameter of the triplet loss, and Indicates The positive sample features and negative sample features of batches of original features, and Indicates The positive and negative sample features of batches of clothing change features, and Indicates The positive and negative sample features of batches of Qianjin features, Indicates taking the maximum value, Represents the calculation of Euclidean distance.

[0118] Step S4 is specifically as follows:

[0119] The data in the test set is input into the trained and optimized clothing-changing pedestrian re-identification model. The features of the input pedestrian image are extracted through the optimized clothing-changing pedestrian re-identification model. The similarity between the extracted features and the features in the cloud database is calculated one by one, and the scores are given according to the similarity scores. The image with the highest similarity score is selected as the final retrieval result.

[0120] Example 2

[0121] The experimental results of this invention are compared with other methods (ViT-B / 16 comes from "Transformers for Large-Scale Image Recognition", Clip-Reid comes from "Image Re-identification without Specific Text Labels Using Visual Language Models", LaST comes from "Large-Scale Spatiotemporal Person Re-identification", Pixel Sampling comes from "Semantic-guided pixel sampling for clothing-changing pedestrian re-identification", AFD-Net comes from "t-adversarial feature disentanglement for long-term person re-identification", CCFA comes from "Clothing change feature enhancement for person re-identification", AFL comes from "A general framework for clothing-changing re-identification through association and forgetting learning", PGAL comes from "Posture-guided attention learning for clothing-changing person re-identification", ACID comes from "Causality inspires cloth decontamination for clothing-changing person re-identification", AIM comes from "Assisted free clothing-changing person re-identification, single-image clothing-changing person re-identification based on gait prediction and regularization", GI-ReID comes from "Single-image clothing-changing pedestrian re-identification based on gait prediction and regularization", SCNe comes from "Application of semantic-aware consistency network in clothing-changing pedestrian re-identification", and 3DinvarReID comes from "Learning clothing and posture-invariant 3D shape representation for long-term pedestrian re-identification") The results of the four public clothing-changing datasets are compared as shown in Table 1. On the CC (PRCC is the dataset proposed in "Pedestrian Re-identification through Contour Sketches with Moderate Clothing Change") dataset, the present invention achieved 63.1% on mAP and 68.4% on rank-1; on the LTCC (LTCC is the dataset proposed in "Pedestrian Re-identification with Long-term Clothing Change") dataset, the present invention achieved the best effect, with 37.6% on mAP and 76.1% on rank-1; on the NKUP (NKUP is the dataset proposed in "Benchmark for Clothing Changes in Personal Re-identification") dataset, the present invention also achieved the best effect, with 19.7% on mAP and 33.3% on rank-1; similarly, it also has good results on Celeb-reID-light (Celeb-reID-light is the dataset proposed in "Benchmark for Clothing Changes in Long-term Pedestrian Re-identification"), with 47.5% on mAP and 67.6% on rank-1. mAP represents an evaluation index for measuring the performance of target detection algorithms, and rank-1 refers to the probability that the model can correctly identify the target object in the first prediction in face recognition or image classification tasks. From the results in Table 1, it can be seen that the data of the present invention on the four data sets are better than other existing methods in terms of the two evaluation indicators of mAP and rank-1, which can prove that the present invention can achieve more reliable pedestrian re-identification in complex actual scenarios.

[0122] Table 1 Comparison results between the existing method and the proposed method on four data sets

[0123]

[0124] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit the present invention. Although the present invention is described in detail with reference to the embodiments, it should be understood by those skilled in the art that any modification or equivalent replacement of the technical solutions of the present invention does not depart from the spirit and scope of the technical solutions of the present invention and should be included in the scope of the claims of the present invention.

Claims

1. A method for re-identifying people with changing clothes based on clothing-aware mixing and foreground enhancement, characterized in that: The following steps are involved: S1, collect image data to build a data set, and give a text description of the collected images, then preprocess the image data in the data set, and finally divide the data set into a training set and a test set; S2. Construct a clothing-changing pedestrian re-identification model, which includes a text feature extraction unit, an image feature extraction unit, and a feature fusion unit. Input pedestrian images in the training set into the clothing-changing pedestrian re-identification model for training to obtain the loss function of each unit. Text feature extraction unit: introduces learnable text tags with specific IDs to learn ambiguous text descriptions. The text extraction unit includes an image encoder and a text encoder. Image feature extraction unit: including locating the center area of ​​the top to expand the cloth content of the clothes, randomly sampling clothes to fill the pedestrian's top part, and backbone network Vision Transformer feature extraction. The center area of ​​the top is located to expand the cloth content of the clothes. Specifically, the information of the clothing style is obtained from the images in the input training set. The auxiliary tool HRNet human posture estimation network is used to extract multiple key points in the human body. The clothes are randomly sampled to fill the pedestrian's top part. Specifically, the SCHP human body parsing network is used to decompose the images in the input data set to generate pseudo labels belonging to different attributes. After the backbone network Vision Transformer extracts the features, the cross entropy loss between the text and the image is calculated based on the extracted features. Feature Fusion Unit : The feature fusion unit extracts foreground features by fusing the ResNet network with the backbone network; S3, sum up the losses to get the total loss, and use the total loss to train and optimize the clothing-changing pedestrian re-identification model; S4. Input the data in the test set into the optimized and trained clothing-changing pedestrian re-identification model. The optimized clothing-changing pedestrian re-identification model is used to extract the features of the input pedestrian image. The extracted features are searched and compared with the data in the cloud database. The image with the highest similarity is the final search result.

2. The method for re-identifying people changing clothes based on clothing-aware mixing and foreground enhancement according to claim 1, characterized in that: S1 is as follows: Capture pedestrian images in different scenes and from different perspectives through the camera, and provide text descriptions for the collected images; Preprocess the image data in the dataset, crop, scale and normalize the collected images to make the images in the dataset have a uniform size and format; Finally, the dataset is divided into training set and test set according to the ratio.

3. The method for re-identifying people changing clothes based on clothing-aware mixing and foreground enhancement according to claim 2, characterized in that: S2 is as follows: The data in the training set is input into the clothing-changing pedestrian re-identification model, the text description of the image is input into the text feature extraction unit to obtain text features, the image is input into the image feature extraction unit to obtain image features, and then the text features and image features are input into the feature fusion unit to obtain enhanced image features. The specific process is as follows: S2.1, text feature extraction unit: Learnable text tags for specific IDs are introduced to learn ambiguous text descriptions. Each ID is independent. The text extraction unit includes an image encoder and a text encoder. The text description of the image is input into the text editor to obtain the following description: "A Photos of people”, Indicates learnable text tags, represents the number of learnable text tags, ; Freeze the parameters of the image encoder and text encoder, and use contrastive losses from image to text and text to image to learn text tags. For optimization, the calculation formula is as follows: ， ， ， ， in, represents any image in the input training set, Represents the text description corresponding to any image in the input training set, represents the operation of linearly projecting the image into the cross-modal embedding space, express The image features obtained after the operation are represents the linear projection embedding of text representation into the cross-modal embedding space, express The text features obtained after the operation, Represents the similarity between image and text features in feature space; Indicates that the identity tag is The image features, Indicates that the identity tag is The text features of Indicates that the identity tag is The similarity between image features and text features in feature space, represents the batch size, Indicates batch indexing, Indicates that the identity tag is The image-to-text contrast loss is express Batch image features, express Batch text features, , , express Batch The similarity between image features and text features in feature space; Indicates that the identity tag is The contrastive loss of text to image is Indicated in batches and target text features The set of all matching positive samples, Representing a collection The elements in image features representing the matched elements, express and The similarity in feature space is express and Similarity in feature space; Represents the text feature loss function.

4. The method for re-identifying pedestrians changing clothes based on clothing-aware mixing and foreground enhancement according to claim 3, characterized in that: S2.2, Image feature extraction unit: S2.2.

1. Locate the center area of ​​the top and expand the clothing fabric content: The information of clothing styles is obtained from the images in the input training set. Multiple key points of the human body are extracted through the auxiliary tool HRNet human posture estimation network. Two key points, the left shoulder and the right hip, are selected from the extracted key points. The midpoint of the line connecting the two key points is used as the center point of the clothing to intercept the cloth with the same size. The size is set to 32 pixels in width and 64 pixels in height. Then, the intercepted cloth image is pixel-copied and expanded to retain the original texture information of the clothing, and an image set of pure clothing fabric is obtained. S2.2.

2. Randomly sample clothes to fill the pedestrian's top: Use the SCHP human body parsing network to decompose the images in the input dataset, generate pseudo labels belonging to different attributes, set the top label, and randomly sample the fabric content in the image set of pure clothing fabrics to fill the top label area, thereby realizing the exchange of clothes between pedestrians in the input image and obtaining the image after the clothing exchange process; S2.2.

3. Backbone network Vision Transformer feature extraction: The pre-trained VisionTransformer visual converter is used to extract features from the images in the input dataset and the images after the clothing change process. The original features extracted are and changing clothes feature , the two extracted features are respectively combined with the text features extracted by the text feature extraction unit The cross entropy loss function from image to text is calculated to constrain the original features and clothing-changing features to be close to the corresponding text features. The calculation formula is as follows: ， ， ， in, represents the number of identity categories, The first level index representing the number of identity categories, The second level index representing the number of identity categories, Indicates The target distribution value for identity categories, Indicates The text features of identity categories, Indicates The text features of identity categories, express and The similarity in feature space is express and The similarity in feature space is Represents the original feature image-to-text cross entropy loss, Indicates the characteristics of changing clothes image-to-text cross entropy loss, express and The similarity in feature space is express and The similarity in feature space is represents the batch size, Indicates batch indexing, represents the consistency loss, Indicates batches of original features, Indicates The changing characteristics of each batch.

5. The method for re-identifying pedestrians changing clothes based on clothing-aware mixing and foreground enhancement according to claim 4, characterized in that: S2.3, feature fusion unit: The ResNet50 residual network pre-trained on the ImageNet network is introduced to supplement the details ignored by the VisionTransformer visual converter in the feature extraction process. First, the images in the input training set are preprocessed to shield the interference factors in the clothes and background in the image. Specifically, the pixels belonging to the background label in the SCHP human body analysis results are set to 255, and the pixels of the top label are set to 0; then, feature information is exchanged between the ResNet50 residual network and the VisionTransformer visual converter, and the outputs of the Vision Transformer visual converter are selected and transmitted to the ResNet network respectively. Specifically, the output of the Vision Transformer visual converter is deformed first, and then adjusted to the same 4D shape as the output format of the ResNet network through convolution, batch normalization, relu activation and linear interpolation. At the same time, channel attention and spatial attention are introduced in each layer of the ResNet network to focus on the key information area, and then the deformed Vision The output of the Transformer visual converter is concatenated with the features extracted by ResNet on the channel, and finally the final foreground features are obtained through convolution, batch normalization and relu activation. The calculation process is as follows: ， ， ， ， ， ， in, Represents the characteristics of the ResNet network output, represents the height of the feature map, represents the width of the feature map, The index representing the height, The index representing the width, represents the operation of the fully connected layer, Indicates the number of channels of the feature map, represents the index of the channel, represents the convolution operation, Represents the operation of the relu activation function, represents the channel attention weight, represents the spatial attention weight, represents multiplication, Represents the output of the ResNet network through the attention mechanism, represents a linear interpolation operation, represents the batch normalization operation, Represents a splicing operation, represents the features after splicing, Represents the foreground features, represents the fusion feature loss, Indicates batches of foreground features, Indicates The original features of the batches.

6. The method for re-identifying people changing clothes based on clothing-aware mixing and foreground enhancement according to claim 5, characterized in that: S3 is as follows: Calculate total loss , the clothing-changing pedestrian re-identification model is trained and optimized according to the total loss. The calculation process is as follows: ， in represents the cross entropy loss, represents triple loss; The loss function is as follows: ， in, Indicates The original features of the batches, Indicates The characteristics of the changing clothes of each batch, Indicates batches of foreground features, Indicates batches of identity tags; The loss function is as follows: ， Where B is the size of the training batch, represents the boundary parameter of the triplet loss, and Indicates The positive sample features and negative sample features of batches of original features, and Indicates The positive and negative sample features of batches of clothing change features, and Indicates The positive and negative features of a batch of foreground features, Indicates taking the maximum value, Represents the calculation of Euclidean distance.

7. The method for re-identifying people changing clothes based on clothing-aware mixing and foreground enhancement according to claim 6, characterized in that: Step S4 is specifically as follows: The data in the test set is input into the trained and optimized clothing-changing pedestrian re-identification model. The features of the input pedestrian image are extracted through the optimized clothing-changing pedestrian re-identification model. The similarity between the extracted features and the features in the cloud database is calculated one by one, and the scores are given according to the similarity scores. The image with the highest similarity score is selected as the final retrieval result.

Images