A pedestrian re-identification representation learning technique based on feature separation and fusion

Abstract

The present application relates to a kind of pedestrian re-identification representation learning method based on feature separation and fusion, belong to artificial intelligence, deep learning and computer vision technical field.The present application considers that there is difference between face and shape, body language and gesture and other features in representation learning, by diversified representation learning and feature fusion, complete pedestrian representation information, improve the accuracy and robustness of pedestrian representation learning:Firstly, using image space separability, the unstable clothing information is separated from face and other stable information, and clothing representation model (Clothing Representation Model, CR) and clothing-independent pedestrian representation model (Clothing-Independent Person Representation Model, CIPR) are learned respectively;Then, CR and CIIR are jointly trained, high-level semantics such as shape, body language and gesture are supplemented from pedestrian clothing, clothes-dependent pedestrian representation model (Clothes-Dependent Person Representation Model, CDPR) is learned based on clothing, and through model fusion, complement each other, enhance pedestrian representation information.

Description

TECHNICAL FIELD

[0001] The present application relates to a pedestrian re-identification feature learning technology based on feature separation and fusion, and belongs to the technical field of artificial intelligence, deep learning and computer vision. BACKGROUND

[0002] Person Re-Identification is a technology that realizes cross-camera pedestrian recognition by image analysis and comparison of pedestrians captured by different monitoring cameras in video monitoring and other scenarios. The process of person re-identification is similar to that of face recognition, including three parts of detection, feature extraction and recognition. Pedestrian detection pre-generates a series of pre-selected frames according to the input image, extracts the area in the image that may include pedestrians and draws the corresponding range, then classifies the pre-selected frames extracted to determine whether they belong to pedestrians or additional background information, and then modifies the position of the bounding box through classification or regression. When a pedestrian is detected by multiple bounding boxes, the non-maximum suppression method is used to select the best bounding box for each detected pedestrian object. The feature matrix of the detected pedestrian frame is extracted using the corresponding feature extraction method, and the similarity between the pedestrian and the pedestrians in the database is measured to determine the similarity of the pedestrians.

[0003] Person re-identification needs to train a model through a machine learning algorithm to realize the extraction and recognition of pedestrian features. The traditional learning method of person re-identification focuses on feature representation and metric learning. Feature representation is mainly extracted by manual design to extract more significant pedestrian features. Common features include color, semantics, etc. The focus of feature representation is to design a reliable pedestrian image feature representation model that can distinguish different pedestrians and is not affected by factors such as viewing angle and lighting. The color space YCbCr obtained by converting HSV and RGB can be combined with Gabor filter to redefine a new feature space. However, the method of extracting only a single feature cannot obtain good recognition accuracy. In order to enhance the feature extraction capability, or to solve the problem of large lighting changes under different cameras and at the same time not to lose the spatial detail information of the image, the image is windowed, and in each patch, the HSV, SILTP features are extracted, and the maximum value of each element between the horizontal patch features is obtained to obtain LOMO (Local Maximum Occurrence Feature). This feature can greatly improve the accuracy of person re-identification and has the ability to distinguish local information. However, the manual feature extraction method has limited ability to extract pedestrian features, resulting in low recognition accuracy of the traditional learning method of person re-identification, which is not enough to apply the technology of person re-identification to real life.

[0004] In deep learning, pedestrian re-identification needs to fully utilize various appearance features of human body, not limited to face, including dressing information, in order to effectively identify pedestrian individuals under complex conditions in an open world. Although most of the surface of the human body is covered by various clothes, the clothes of the pedestrian are not fixed and unique, so the pedestrians cannot be distinguished and identified simply according to the differences in clothes. However, the shape, posture and gesture of the pedestrian dressing contain rich individual distinguishable information, which provides important evidence for pedestrian re-identification. Obviously, combining these information with the face can make up for the deficiency of the face representation and effectively improve the pedestrian representation ability. However, how to learn stable pedestrian representation from the variable dressing, including shape, posture and gesture, and eliminate the interference caused by unstable clothes is a problem faced by pedestrian re-identification at present

[0005] At present, some pedestrian re-identification methods use shape, posture and gesture label information to train shape and gait models. Such methods rely on additional labeling models or equipment, and the cost is large; in addition, this labeling method based on human modeling cannot cover the diversity of human posture changes, lacks description of the face, and cannot reflect the influence of the outside world on the image, such as occlusion, and the expression and generalization ability of the model is slightly insufficient. Or directly extract pedestrian representation information from the RGB image. This method eliminates or reduces the sampling of clothes information by the encoder through the design of a special loss function or the introduction of adversarial learning, so as to learn pedestrian representation information (such as face, shape, posture and gesture) that is irrelevant to clothes. However, due to the inseparable relationship between dressing and clothes, this approach inevitably weakens the ability to learn shape, posture and gesture representation from dressing. Researchers introduce feature decomposition representation learning to separate shape features and appearance features from RGB images or contour images, and this work also ignores the use of face information. In fact, face, shape, posture, gesture and clothes all play an important role in pedestrian re-identification, and the current research work does not fully utilize them, and there is still a lot of room for improvement. SUMMARY

[0006] The purpose of the present application is to overcome the problem that the prior art does not fully utilize the pedestrian information in the image. In order to fully utilize the pedestrian representation contained in the image and improve the accuracy of the clothes-changing pedestrian re-identification, a pedestrian re-identification representation learning technology based on feature separation and fusion is creatively proposed, which is a feature decomposition and fusion representation learning method, referred to as DRLF.

[0007] The innovation of the present application lies in that, considering that there is a difference between the features such as face, shape, posture and gesture in representation learning, a representation learning method based on feature decomposition and fusion (DRLF) is proposed in the present application, through diversified representation learning and feature fusion, the pedestrian representation information is completed, and the accuracy and robustness of the pedestrian representation learning are improved: firstly, the unstable clothing information is separated from the face and other stable information by using the separability of the image space, and the clothing representation model (CR) and the clothing-independent person representation model (CIPR) are learned respectively; then, the CR and the CIPR are jointly trained, the high-level semantics such as shape, posture and gesture are supplemented from the clothing of the pedestrian, the clothes-dependent person representation model (CDPR) is learned, and the pedestrian representation information is complemented through model fusion.

[0008] The present application is realized by the following technical solutions.

[0009] A pedestrian re-identification representation learning technology based on feature separation and fusion, comprising the following steps:

[0010] Step 1: Establishing a clothing-independent person representation model in diversified representation learning

[0011] In pedestrian re-identification, clothing change is one of the main factors leading to reduced recognition accuracy. Since clothing often changes at different times or in different scenes, it is crucial to eliminate the influence of clothing change in pedestrian re-identification. In order to solve this problem, the present method establishes a clothing-independent person representation model (CIPR) in diversified representation learning to capture the clothing-independent pedestrian features.

[0012] The clothing-independent person representation model (CIPR) includes a pedestrian encoder, a pedestrian classifier and a clothing classifier. The pedestrian encoder is used to capture the clothing-independent pedestrian features; the pedestrian classifier is used to capture the components related to identity recognition in the pedestrian encoder; and the clothing classifier is used to capture the components related to clothing in the pedestrian encoder.

[0013] Step 1.1: Training the pedestrian classifier for the pedestrian encoder;

[0014] The pedestrian classification loss is optimized by using formula (1) to make the pedestrian encoder preliminarily capture the pedestrian features;

[0015]

[0016] wherein represents the pedestrian classification loss of CIPR, CE(·) represents the cross-entropy loss, represents the pedestrian classifier of CIPR, represents the pedestrian encoder, x i represents the input pedestrian image, represents the true pedestrian label of the image x i ;

[0017] Step 1.2: On the basis of step 1.1, the pedestrian encoder is trained for dress classification;

[0018] The pedestrian encoder is frozen, and the dress classification loss is optimized by formula (2), the purpose of which is to train the dress classifier;

[0019]

[0020] wherein represents the dress classification loss of CIPR, Loss(·) represents the loss function, represents the dress classifier of CIPR, represents the true dress label of the image x i ;

[0021] Step 1.3: At the same time as step 1.2, the pedestrian encoder is trained for dress adversarial learning to capture the pedestrian features irrelevant to the clothes;

[0022] The dress classifier is frozen, and the pedestrian encoder and the pedestrian classifier are trained by formula (3);

[0023]

[0024] wherein, represents the dress adversarial loss function;

[0025] The whole adversarial process includes two optimization objectives, the first optimization objective is the difference between the average cross-entropy loss of the current pedestrian and the rest of the clothes and the cross-entropy of the current clothes of the current pedestrian, as shown in formula (4); the second optimization objective is the pedestrian classification loss;

[0026]

[0027] wherein, y represents the dress label, represents the multiple dress label sets of the pedestrian corresponding to the current image x i , and Avg represents the calculation of the average;

[0028] Step 2, establish a dress representation model (CR) in diversified representation learning

[0029] Since the clothing also contains high-level semantics such as body shape, posture and gesture, the method establishes a clothing representation model (CR) in diversified representation learning to capture clothing features.

[0030] The clothing classifier is trained, and the clothing classification loss of the CR is optimized by formula (5), and the purpose is to enable the clothing encoder to capture features related to clothing;

[0031]

[0032] wherein, the clothing classification loss of the CR, the clothing encoder of the CR, the clothing classifier of the CR;

[0033] Step 3, on the basis of step 2, learn a clothing-based pedestrian representation model (CDPR) in diversified representation learning

[0034] The clothing representation information contains variable clothing features, and the clothing change mainly distributes in the low-level network of the clothing encoder, and the body shape, posture and gesture are as implicit semantics, which are distributed in the high-level network.

[0035] The bottom network of the clothing encoder is frozen, and the high-level network is fine-tuned by formula (6), and the purpose is to capture stable feature information such as body shape, posture and gesture contained in the clothing;

[0036]

[0037] wherein, the pedestrian classification loss of the CDPR, the pedestrian classifier of the CDPR, the clothing encoder of the CR with the bottom network frozen;

[0038] Step 4: on the basis of step 1 and step 3, model fusion is carried out.

[0039] In the model fusion stage, under the condition that the CIPR encoder is completely frozen and the low-level network of the CR encoder is partially frozen, the high-level network of the CR is fine-tuned through pedestrian classification learning, the clothing-based pedestrian representation model (CDPR) is trained, the high-level semantics such as body shape, posture and gesture are extracted from the clothing, the pedestrian representation information of the CIPR is enhanced through feature fusion, and the accuracy and robustness of pedestrian re-identification are improved.

[0040] For images and video sequences, the application proposes a model fusion method of probability layer, feature layer and video sequence layer.

[0041] Step 4.1, probability layer fusion;

[0042] The scores of CIPR and CDPR are fused by using formula (7) or formula (8);

[0043] v = v CIPR +(1-v CIPR )×v CDpR (7)

[0044] v = λv CIPR +(1-λ)×v CDpR (8)

[0045] Wherein, v represents the fused score, v CIPR represents the score of CIPR model, v CDPR represents the score of CDPR model, and λ represents the hyperparameter;

[0046] Step 4.2, feature layer fusion;

[0047] The features of CIPR and CDPR are fused using maximum pooling, and formula (9) is used for joint training, the purpose is to enhance the representation ability of pedestrians;

[0048]

[0049] Wherein, represents the pedestrian classification loss of the fused feature, represents the fused pedestrian classifier, and Maxpooling represents maximum pooling;

[0050] Step 4.3, video sequence fusion

[0051] Step 4.3.1, intra-frame multi-model fusion, inter-frame multi-representation fusion;

[0052] Formula (10) is used to fuse the features within the image frame; formula (11) is used to fuse the sequences between the image frames by average pooling, and the fused pedestrian representation is obtained;

[0053]

[0054] v = AvgPooling(v1, …, v n ) (11)

[0055] Wherein, v i (i≥0, i≤n) represents the fused feature vector of image frame i, n represents the total number of image frames in the video sequence, represents the CIPR representation vector of image frame i, CDPR representation vector of image frame i, v represents the sequence fused pedestrian representation vector, and Avgpooling represents average pooling;

[0056] Step 4.3.2, single model inter-frame representation fusion, multi-model representation fusion;

[0057] The CIPR representation and the CDPR representation in the video sequence are extracted respectively by using formula (12); and then the pedestrian representation of the video sequence is fused by using formula (13);

[0058]

[0059] Wherein, v CIPR And v CDPR Respectively represent the pedestrian representation irrelevant to clothing and the pedestrian representation based on dressing in the video sequence.

[0060] v=MaxPooling(v CIPR , v CDPR ) (13)

[0061] Beneficial effects

[0062] Compared with the prior art, the present application has the following beneficial effects:

[0063] 1. The present application concept is simple and innovative. For the first time, feature decomposition and adversarial learning are combined to solve the problem of pedestrian dressing change, and through feature fusion, the features with discriminative ability in the pedestrian image are fully utilized, and the effectiveness of the model is proved by experimental results.

[0064] 2. In view of the problem that the existing method ignores the pedestrian representation contained in the dressing, the present application adopts a finding strategy to recover the lost pedestrian representation information from the subsequent dressing representation model, and enhances the representation ability of the pedestrian.

[0065] 3. The present application is a general network architecture, which can not only be used for image dressing pedestrian re-identification, but also be used for processing video data pedestrian re-identification.

[0066] 4. Experiments show that the present application is obviously superior to the benchmark model in precision and efficiency, and under the condition of not needing fine-tuning, the Top-1 and mAP of the DRLF model on the LTCC, PRCC, CCVID and other data sets reach the best effect of the existing.

[0067] 5. The idea of "feature decomposition and fusion" of the present application can be extended to a wide range of application fields to realize unified representation in the field. BRIEF DESCRIPTION OF DRAWINGS

[0068] Figure 1The figure is a schematic diagram of the DRLF of the present application. In the diversified representation learning stage, it respectively includes a dressing representation model, a clothing-independent pedestrian representation model and a dressing-based pedestrian representation model. In the model fusion stage, it includes a probability layer and a feature fusion method.

[0069] Figure 2 The figure is a schematic diagram of the two fusion methods of the video sequence layer of the present application.

[0070] Figure 3 The figure is a schematic diagram of the specific implementation process of the present application. DETAILED DESCRIPTION

[0071] The present application will be further illustrated and described in detail in combination with the drawings and examples.

[0072] EMBODIMENT

[0073] The present application is a general network framework, which can solve the clothing-changing pedestrian re-identification problem in images and videos. The steps of the entire implementation method are shown in the figure. Figure 3

[0074] For images and videos, Resnet50 is selected as the backbone network. In videos, spatial maximum pooling and temporal average pooling are used to integrate the output feature maps of the backbone network. The standard input size of the image is 384x192, and random horizontal flipping, random cropping and random erasing are used for training data enhancement. The standard input size of the video is 256x128, and horizontal flipping is used for training data enhancement. For videos of different lengths, 8 frames are randomly sampled with a step size of 4 to generate training data. The number of image training iterations is set to 100, and the number of video training iterations is set to 200. Adam optimizer is used for training, the learning rate of image training is initialized to 2.5e -4 , and the learning rate is decayed by 10% every 20 epochs. The learning rate of video training is initialized to 2.8e -4 , and the learning rate is decayed by 10% every 40 epochs.

[0075] Among them, ResNet50 is a deep convolutional network proposed by K. He in Deep residual learning for image recognition (in Proc. CVPR, 2016).

[0076] Embodiment 1: Taking probability fusion as an embodiment for illustration:

[0077] Step 1: First, train the clothing-independent pedestrian representation model (CIPR).

[0078] ​Step 1.1: In the first stage of training (i.e., the first 20 epochs), the pedestrian encoder is trained for pedestrian classification

[0079] The pedestrian classification loss is optimized using equation (1) to allow the pedestrian encoder to initially capture pedestrian features.

[0080]

[0081] Step 1.2: In the second stage of training (i.e., the last 80 epochs), the clothing classifier is first learned based on step 1.1, while the pedestrian encoder parameters are frozen, and the clothing classification loss is optimized using equation (2).

[0082]

[0083] To prevent overfitting during training, both the encoder and the clothing classifier output are subject to l-2 regularization. Since clothing belongs to a fine classification, unlike pedestrian classification, ArcFace Loss proposed by J. Deng et al. in Arcface: Additive angular margin loss for deep face recognition (in Proc. CVPR, 2019) is used as the loss function for training.

[0084] Step 1.3: While step 1.2 is being performed, the clothing classifier is frozen, and the pedestrian encoder and pedestrian classifier are trained.

[0085] With the clothing classifier frozen, the pedestrian encoder and pedestrian classifier are trained using equation (3);

[0086]

[0087] The entire adversarial process includes two optimization objectives. The first optimization objective is the difference between the average cross-entropy loss of the current pedestrian's remaining clothing and the current pedestrian's current clothing cross-entropy, as shown in equation (4); the second optimization objective is the pedestrian classification loss.

[0088]

[0089] Step 2: Establishing a clothing representation model (CR) in diverse representation learning

[0090] In the first 20 epochs of training, the clothing classifier is trained, and the clothing classification loss is optimized using equation (5) to allow the clothing encoder to capture features related to clothing.

[0091]

[0092] Similarly, Arcface is used as the loss function.

[0093] Step 3: On the basis of step 2, diversify the learning of the clothing-based pedestrian representation model (CDPR) in the representation learning

[0094] In the last 80 epochs of training, the bottom network of the clothing encoder is frozen, and the high-level network is fine-tuned by formula (6), aiming to capture stable feature information such as shape, posture and pose contained in the clothing;

[0095]

[0096] Step 4: On the basis of step 1 and step 3, the features of the clothing-independent pedestrian representation model and the clothing-based pedestrian representation model are fused by probability fusion. The model fusion process is carried out at the same time as the training process of the clothing-based pedestrian representation model.

[0097] CIPR and CDPR have their own encoders and pedestrian classifiers, and the prediction process is completely independent, and only the score is fused for calculation. During training, the score is the pedestrian classification prediction probability, and during inference, the score is the cosine similarity distance.

[0098] The present application adopts two score fusion methods, as shown in formulas (7) and (8).

[0099] v = v CIPR +(1-v CIPR )×v CDPR (7)

[0100] v = λv CIPR +(1-λ)×v CDPR (8)

[0101] Wherein, λ is a hyperparameter.

[0102] Example 2: Take feature layer fusion as an example:

[0103] Step 1: First, train the clothing-independent pedestrian representation model (CIPR).

[0104] Step 1.1: In the first stage of training (i.e. the first 20 epochs), train the pedestrian encoder for pedestrian classification

[0105] Optimize the pedestrian classification loss using formula (1) to make the pedestrian encoder preliminarily capture pedestrian features;

[0106]

[0107] Step 1.2: The second stage of training (i.e., the last 80 epochs), based on step 1.1, first learns the dressing classifier, freezes the pedestrian encoder parameters, and optimizes the dressing classification loss using equation (2).

[0108]

[0109] To prevent overfitting during training, both the encoder and the dressing classifier output are subject to l-2 regularization. Since dressing belongs to a fine classification, unlike pedestrian classification, we use ArcFace Loss proposed by J. Deng et al. in Arcface: Additive angular margin loss for deep face recognition (in Proc. CVPR, 2019) as the loss function for training.

[0110] Step 1.3: While step 1.2 is running, freeze the dressing classifier and train the pedestrian encoder and pedestrian classifier.

[0111] Freeze the dressing classifier and train the pedestrian encoder and pedestrian classifier using equation (3);

[0112]

[0113] The entire adversarial process includes two optimization objectives. The first optimization objective is the difference between the average cross-entropy loss of the current pedestrian's remaining clothing and the current pedestrian's current clothing cross-entropy, as shown in equation (4); the second optimization objective is the pedestrian classification loss.

[0114]

[0115] Step 2: Establishing a dressing representation model (CR) in diverse representation learning

[0116] In the first 20 epochs of training, the dressing classifier is trained, and the dressing classification loss is optimized using equation (5), with the goal of allowing the dressing encoder to capture features related to dressing;

[0117]

[0118] Similarly, Arcface is used as the loss function.

[0119] Step 3: Learning a dressing-based pedestrian representation model (CDPR) in diverse representation learning based on step 2

[0120] In the last 80 epochs of training, the bottom network of the dressing encoder is frozen, and the high-level network is fine-tuned using equation (6), with the goal of capturing stable feature information such as shape, posture, and pose contained in dressing.

[0121]

[0122] Step 4: On the basis of step 1 and step 3, the features of the clothing- independent pedestrian representation model and the clothing-based pedestrian representation model are fused by feature fusion. The model fusion process is carried out at the same time as the training process of the clothing-based pedestrian representation model.

[0123] CIPR and CDPR have independent encoders, and share a pedestrian classifier On the full connection layer of the backbone network, the features of CIPR and CDPR are fused by using maximum pooling, and formula (9) is used for joint training, so as to enhance the pedestrian representation ability;

[0124]

[0125] Embodiment 3: Take video sequence fusion, intra-frame multi-model fusion, and inter-frame multi-representation fusion as an embodiment for illustration:

[0126] Step 1: First, train the clothing-independent pedestrian representation model (CIPR).

[0127] Step 1.1: In the first stage of training (i.e. the first 20 epochs), the pedestrian encoder is trained for pedestrian classification

[0128] The pedestrian classification loss is optimized by formula (1), so that the pedestrian encoder can preliminarily capture the pedestrian features;

[0129]

[0130] Step 1.2: In the second stage of training (i.e. the last 80 epochs), on the basis of step 1.1, first learn the clothing classifier, freeze the pedestrian encoder parameters, and optimize the clothing classification loss by formula (2)

[0131]

[0132] In order to prevent overfitting during training, both the encoder and the clothing classifier output are subject to l-2 regularization. Since clothing belongs to fine classification, which is different from pedestrian classification, ArcFace Loss proposed by J. Deng et al. in Arcface: Additive angular margin loss for deep face recognition (in Proc. CVPR, 2019) is used as the loss function for training.

[0133] Step 1.3: At the same time as step 1.2, freeze the clothing classifier, and train the pedestrian encoder and the pedestrian classifier.

[0134] Freezing the dressing classifier, train the pedestrian encoder and pedestrian classifier using formula (3);

[0135]

[0136] The whole confrontation process includes two optimization objectives, the first optimization objective is the difference between the average cross-entropy loss of the current pedestrian and the rest of the clothing and the cross-entropy of the current clothing of the current pedestrian, as shown in formula (4); the second optimization objective is the pedestrian classification loss;

[0137]

[0138] Step 2: Establishing a dressing representation model (CR) in diversified representation learning

[0139] In the first 20 epochs of training, the dressing classifier is trained, and the dressing classification loss is optimized using formula (5), the purpose of which is to let the dressing encoder capture features related to dressing;

[0140]

[0141] Similarly, Arcface is used as the loss function.

[0142] Step 3: On the basis of step 2, learning a dressing-based pedestrian representation model (CDPR) in diversified representation learning

[0143] In the last 80 epochs of training, the bottom network of the dressing encoder is frozen, and the high-level network is fine-tuned using formula (6), the purpose of which is to capture stable feature information such as shape, posture and pose contained in dressing;

[0144]

[0145] Step 4: On the basis of steps 1 and 3, the features of the clothing-independent pedestrian representation model and the dressing-based pedestrian representation model are fused through intra-frame multi-model fusion and inter-frame multi-representation fusion. The model fusion process is carried out at the same time as the training process of the dressing-based pedestrian representation model.

[0146] Using formula (10), intra-frame feature fusion is performed; using formula (11), inter-frame sequence fusion is performed through average pooling to obtain the fused pedestrian representation;

[0147]

[0148] v = AvgPooling(v1, ···, v n ) (11)

[0149] Embodiment 4: Take video sequence fusion, single model frame question representation fusion, and multi-model representation fusion as examples for illustration

[0150] Step 1: First, train the clothing-independent pedestrian representation model (CIPR).

[0151] Step 1.1: In the first stage of training (i.e., the first 20 epochs), train the pedestrian encoder for pedestrian classification

[0152] Optimize the pedestrian classification loss using formula (1) to preliminarily capture the pedestrian features with the pedestrian encoder;

[0153]

[0154] Step 1.2: In the second stage of training (i.e., the last 80 epochs), based on step 1.1, first learn the clothing classifier, freeze the pedestrian encoder parameters, and optimize the clothing classification loss using formula (2).

[0155]

[0156] To prevent overfitting during training, both the encoder and the clothing classifier output are regularized by 1-2. Since clothing belongs to a fine classification, unlike pedestrian classification, ArcFace Loss proposed by J. Deng et al. in Arcface: Additive angular margin loss for deep face recognition (in Proc. CVPR, 2019) is used as the loss function for training.

[0157] Step 1.3: While step 1.2 is being performed, freeze the clothing classifier and train the pedestrian encoder and pedestrian classifier.

[0158] Freeze the clothing classifier and train the pedestrian encoder and pedestrian classifier using formula (3);

[0159]

[0160] The entire adversarial process includes two optimization objectives. The first optimization objective is the difference between the average cross-entropy loss of the current pedestrian and the rest of the clothing and the cross-entropy of the current pedestrian's current clothing, as shown in formula (4); the second optimization objective is the pedestrian classification loss.

[0161]

[0162] Step 2: Establish the clothing representation model (CR) in diversified representation learning

[0163] In the first 20 epochs of training, the dressing classifier is trained to optimize the dressing classification loss using formula (5), aiming to let the dressing encoder capture dressing-related features;

[0164]

[0165] Similarly, Arcface is used as the loss function.

[0166] Step 3: Based on step 2, diversify the representation learning and learn the dressing-based pedestrian representation model (CDPR)

[0167] In the last 80 epochs of training, the bottom network of the dressing encoder is frozen, and the high-level network is fine-tuned using formula (6), aiming to capture stable features such as shape, posture, and pose contained in the dressing;

[0168]

[0169] Step 4: Based on steps 1 and 3, the features of the clothing-independent pedestrian representation model and the dressing-based pedestrian representation model are fused through single-model inter-frame representation fusion and multi-model representation fusion. The model fusion process is carried out simultaneously with the training process of the dressing-based pedestrian representation model

[0170] First, for CIPR and CDPR, between image frames, get the clothing-independent pedestrian representation vector v CIPR and the dressing-based pedestrian representation vector v CDPR ; Finally, through formula (13), the pedestrian representation of the video sequence is obtained, which is used for pedestrian classification training.

[0171]

[0172] v = MaxPooling(v CIPR , v CDPR ) (13)

[0173] To verify the effectiveness and universality of the method, comprehensive verification experiments are carried out on image pedestrian re-identification datasets and video pedestrian re-identification datasets. The datasets are as follows:

[0174] (1) LTCC dataset: is a long time span clothes change person re-identification dataset. It is collected by 12 cameras, including 15 people with 478 sets of clothes, a total of 17138 pictures. It contains two sets, the clothes change set contains 91 people with 417 sets of clothes, with 14756 pictures; the non-clothes change set contains 61 people, with 2382 pictures. Each person is captured by at least two cameras, the dataset has changes in light, angle and posture, and contains changes in clothes and carried items, even including hairstyle changes.

[0175] (2) PRCC dataset: The images in the PRCC dataset not only include the change of clothes of the same person in different camera views, but also include changes in lighting, occlusion, posture and angle. It includes 33698 pictures from 221 people, including 3 different angles, and also provides a sketch of the task outline to facilitate the extraction of the outline information of the person.

[0176] (3) CCVID dataset: CCVID is a video clothes change person re-identification dataset, including 2856 video sequences of 226 people. The entire dataset includes 347,833 bounding boxes, each sequence length varies from 27 frames to 410 frames, with an average length of 122 frames. Fine-grained clothing labels are provided, including tops, bottoms, shoes, wearing states and accessories, etc. Among them, 75 people's identities are used for training, and 151 people's identities are used for testing, with 834 sequences in the test set as the query set and 1074 sequences as the gallery set.

[0177] (4) VC-Clothes dataset: is a virtual dataset from the game GTA-V, including 512 ids and 19060 images. The training set and the test set include 256 people respectively. In the test data, 4 images of each person are randomly selected from each camera as the query set, and the remaining images are as the gallery set.

[0178] (5) LaST dataset: is a large dataset containing more than 228k person images collected from movies, used to study person activity range and time span large scenes. The training set includes 5000 person identities with a total of 71248 images, the validation set contains 56 person identities with a total of 21379 images, and the test set contains 5806 person identities with a total of 135529 images.

[0179] The present application is compared with GI-ReID (Cloth-Changing Person Re-identification from A Single Image with Gait Prediction and Regularization) and Clothes-based Adversarial Loss (CAL) method respectively to verify the effectiveness of the proposed method. GI-ReID is an algorithm proposed by Xin Jin, Tianyu He, Kecheng Zheng in Cloth-Changing Person Re-identification from A Single Image with Gait Prediction and Regularization, (in Proc. CVPR, 2022), which introduces gait recognition as an auxiliary task. By utilizing the personal unique and clothing-independent gait information, it drives the image ReID model to learn clothing-agnostic representation. GI-ReID adopts a dual-stream structure composed of an image ReID stream and an auxiliary gait recognition stream. The gait stream acts as a regulator to encourage the ReID stream to capture clothing-invariant biological feature motion characteristics during training. CAL is an algorithm proposed by Xinqian Gu et al. in Cloth-Changing Person Re-identification with RGB Modality Only, (in Proc. CVPR, 2022), which is currently considered to be the best performing method. In the CAL algorithm, a clothing classifier is added after the backbone of the re-id model, and CAL is defined as a multi-positive classification loss, where all clothing classes belonging to the same identity are mutually positive classes. By punishing the re-id model's clothing prediction ability, it extracts clothing-independent features from the original RGB image.

[0180] The algorithm performance is measured using the evaluation standard commonly used in the field of pedestrian re-identification. The evaluation indicators include cumulative matching characteristics (CMC) curve and mean average precision (mAP), where the specific meaning of the CMC curve is that the correct matching rate of the first r retrieval results is included in the candidate pedestrian library (gallery) when searching for the query (probe) pedestrian. The formula for calculating the first k matching rate of CMC is as follows:

[0181]

[0182] Where prob represents the target pedestrian to be queried, N q represents the total number of samples in the query set, and k represents any sample in the gallery set.

[0183]

[0184] The mean average precision (mAP) is a performance measure for this kind of algorithm that predicts the location and the class of the target. The definitions of precision P, average precision AP and mean average precision mAP are as follows:

[0185]

[0186]

[0187] where M is the number of images in the candidate images with the same label as the probe image for the probe image. {j1, j2, …, jM} is the index position of the M positive samples in the ranking result. M

[0188]

[0189] The running results of the present application on five data sets are shown in Tables 1-5. Among them, General represents calculating the evaluation index on the entire data set, CC represents calculating the evaluation index of only clothing change samples, and SC represents calculating the evaluation index of only clothing unchanged samples.

[0190] Table 1 Performance of the algorithm on the LTCC data set

[0191]

[0192] Table 2 Performance of the algorithm on the PRCC data set

[0193]

[0194] Table 3 Performance of the algorithm on the CCVID data set

[0195]

[0196] Table 4 Performance on the VC-Clothes data set

[0197]

[0198] Table 5 Performance on the LaST data set

[0199]

[0200] ​The values of Top-1 and mAP of DRLF are 77.1% and 42.2% respectively under the General setting of LTCC dataset, and the mAP is increased by 12.8% and 1.4% compared with GI-ReID and CAL respectively, which shows the effectiveness of DRLF in solving the problem of cross-attire pedestrian re-identification.

[0201] On the PRCC image cross-attire pedestrian re-identification dataset, the accuracy of GI-ReID, CAL and DRLF algorithms is low under the cross-attire setting. The reason is that the number of pedestrian samples with clothing changes is unevenly distributed on the PRCC dataset, which leads to deviation in the process of feature extraction. In addition, compared with the LTCC dataset, the image quality of PRCC is lower, which affects the modeling ability of high-level semantic of clothing.

[0202] On the CCVID video cross-attire pedestrian re-identification dataset, due to the uneven distribution of the length of each pedestrian video sequence, and the large difference between some frames, the current methods cannot achieve very good results. The experimental results of CCVID show that DRLF algorithm is better than CAL algorithm under two different settings, with an average increase of 2.8% in Top-1 and 1.85% in mAP.

[0203] On the VC-Clothes dataset, since the data is a virtual dataset from the game world, the clothing change is more obvious, and the image quality is significantly better than the rest of the datasets, so the CAL algorithm and DRLF algorithm have good performance on this dataset. The experimental results of VC-Clothes show that although DRLF algorithm has an average increase of 0.3% in mAP compared with CAL algorithm, it has an increase of 4.9% in top-1 under the cross-attire setting, which proves the accuracy of DRLF algorithm in identifying pedestrian identity under clothing change.

[0204] On the LaST dataset, DRLF has an increase of 5.6% in mAP and 3.5% in Top-1 compared with CAL algorithm, which proves that DRLF method has good performance in large cross-attire pedestrian re-identification datasets.

[0205] Overall, DRLF has obvious improvement compared with the CAL and GI-ReID algorithms with good performance on the five cross-attire pedestrian re-identification datasets of LTCC, PRCC, VC-Clothes, CCVID and LaST, with an average increase of 2.16% in mAP and 1.9% in Top-1. This result shows that the analysis of features related to pedestrian identity recognition in cross-attire pedestrian re-identification is effective, and the method of feature decomposition and fusion can fully capture the features related to pedestrian recognition in images or videos, and the pedestrian re-identification algorithm designed based on this analysis has high performance.

Claims

1. A person re-identification representation learning method based on feature separation and fusion, characterized in that, Includes the following steps: Step 1: Building a clothing-independent pedestrian representation model in diverse representation learning (CIPR) Step 2: Establishing a Clothing Representation Model (CR) in Diverse Representation Learning The clothing classifier is trained and the clothing classification loss is optimized using formula (5). The purpose is to enable the clothing encoder to capture clothing-related features. Where, x i This represents the input pedestrian image. Representing image x i Authentic clothing labels This represents the clothing classification loss of CR. This indicates the CR's clothing encoder. Let represent the clothing classifier for CR, and Loss(·) represent the loss function; Step 3: Building upon Step 2, learn a clothing-based pedestrian representation model CDPR in the diversified representation learning process. The bottom layer network of the clothing encoder is frozen, and the high layer network is fine-tuned using formula (6) in order to capture the characteristic information of body shape, posture and stability contained in clothing. in, This represents the pedestrian classification loss of CDPR. This represents the pedestrian classifier for CDPR. This indicates that the clothing encoder of the underlying network's CR is frozen. Representing image x i The actual pedestrian labels, where CE(·) represents the cross-entropy loss; Step 4: Based on Step 1 and Step 3, perform model fusion; The CIPR encoder is completely frozen, and the CIPR model is fused with the CDPR model while training CDPR.

2. The person re-identification representation learning method based on feature separation and fusion according to claim 1, characterized in that, In step 1 Step 1.1: Train the pedestrian encoder for pedestrian classification. The pedestrian classification loss is optimized using formula (1) so that the pedestrian encoder can initially capture pedestrian features; in Let CE(·) represent the pedestrian classification loss of CIPR, and let CE(·) represent the cross-entropy loss. This represents the pedestrian classifier for CIPR. Indicates pedestrian encoder, x i This represents the input pedestrian image. Representing image x i Real pedestrian tags; Step 1.2: Based on Step 1.1, perform clothing classification training on the pedestrian encoder; Freeze the pedestrian encoder and use formula (2) to optimize the clothing classification loss. The purpose is to train the clothing classifier. in This represents the clothing classification loss of CIPR, where Loss(·) represents the loss function. This represents the CIPR clothing classifier. Representing image x i Authentic clothing labels; Step 1.3: Simultaneously with Step 1.2, perform clothing adversarial learning on the pedestrian encoder to capture pedestrian features unrelated to clothing. Freeze the clothing classifier and train the pedestrian encoder and pedestrian classifier using formula (3); in, This represents the dress code adversarial loss function; The entire adversarial process includes two optimization objectives. The first optimization objective is the difference between the average cross-entropy loss of the current pedestrian's other clothing and the cross-entropy of the current pedestrian's current clothing, as shown in formula (4). The second optimization objective is the pedestrian classification loss. Where y represents the clothing label, Represents the current image x i The set of multiple clothing tags corresponding to the pedestrians, where Avg represents the calculated mean.

3. The person re-identification representation learning method based on feature separation and fusion according to claim 2, characterized in that, The fusion in step 4 includes three types. Step 4.1, Probabilistic Layer Fusion The scores of CIPR and CDPR are merged using formula (7) or formula (8); v=v CIPR +(1-v CIPR )×v CDPR (7) v=λv CIPR +(1-λ)×v CDPR (8) Where v represents the score after fusion, v CIPR v represents the score of the CIPR model. CDPR λ represents the score of the CDPR model, and λ represents the hyperparameters. Step 4.2, Feature Layer Fusion Max pooling is used to fuse the features of CIPR and CDPR, and joint training is performed using formula (9) to enhance the pedestrian representation ability. in, The pedestrian classification loss represents the fusion features. This represents the fused pedestrian classifier, and Maxpooling represents max pooling. Step 4.3, Video Sequence Fusion Step 4.3.1: Intra-frame multi-model fusion and inter-frame multi-representation fusion Using formula (10), feature fusion is performed within the image frame; using formula (11), sequence fusion is performed between image frames through average pooling to obtain the fused pedestrian representation. v = AvgPooling(v1, ..., v) n (11) where v i (i≥0, i≤n) represents the feature vector after fusion of image frame i, and n represents the total number of image frames in the video sequence. This represents the CIPR representation vector of image frame i. Let v represent the CDPR representation vector of image frame i, v represent the pedestrian representation vector after sequence fusion, and Avgpooling represent average pooling. Step 4.3.2: Single-model inter-frame representation fusion, multi-model representation fusion Using formula (12), the CIPR and CDPR representations in the video sequence are extracted respectively; then, using formula (13), the pedestrian representations of the video sequence are fused to obtain the pedestrian representations. Among them, v CIPR and v cDPR These represent pedestrian representations in the video sequence that are unrelated to clothing and pedestrian representations that are based on clothing, respectively. v=MaxPooling(v CIPR ,v CDPR ) (13) The above steps complete the entire pedestrian re-identification process involving feature decomposition and fusion.

Images