A method for pedestrian re-identification after changing clothes

Abstract

The present invention discloses a method for re-identifying a person who has changed clothes, including data processing, training phase and testing phase. The network structure adopted includes two feature extraction networks, which respectively extract global features and key point features of human posture, and design a feature enhancement method to improve the robustness of global features; the global features after feature enhancement are fused with the key point features of human posture to obtain features of different regions. At the same time, different training strategies are used for global features and features of different regions during training so that the model focuses on areas with strong resolution and weakens the influence of areas with weak resolution; in the testing phase, the original image and the original image after cutting are used to obtain global features and local features, and the two are fused to obtain fused features for cosine similarity sorting to obtain test results. The problem of feature failure in areas with a large area such as clothing in re-identifying a person who has changed clothes is alleviated, and a new idea is provided for the application of pedestrian re-identification in real scenes.

Description

Technical Field

[0001] The present invention belongs to the field of computer vision and pattern recognition, and in particular relates to a method for re-identifying pedestrians after changing clothes. Background Art

[0002] With the development of society and the improvement of people's material life, public safety has been paid more and more attention by people. In order to maintain social stability and personal and property safety, video surveillance systems have been widely used. So far, video surveillance systems have been involved in various fields, including public transportation, supermarkets, banks, etc. At the same time, the progress of technology and the times urgently requires the monitoring system to be upgraded accordingly, requiring it to be automated and intelligent, and pedestrian re-identification technology, as a key technology of intelligent video surveillance, is particularly important.

[0003] In real monitoring situations, the same pedestrian may change clothes or carry items for a long time. Some criminals may even change clothes to avoid being found by the monitoring system when escaping. The task of pedestrian re-identification with changing clothes is proposed to solve the pedestrian re-identification task under the above situation. The general pedestrian re-identification method will fail when facing the situation of changing clothes. At present, most pedestrian target-based re-identification methods are based on traditional methods. Such pedestrian re-identification methods are difficult to apply in the real scene where pedestrians frequently change clothes. Summary of the invention

[0004] In view of the technical problems faced by traditional pedestrian re-identification under clothing-changing conditions in the above-mentioned prior art, the purpose of the present invention is to provide a method for pedestrian re-identification under clothing-changing conditions, which provides a new idea for the application of pedestrian re-identification in real scenarios.

[0005] In order to achieve the above tasks, the present invention adopts the following technical solutions:

[0006] A method for re-identifying a pedestrian after changing clothes, characterized by comprising the following steps:

[0007] Step 1: Collect experimental data and divide the images in the data set by image cutting, save and add them to the original data set;

[0008] Step 2: Before the training phase begins, the input image is preprocessed;

[0009] Step 3: In the training phase, deep feature extraction networks A and B are used to extract the global features and key features of human body posture of the input image, and the features extracted by network A are fused with the key features of human body posture extracted by network B after feature enhancement.

[0010] Step 4: training the network to differentiate the weights of different human body regions;

[0011] Step 5, the testing phase, uses the fusion features of the global features and local features of the image to generate the test results.

[0012] According to the present invention, the steps for implementing the picture cutting described in step 1 are:

[0013] Step 1.1: Divide the original dataset image P into two equal parts according to its height to obtain P 2-1 , P 2-2 ;

[0014] Step 1.2: Trim the original dataset image P by its height to get P 3-1 , P 3-2 , P 3-3 .

[0015] Specifically, the image preprocessing in step 2 includes the following steps:

[0016] Step 2.1, resize the input image;

[0017] Step 2.2, perform flipping operation on the input image;

[0018] Step 2.3, perform Randomerasing operation on the input image.

[0019] Furthermore, the feature extraction and fusion of the feature extraction networks A and B in step 3 include the following steps:

[0020] Step 3.1, input the image into network A to get f g , and input the image into the feature extraction network B to obtain the feature f p 、f p1 、f p2 、f p3 , where the feature f p is the complete human posture key point feature, feature f p1 、f p2 、f p3 They are the key point features of the head, the key point features of the upper body, and the key point features of the lower body after distinguishing the key points of the human body posture;

[0021] Step 3.2, feature f g After feature enhancement, we get f′ g ;

[0022] Step 3.3, f′ g With f p 、f p1 、f p2 、f p3 Multiply pixel by pixel to get f′p , f′ p1 , f′ p2 , f′ p3 .

[0023] Preferably, step 4 network training includes the following steps:

[0024] Step 4.1, for f′ obtained in step 3 p Perform cross entropy loss and triplet loss;

[0025] Step 4.2, for f′ p1 Do cross entropy loss and use f′ p Make mean square error loss for target;

[0026] Step 4.3, according to f′ p1 , f′ p2 , f′ p3 Find the regional features with strong discriminability and bring the distance between the features with strong discriminability and the global features closer.

[0027] Furthermore, the fusion strategy in step 5 is: P′=P+P 2-1 +0.5×P 2-2 +P 3-1 +0.5×(P 3-2 +P 3-3 ).

[0028] The method for pedestrian re-identification in clothing of the present invention effectively alleviates the problem of changes in the features of clothing and other areas with a large area in pedestrian re-identification under clothing changes. By combining the deep learning method with the traditional manual method to extract the key point features of human posture, the task of pedestrian re-identification in clothing changes is effectively solved, providing a new idea for the application of pedestrian re-identification in real-life scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 A schematic diagram of the overall network structure adopted by the method for re-identifying pedestrians in clothing changes of the present invention;

[0030] Figure 2 This is the network training flow chart;

[0031] Figure 3 It is a flow chart of the feature enhancement method;

[0032] Figure 4 This is the flow chart of the testing phase;

[0033] Figure 5 Flowchart for the fusion of global features and local features;

[0034] Figure 6 Schematic diagram of the input picture;

[0035] Figure 7 Cut diagram for the picture.

[0036] The present invention is further described in detail below with reference to the accompanying drawings and embodiments. DETAILED DESCRIPTION

[0037] This embodiment provides a method for re-identifying pedestrians after changing clothes, and the overall framework of the network structure specifically adopted is as follows: Figure 1 As shown:

[0038] The network consists of two main branches: feature extraction network A and feature extraction network B. Feature extraction network A is a ResNet50 network with ImageNet as a pre-trained model, and feature extraction network B is a HRNet network with ImageNet as a pre-trained model. Feature extraction network A is mainly used to extract the global features of the image, and feature extraction network B is mainly used to extract the key point features of the human body posture in the image.

[0039] The specific training process is as follows: Figure 2 As shown, the following steps are included:

[0040] Step 1: Collect experimental data and divide the images in the data set by image cutting, save them and add them to the original data set.

[0041] In this embodiment, the experimental data set images used are as follows: Figure 6 As shown, there are several pictures of the same pedestrian wearing different clothes, where the pedestrians on the left and the middle are wearing the same clothes, while the pedestrian on the right is wearing different clothes.

[0042] The specific steps for picture cutting are as follows:

[0043] Step 1.1: Divide the original dataset image P into two equal parts according to its height to obtain P 2-1 , P 2-2 ;

[0044] Step 1.2: Trim the original dataset image P by its height to get P 3-1 , P 3-2 , P 3-3 ,like Figure 7 As shown, the left side is the original image P, and the middle is P 2-1 , P 2-2 , the right side is P 3-1 , P 3-2 , P 3-3 .

[0045] Step 2: Before the training phase begins, the input images are preprocessed.

[0046] The specific image preprocessing operations are:

[0047] Step 2.1, resize the input image to a uniform size of 256 pixels in length and 128 pixels in width;

[0048] Step 2.2, perform a flipping operation on the input image, and randomly flip the input image horizontally with a probability of 0.5;

[0049] Step 2.3, perform Randomerasing on the input image, and randomly block the input image with gray blocks of random size with a probability of 0.5, where the area ratio of the blocked area ranges from 0.02 to 0.33, and the aspect ratio of the blocked area ranges from 0.3 to 3.3.

[0050] Step 3: During the training phase, the input image is passed through feature extraction networks A and B to obtain global features and key features of human posture, and the global features are enhanced and then fused with the key features of human posture. The specific steps are:

[0051] Step 3.1: The preprocessed input image is passed through the deep feature extraction network A to extract the global features f of the image. g At the same time, the input image is passed through the deep feature extraction network B to extract the key point features of the human posture f p .

[0052] Step 3.1.a, the present invention divides the key point features of human body posture into regions, specifically, divides them into head region features f according to the distribution of key points of human body posture p1 , Upper body area featuresf p2 , lower body area characteristicsf p3 .

[0053] Step 3.2: Extract the global feature f from the feature extraction network g Through feature enhancement method, a more robust global feature f′ is obtained g , the specific feature enhancement method process is as follows Figure 3 shown.

[0054] Step 3.2.a, input feature f g First, the maximum value of each key point in the key point feature of the human body posture is multiplied pixel by pixel to obtain the mask feature;

[0055] Step 3.2.b, input feature f g It passes through 1×1 convolution, BatchNorm layer and Relu layer in sequence;

[0056] Step 3.2.c, pass the features obtained in step 3.2.b through the MultiHeadSelfAttention mechanism, BatchNorm layer and Relu layer in sequence;

[0057] Step 3.2.d, pass the features obtained in step 3.2.c through 1×1 convolution and BatchNorm layers in sequence;

[0058] Step 3.2.e, add the features obtained in step 3.2.d and the mask features obtained in step 3.2.a, and after passing through a Relu layer, obtain the enhanced feature f′ g .

[0059] Step 3.3: The feature f obtained in step 3.1 p 、f p1 、f p2 、f p3 And the feature f′ obtained in step 3.2 g Multiply pixel by pixel in sequence to obtain different regional features f′ that integrate the key points of human posture p , f′ p1 , f′ p2 , f′ p3 .

[0060] Step 4: Train the network to distinguish the weights of different human body regions in the image. Different training optimization strategies are used for different regions according to the discriminability of the features of different regions.

[0061] Step 4.1, for the feature f′ obtained in step 3 p Use triplet loss and cross entropy loss. The cross entropy loss function formula is as follows:

[0062]

[0063] in That is y i The output obtained after softmax activation is between 0 and 1. The cross entropy loss function is used to describe the distance between two probability distributions. For positive samples, the greater the output probability, the smaller the loss. For negative samples, the smaller the output probability, the smaller the loss. At the same time, the present invention introduces labelsmooth to alleviate the problems of too many negative samples and easy overfitting of the model, and improves the generalization ability of the model. It does not force the image category to be classified as 0 / 1, but performs a certain probability calculation on the image category. The probability calculation formula is as follows:

[0064]

[0065] Where N represents the total number of categories and ε is a small hyperparameter, taking 0.1.

[0066] The triplet loss function is used to shorten the distance between the same pedestrian images and increase the distance between different pedestrian images, where:

[0067]

[0068] Where D(:,:) represents the Euclidean distance between two images. Refers to the anchor, which is used as a sample of triple loss. Refers to positive, that is, Different pictures belonging to the same group of people, Refers to negative, that is, For pictures belonging to different rows, m represents the boundary distance. A boundary value of 0.3 is set for positive samples and negative samples.

[0069] Step 4.2, for f′ p1 Do cross entropy loss and use f′ p Make mean square error loss for target.

[0070] Among them, the mean square error loss is:

[0071]

[0072] In the formula, n represents the number of samples.

[0073] Step 4.3, according to f′ p1 , f′ p2 , f′ p3 Find the regional features with strong discrimination and shorten the distance between the features with strong discrimination and the global features. The Euclidean distance is used to calculate the distance between the features of different regions and the global features, and then the pair of features with the closest distance, that is, the feature with the strongest discrimination in the regional features and the global features, are shortened.

[0074]

[0075] Where D(:,:) represents the Euclidean distance between two images. represents the feature of the kth region, f p Represents global features.

[0076] Step 5: In the test phase, the original image and the image cut in step 1 are used as input to obtain the global features and local features of an image. The global features and local features are then fused to generate the test results. The specific process is as follows: Figure 4 shown.

[0077] Step 5.1, load the model trained in the training phase, input the test image, and after image preprocessing, extract the features of the image through the trained model.

[0078] Step 5.2, fuse the global features and local features. The fusion process is as follows Figure 5 As shown:

[0079] Step 5.2.a, input the image, which is divided into the original image and the cropped image to extract global features and local features respectively;

[0080] Step 5.2.b, weight the global features and local features according to the fusion strategy to obtain the fused features. Figure 6 As shown. Input the cut picture as Figure 7 As shown,

[0081] The fusion strategy:

[0082] P′=P+P 2-1 +0.5×P 2-2 +P 3-1 +0.5×(P 3-2 +P 3-3 )

[0083] In step 5.3, the fused test features and the fused features of the query image are sorted according to cosine similarity, and finally the test results are saved and the test ends.

Claims

1. A method for re-identifying pedestrians after changing clothes, characterized in that: The following steps are involved: Step 1: Collect experimental data and divide the images in the data set by image cutting, save and add them to the original data set; Step 2: Before the training phase begins, the input image is preprocessed; Step 3, training phase, extracts the global features and key features of human body posture of the input image, and fuses the global features with the key features of human body posture; specifically includes the following steps: Step 3.1: Input the image into the feature extraction network A to obtain feature f g , and input the image into the feature extraction network B to obtain the feature f p 、f p1 、f p2 、f p3 ; Among them, the feature extraction network A adopts the ResNet50 network structure, and the feature extraction network B adopts the HRNet network structure; the feature extraction network B extracts the feature f p is the complete human posture key point feature, feature f p1 、f p2 、f p3 They are the key point features of the head, the key point features of the upper body, and the key point features of the lower body after distinguishing the key points of the human body posture; Step 3.2, feature f g After feature enhancement, we get f′ g ; Step 3.3, f′ g With feature f p 、f p1 、f p2 、f p3 Multiply pixel by pixel to get the overall feature f′ that integrates the key points of human posture p , head key point feature f′ p1 , upper body key point features f′ p2 , lower body key point features f′ p3 ; Step 4: training the network to differentiate the weights of different human body regions; Step 5, the testing phase, uses the fusion features of the global features and local features of the image to generate the test results.

2. The method according to claim 1, characterized in that The image cutting described in step 1 is to cut the original image into two equal parts and three equal parts at the height.

3. The method according to claim 1, characterized in that The image preprocessing described in step 2 is performed according to the following steps: Step 2.1: Resize the input image e operate; Step 2.2, perform flipping operation on the input image; Step 2.3, perform random erasing operation on the input image.

4. The method according to claim 1, characterized in that The feature enhancement described in step 3.2 is implemented as follows: Step 3.2.a, input feature f g First, the maximum value of each key point in the key point feature of the human body posture is multiplied pixel by pixel to obtain the mask feature; Step 3.2.b, input feature f g It passes through 1×1 convolution, BatchNorm layer and Relu layer in sequence; Step 3.2.c, pass the features obtained in step 3.2.b through the Multi Head SelfAttention mechanism, BatchNorm layer and Relu layer in sequence; Step 3.2.d, pass the features obtained in step 3.2.c through 1×1 convolution and BatchNorm layers in sequence; Step 3.2.e, add the features obtained in step 3.2.d and the mask features obtained in step 3.2.a, and after passing through a Relu layer, obtain the enhanced feature f′ g .

5. The method according to claim 1, characterized in that The specific implementation method of training the network in step 4 so as to distinguish the weights of different human body regions is as follows: Step 4.1, for the feature f′ obtained in step 3 p Use triplet loss and cross entropy loss; Step 4.2, for f′ p1 Do cross entropy loss and use f′ p Make mean square error loss for target; Step 4.3, according to f′ p1 , f′ p2 , f′ p3 Find the regional features with strong discriminability and bring the distance between the features with strong discriminability and the global features closer.

6. The method according to claim 1, characterized in that To generate the test results described in step 5, follow these steps: Step 5.1, load the model trained in the training phase, input the test image, and after image preprocessing, extract the features of the image through the trained model; Step 5.2, fusing global features and local features; Step 5.3, compare and sort the fused test features and the features of the query image according to cosine similarity, and save the test results.

7. The method according to claim 6, characterized in that The feature fusion described in step 5.2 is implemented as follows: Step 5.2.a, input the image, which is divided into the original image and the cropped image, which are used to extract global features and local features respectively; In step 5.2.b, the global features and local features are weighted according to the fusion strategy to obtain the fused features.

Images