Occluded person re-identification method based on feature enhancement and noise channel removal

Abstract

The present invention discloses a method for re-identifying occluded pedestrians based on feature enhancement and noise channel removal. A special occlusion generator is used to generate images with occlusions. During the training phase, a foreground generation feature enhancement module uses the generated pseudo foreground to enhance the previous features. The pseudo foreground binary cross entropy loss supervises the generated pseudo pedestrian foreground during training. During the testing phase, a noise channel discarding method further reduces the impact of occlusion on the final metric distance calculation, thereby improving the recognition accuracy.

Description

Technical Field

[0001] The present invention relates to a pedestrian re-identification method, in particular to an occluded pedestrian re-identification method based on feature enhancement and noise channel removal. Background Art

[0002] Person re-identification aims to determine whether pedestrian images from different cameras belong to the same person. Due to the huge demand in smart cities, intelligent security, cross-border tracking, etc., the pedestrian re-identification task has attracted more and more researchers to study it. In recent years, with the rapid development of deep learning and the release of large-scale datasets, pedestrian re-identification technology has made rapid progress. However, most pedestrian images in large datasets are fully visible, while pedestrian images in actual shooting are often interfered with by occlusions. Since most of the previously proposed methods did not take this situation into account, directly applying the previous methods to this task will cause a significant loss of accuracy. Therefore, the occluded pedestrian re-identification task introduces new challenges to the direction of pedestrian re-identification. Summary of the Invention

[0003] The purpose of the present invention is to provide a method for re-identifying occluded pedestrians based on feature enhancement and noise channel removal to solve the problems raised in the above background technology.

[0004] The object of the present invention is to achieve this through the following technical solution: a method for re-identifying pedestrians from obscured spaces based on feature enhancement and noise channel removal, comprising a training phase and a testing phase, wherein the steps of the training phase are as follows:

[0005] Step 1: Start by inputting the original image;

[0006] Step 2: Data augmentation: augment the original image with data through the occlusion generator;

[0007] Step 3: Input the image into the network, and input the image after data amplification into the network;

[0008] Step 4: Calculate the loss and propagate backward. The image input to the network is passed through the foreground feature enhancement module to calculate the loss, and the image after loss calculation is propagated backward.

[0009] Step 5: When the number of iterations is reached, the propagated image is iterated and output after the required number of iterations is reached;

[0010] The steps of the testing phase are as follows:

[0011] Step 1: Start by inputting the original image;

[0012] Step 2: Input the image into the network and input the original image into the network;

[0013] Step 3: Feature extraction, feature extraction of the input image and the searched image;

[0014] Step 4: Channel discarding and refining features, through the noise channel discarding method, the final refined features are obtained;

[0015] Step 5: Metric distance calculation, using the two final refined features to calculate the cosine metric distance;

[0016] Furthermore, the occlusion generator in step 2 of the training phase modifies the random patch erasing method. Instead of replacing the original area with the patch after scaling, the patch is linearly superimposed with the content of the original area. This is called a hybrid occlusion method and consists of the following steps:

[0017] (101) Input the original image I and various parameters;

[0018] (102) Randomly select an area to be blocked and set the content therein to be I0;

[0019] (103) Randomly select a patch from the original image, whose area is S P ;

[0020] (104) Put S P Use the interpolation algorithm to scale to the size of S0, and let the content obtained after scaling be I P ;

[0021] (105) Replace the content of area S0 in the pedestrian image I with the superposition of the original content and the patch content

[0022] (106) outputting the final pedestrian image with occlusion;

[0023] Furthermore, the foreground feature enhancement module in step 4 of the training phase is composed of the following steps:

[0024] (107) Given an original pedestrian image I, Resnet50 is used as the basic network skeleton to generate a pedestrian pseudo foreground PF through the FG module;

[0025] (108) Multiply the pseudo foreground PF by the feature map of the third layer of the previous network skeleton and propagate it backward to further enhance the extraction of foreground information, strengthen the flow of gradients to the foreground area, further eliminate the interference of occlusion noise, and enhance the final recognition accuracy;

[0026] (109) The feature map is weighted with the foreground heat map H to obtain a feature vector;

[0027] (110) In order to effectively extract pedestrian foreground information, pedestrian identity classification loss and triplet loss are used to supervise this feature vector;

[0028] Furthermore, generating accurate pedestrian foregrounds is crucial for the foreground generation feature enhancement module. Since the person re-identification dataset lacks real pedestrian foreground labels, the prior information of the network can be used to generate a self-supervisory signal to supervise the pedestrian foreground. The self-supervisory signal consists of the following steps:

[0029] (111) First, sum F4 along the channel direction to obtain B;

[0030] (112) The area with larger value in the feature activation map usually corresponds to the foreground area, and the area with smaller value in the feature activation map usually corresponds to the background area. The maximum value point in B is set to 1, and the minimum value point is set to 0 to obtain the self-supervisory signal D;

[0031] (113) Using the self-supervisory signal D to supervise the foreground H generated by the foreground generation module, the supervisory signal used is binary cross entropy loss;

[0032] Furthermore, the noise channel discarding method in step 4 of the test phase is mainly to calculate a channel mask based on the extracted feature vector, discard the feature channel with the channel mask to obtain the final refined feature, and then use the two features to calculate the cosine metric distance. The specific steps are as follows:

[0033] (114) obtaining the query image Iq and the image to be searched Ig, and obtaining the feature vectors fq and fg of the query image and the image to be searched, respectively, through the forward process of the network;

[0034] (115) A certain ratio of the average value of the feature vector is used as a mask. The area greater than this threshold is set to the original value, and the area less than this threshold is set to 0. The feature vector of the query image and the feature vector of the gallery image can both extract their corresponding masks;

[0035] (116) Refine the previous feature vector according to the mask and discard the noisy channels;

[0036] (117) The cosine metric distance is calculated based on the refined features.

[0037] The beneficial effects of the present invention are: this occluded pedestrian re-identification method based on feature enhancement and noise channel removal generates an occluded picture through a special occlusion generator, and in the training stage, the foreground generation feature enhancement module uses the generated pseudo foreground to enhance the previous features, and the pseudo foreground binary cross entropy loss supervises the generated pseudo pedestrian foreground during training. In the test stage, the noise channel discarding method further reduces the impact of occlusion on the final metric distance calculation, thereby improving the recognition accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 is a flow chart of the training phase of the present invention;

[0039] Figure 2 is a flow chart of the testing phase of the present invention;

[0040] Figure 3 Schematic diagram of the data amplification method of the present invention and several other methods;

[0041] Figure 4 Comparison of the foreground heatmaps generated without using the pseudo foreground binary cross entropy loss and with using the pseudo foreground binary cross entropy loss. DETAILED DESCRIPTION

[0042] The following is further described in conjunction with specific embodiments, but the specific embodiments below should not be understood as limiting the invention. Various changes and modifications that can be made by ordinary technicians in this field based on the present invention should all be within the scope of the invention.

[0043] like Figures 1 to 4 A pedestrian re-identification method under occlusion based on feature enhancement and noise channel removal includes a training phase and a testing phase. The steps of the training phase are as follows:

[0044] Step 1: Start by inputting the original image;

[0045] Step 2: Data augmentation: augment the original image with data through the occlusion generator;

[0046] Step 3: Input the image into the network, and input the image after data amplification into the network;

[0047] Step 4: Calculate the loss and propagate backward. The image input to the network is passed through the foreground feature enhancement module to calculate the loss, and the image after loss calculation is propagated backward.

[0048] Step 5: When the number of iterations is reached, the propagated image is iterated and output after the required number of iterations is reached;

[0049] The steps of the testing phase are as follows:

[0050] Step 1: Start by inputting the original image;

[0051] Step 2: Input the image into the network and input the original image into the network;

[0052] Step 3: Feature extraction, feature extraction of the input image and the searched image;

[0053] Step 4: Channel discarding and refining features, through the noise channel discarding method, the final refined features are obtained;

[0054] Step 5: Metric distance calculation, using the two final refined features to calculate the cosine metric distance;

[0055] It is worth mentioning that the occlusion generator in step 2 of the training phase modifies the random patch erasing method. Instead of replacing the original area with the patch after scaling it, it linearly superimposes it with the content of the original area. This is called a hybrid occlusion method and consists of the following steps:

[0056] (101) Input the original image I and various parameters;

[0057] (102) Randomly select an area to be blocked and set the content therein to be I0;

[0058] (103) Randomly select a patch from the original image, whose area is S P ;

[0059] (104) Put S P Use the interpolation algorithm to scale to the size of S0, and let the content obtained after scaling be I P ;

[0060] (105) Replace the content of area S0 in the pedestrian image I with the superposition of the original content and the patch content

[0061] (106) outputting the final pedestrian image with occlusion;

[0062] It is worth mentioning that the foreground feature enhancement module in step 4 of the training phase consists of the following steps:

[0063] (107) Given an original pedestrian image I, Resnet50 is used as the basic network skeleton to generate a pedestrian pseudo foreground PF through the FG module;

[0064] (108) Multiply the pseudo foreground PF by the feature map of the third layer of the previous network skeleton and propagate it backward to further enhance the extraction of foreground information, strengthen the flow of gradients to the foreground area, further eliminate the interference of occlusion noise, and enhance the final recognition accuracy;

[0065] (109) The feature map is weighted with the foreground heat map H to obtain a feature vector;

[0066] (110) In order to effectively extract pedestrian foreground information, pedestrian identity classification loss and triplet loss are used to supervise this feature vector;

[0067] It is worth mentioning that generating accurate pedestrian foreground is crucial for the foreground feature enhancement module. Since the pedestrian re-identification dataset lacks real pedestrian foreground labels, the prior information of the network can be used to generate a self-supervisory signal to supervise the pedestrian foreground. The self-supervisory signal consists of the following steps:

[0068] (111) First, sum F4 along the channel direction to obtain B;

[0069] (112) The area with larger value in the feature activation map usually corresponds to the foreground area, and the area with smaller value in the feature activation map usually corresponds to the background area. The maximum value point in B is set to 1, and the minimum value point is set to 0 to obtain the self-supervisory signal D;

[0070] (113) Using the self-supervisory signal D to supervise the foreground H generated by the foreground generation module, the supervisory signal used is binary cross entropy loss;

[0071] It is worth mentioning that the noise channel discarding method in step 4 of the test phase is mainly to calculate the channel mask based on the extracted feature vector, discard the feature channel with the channel mask, obtain the final refined feature, and then use the two features to calculate the cosine metric distance. The specific steps are as follows:

[0072] (114) obtaining the query image Iq and the image to be searched Ig, and obtaining the feature vectors fq and fg of the query image and the image to be searched, respectively, through the forward process of the network;

[0073] (115) A certain ratio of the average value of the feature vector is used as a mask. The area greater than this threshold is set to the original value, and the area less than this threshold is set to 0. The feature vector of the query image and the feature vector of the gallery image can both extract their corresponding masks;

[0074] (116) Refine the previous feature vector according to the mask and discard the noisy channels;

[0075] (117) The cosine metric distance is calculated based on the refined features.

[0076] Specific implementation process: The present invention is mainly tested on two datasets, including Occluded ReID and Occluded Duke. Due to the small size of the Occluded ReID dataset, we use the Market1501 dataset for training and then test on these two datasets. Partial ReID contains 900 pictures, and each ID has 5 full-body pictures, 5 half-body pictures and 5 occluded pictures. Since the task is to re-identify occluded pedestrians, occluded pictures are used as queries for testing; the Occluded ReID dataset has 200 IDs, and each ID has 5 occluded pictures and 5 full-body pictures; the measurement standards commonly used in pedestrian re-identification tasks are used, including the Cumulative Match Characteristic (CMC), including Rank1 (first hit rate), Rank5, average average precision (meanAverage Precision mAP); using Resnet50 as the network backbone, similar to most pedestrian re-identification methods, etc., the step size of the fourth module of Resnet50 is changed from 2 to 1; using the SGD optimizer, the momentum is set to 0.9, the weight decay is set to 5e-4, and the learning rate warm-up strategy is used. In the first 10 epochs, the learning rate is increased from 3e-6 to 3e-4, and then the learning rate is reduced from 3e-4 to 3e-6. The learning rate is changed to the original 0.1 in the 40th and 70th epochs respectively, and the learning rate is kept unchanged in other epochs. A total of 90 epochs are trained; each batch contains 16 ID pedestrians, and each ID pedestrian contains four pictures. The occlusion augmentation method is used as the data augmentation method. In addition, in order to reduce cross-domain bias, for the Occluded ReID dataset, we use color jittering to reduce the cross-domain impact.

[0077] In the implementation plan of the training phase, we first introduce the specific method used by the occlusion generator. The methods that were widely used before were random erasing and random patch occlusion. The random erasing method randomly selects a certain area in the pedestrian image and replaces the pixel values ​​in it with random pixel values. The random patch occlusion method uses a small patch in the pedestrian image to scale to the size of the occlusion area and replaces the original content of the occlusion area. This method can obtain richer occlusion content than the random erasing method. These two methods have obvious disadvantages, because the original content must be blocked. If the content of the occluded area covers all parts of the human body, the occluded image will not have effective distinguishing information, similar to the background noise image, which may affect the accuracy of network recognition. Therefore, we modify the previous random patch erasing method. Instead of replacing the original area with the patch after scaling, we linearly superimpose it with the content of the original area; we call it hybrid erasing; several data augmentation methods such as Figure 3 Even if our method covers all pedestrian areas in the pedestrian image, it still has certain discriminative information. In addition, it may also have a certain effect on data amplification when changing clothes. Other methods may cover the entire human body area. The specific steps of our proposed hybrid occlusion method are as follows:

[0078] Input the original image I and various parameters, including the patch size s used for occlusion, the occlusion area ratio [r1, r2], the blending ratio λ, and the aspect ratio of the image in the area to be occluded [a1, a2];

[0079] (1) Randomly select an area S to be blocked o ∈[S I ×r1,S I ×r2], randomly select an aspect ratio A∈[a1, a2], calculate the width and height of the occluded area according to the area S0 and the aspect ratio A of the occluded area. If the width and height are smaller than the width and height of the image, it is a reasonable choice. Let the content be I0;

[0080] (2) Randomly select a patch from the original image, whose area size is S P =s×s;

[0081] (3) Put S P Use the bicubic interpolation algorithm to scale to the size of S0, and let the content obtained after scaling be I P ;

[0082] (4) Replace the content of the area in the pedestrian image I with λ×I P +(1-λ)×I O ;

[0083] (5) Output the final pedestrian image with occlusion;

[0084] In the implementation plan of the training phase, we introduce the foreground generation feature enhancement module (FGFE) mentioned above. First, we introduce the forward process of the FGFE module; its specific steps are as follows:

[0085] (1) Given an original pedestrian image I, we use Resnet50 as the basic network skeleton and pass the last layer feature map F4 of Resnet50 through the pseudo foreground generation module FG to generate a pseudo foreground PF = FG (F4, θ) where θ is the parameter of the FG module, including three convolutional layers and their corresponding batch normalization layers (BN) and ReLU activation layers;

[0086] (2) Then multiply the pseudo foreground PF by the feature map F3 of the third layer of the previous network skeleton to extract the foreground feature F fore3 =F3*F, and then propagate F backward fore4 =Resnet block4 (F fore3 ), further strengthen the extraction of foreground information, strengthen the flow of gradients to the foreground area, further eliminate the interference of occlusion noise, and enhance the final recognition accuracy;

[0087] (3) We weight the feature map by the foreground heat map H after Softmax by F4 to obtain a feature vector f 4H , specifically f H4 =sum(F4⊙H), where ⊙ is element-wise multiplication and sum is the sum along the height and width directions;

[0088] (4) Then, in order to effectively extract the relevant information of the pedestrian foreground, we use the pedestrian identity classification loss and triplet loss to supervise this feature vector. Both losses are widely used in pedestrian re-identification tasks.

[0089] As mentioned earlier, the foreground generation feature enhancement module mainly uses the foreground generated by the foreground generation module to enhance the final foreground features. Therefore, generating accurate pedestrian foregrounds is crucial. To generate better pedestrian foregrounds, we use the network's prior information to generate a self-supervisory signal to supervise the pedestrian foreground. The specific steps are as follows:

[0090] (1) First, sum F4 along the channel direction to obtain B. The shape of F4 is C*H*W, where C is the number of feature channels, H and W are the height and width of the feature map, so the shape of B is H*W;

[0091] (2) The area with larger value in the feature activation map usually corresponds to the foreground area, and the area with smaller value in the feature activation map usually corresponds to the background area. We set the maximum value point in B to 1 and the minimum value point to 0 to obtain the self-supervisory signal D;

[0092] (3) The foreground H generated by the foreground generation module is supervised by the self-supervisory signal D. The supervisory signal used is the binary cross entropy loss, which we call the pseudo-foreground binary cross entropy loss (PFBC). Its specific form is as follows:

[0093]

[0094] It is an indicator function. When the input is not -1, the output is 1, and when the input is -1, the output is 0.

[0095]

[0096] Figure 4 The figure shows a comparison of the foreground heatmaps generated without using the pseudo foreground binary cross entropy loss and with using this loss. As can be seen from the figure, the foreground heatmap generated with this loss (the third row) has a darker color in the pedestrian area than the heatmap generated without using this loss (the second row), which means that the generated pedestrian images are more discriminative and focus more on the pedestrian foreground area.

[0097] During the testing phase of this patent, compared with general pedestrian re-identification methods, a noise channel discarding method was added to further reduce the interference of occlusion noise on the final recognition. The specific steps are as follows:

[0098] (1) Obtain the query image Iq and the search image Ig; obtain the feature vectors fq and fg of the query image and the search image respectively through the forward process of the network;

[0099] (2) A certain ratio of the average value of the feature vector is used as a mask. The area greater than this threshold is set to the original value, and the area less than this threshold is set to 0. The feature vector of the query image and the feature vector of the search image can both extract their corresponding masks. The specific operation is as follows:

[0100] m q =mean(f q ), mask q =f q >(λ q *m q )

[0101] m g =mean(f g ), mask g =f g >(λ g *m g )

[0102] mask=mask q *maskg

[0103] here

[0104] (3) According to the Mask, refine the previous feature vector and set the refined feature as f fq and f fg The specific calculation method is as follows:

[0105] f fq =f q ⊙mask

[0106] f fg =f g ⊙mask

[0107] (4) Calculate the cosine metric distance based on the refined features;

[0108] dist=cosine(f fq , f fg )

[0109] The cosine(.,.) function calculates the cosine distance between two eigenvectors.

[0110] The present invention gradually enhances the robustness of the network to occlusion interference and improves the performance of the occluded pedestrian re-identification task through the design of a series of modules and loss functions. Table 1 shows the comparison of the Rank1, Rank5 and mAP indicators of the proposed method and the best method on the Occluded re-id dataset and the Occluded Duke dataset. It can be seen from the results of Table 1 that the proposed method achieves excellent performance and surpasses other methods in all three indicators. In addition, we also designed an ablation experiment, as shown in Table 2. The ablation experiment in Table 2 shows that our carefully designed series of modules and loss functions have a certain improvement effect on the final result and can improve the performance to a certain extent. In addition, this example does not utilize additional key points or human semantic parsing datasets, does not require an additional detection network, and is also conducive to reducing cross-domain bias.

[0111] Table 1

[0112]

[0113] Table 2

[0114]

[0115] The above description is merely a preferred embodiment of the present application and an illustration of the technical principles employed. Those skilled in the art should understand that the scope of the invention herein is not limited to the technical solutions formed by the specific combination of the above-mentioned technical features, but also encompasses other technical solutions formed by any combination of the above-mentioned technical features or their equivalents without departing from the inventive concept. For example, a technical solution formed by replacing the above-mentioned features with (but not limited to) technical features having similar functions disclosed in this application.

Claims

1. A method for occluded person re-identification based on feature enhancement and noise channel removal, comprising a training phase and a testing phase, characterized by: The steps of the training phase are as follows: Step 1: Start by inputting the original image; Step 2: Data augmentation: augment the original image with data through the occlusion generator; Step 3: Input the data-amplified image into the network; Step 4: Calculate the loss and propagate backward. The image input to the network is passed through the foreground feature enhancement module to calculate the loss, and the image after loss calculation is propagated backward. Step 5: Iterate the propagated image and output it after reaching the required number of iterations; The foreground generation feature enhancement module in step 4 of the training phase operates as follows: (107) Given an original image I, Resnet50 is used as the basic network skeleton to generate pedestrian pseudo foreground PF through the pseudo foreground generation module; (108) Multiplying the pseudo foreground PF by the feature map of the third layer of the network skeleton and then propagating it backward; (109) The feature map is weighted with the foreground heat map H to obtain a feature vector; (110) This feature vector is supervised using pedestrian identity classification loss and triplet loss; The steps of the testing phase are as follows: Step 1: Start by inputting the original image; Step 2: Input the original image into the network; Step 3: Feature extraction, feature extraction of the input image and the searched image; Step 4: Channel discarding and refining features, through the noise channel discarding method, the final refined features are obtained; Step 5: Metric distance calculation, using the two final refined features to calculate the cosine metric distance; The noise channel discarding method in step 4 of the testing phase is to calculate a channel mask based on the extracted feature vector, discard the feature channel with the channel mask to obtain the final refined feature, and then calculate the cosine metric distance using the two features. The specific steps are as follows: (114) obtaining the query image Iq and the image to be searched Ig, and obtaining the feature vectors fq and fg of the query image and the image to be searched, respectively, through the forward process of the network; (115) A certain ratio of the average value of the feature vector is used as the threshold, and the feature vector is binarized to generate a mask containing only 0 and 1. The mask corresponding to the feature vector of the query image and the feature vector of the search image are respectively denoted as mask q and mask g ; (116) According to the mask, the feature vectors fq and fg are refined, and the channels with noise are discarded. The mask is composed of mask q and mask g Generate, as shown in the formula: mask = mask q *mask g ; (117) The cosine metric distance is calculated based on the refined features.

2. The method for occluded person re-identification based on feature enhancement and noise channel removal according to claim 1, characterized in that: The occlusion generator in step 2 of the training phase modifies the random patch erasing method. Instead of replacing the original area with the patch after scaling, it linearly superimposes the patch with the content of the original area. This is called a hybrid occlusion method and consists of the following steps: (101) Input the original image I and various parameters; (102) Randomly select an area S0 to be blocked and set the content therein to be I0; (103) Randomly extract a patch area S from the original image I P ; (104) Put S P Use the interpolation algorithm to scale to the size of S0, and let the content obtained after scaling be I P ; (105) Replace the content of the S0 region in the original image I with I0 and I P Linear superposition of ; (106) Output the final pedestrian image with occlusion.

3. The method for occluded person re-identification based on feature enhancement and noise channel removal according to claim 1, characterized in that: Using the network's prior information to generate a self-supervisory signal to supervise the pedestrian foreground consists of the following steps: (111) First, sum F4 along the channel direction to obtain B; F4 represents the last layer feature map of the original image I after Resnet50, and B represents the feature activation map; (112) Set the maximum value point in B to 1 and the minimum value point to 0 to obtain the self-supervisory signal D; (113) The foreground heat map H generated by the foreground generation module is supervised by the self-supervisory signal D, and the supervisory signal used is the binary cross entropy loss.

Images