A monitoring method and system for occluding pedestrians

Abstract

The present invention discloses a monitoring method and system for occluded pedestrians. The method comprises the following steps: collecting pedestrian images to be identified; extracting features of the collected pedestrian images through extractor ResNet50 to obtain pedestrian image feature maps; erasing the feature maps using a cross-erasing strategy, constructing an adversarial loss function, and iteratively optimizing the parameters of the feature map cross-erasing module; inputting the erased feature maps into the occlusion-aware hybrid attention module to make the network focus on the non-occluded area of ​​the pedestrians; using the full-view enhancement module to expand the network receptive field to obtain the final identification features of the pedestrian images, constructing a baseline loss function, and iteratively optimizing the parameters of the network; performing similarity measurement between the extracted final identification features of the pedestrian images and each pedestrian image in the image library, and outputting a matching list. The present invention effectively solves the problem of pedestrian occlusion in monitoring scenarios.

Description

Technical Field

[0001] The invention relates to a monitoring method and system for occluding pedestrians, and belongs to the technical field of computer vision. Background Art

[0002] With the rapid development of intelligent surveillance equipment and the increase in public safety needs, a large number of cameras are deployed in public places such as airports, communities, streets and campuses. These camera networks usually span large geographical areas with non-overlapping coverage, generating a large number of surveillance videos every day. Among them, pedestrians are one of the most important subjects in surveillance videos. How to analyze, identify, locate and find pedestrians in surveillance videos has received widespread attention in academia. However, in real environments, pedestrian occlusion is inevitable. One of the current challenges is to accurately identify the same pedestrian in different camera perspectives when the target pedestrian is partially or completely occluded by other objects or pedestrians. This problem is particularly prominent in places with large traffic, such as airports, shopping malls, hospitals and other complex scenes.

[0003] Although the existing research on pedestrian monitoring methods and systems has been relatively mature in improving the accuracy of pedestrian retrieval, it ignores the problem of pedestrian occlusion and is difficult to apply in real scenes. Therefore, a monitoring method and system focusing on occluded pedestrians is urgently needed to solve the above problem. Summary of the invention

[0004] In view of the problems existing in the existing methods, the object of the present invention is to provide a monitoring method and system for obstructed pedestrians.

[0005] In a first aspect of an embodiment of the present invention, a monitoring method for blocking pedestrians is provided, comprising the following steps:

[0006] Collection end: collects pedestrian images to be identified;

[0007] Feature extraction end: The collected pedestrian images are subjected to feature extraction through the extractor ResNet50 to obtain the pedestrian image feature map;

[0008] Feature erasing end: adopt a cross-erasing strategy to erase the feature map, construct an adversarial loss function, and iteratively optimize the parameters of the feature map cross-erasing module;

[0009] Feature attention end: The erased feature map is input into the occlusion-aware hybrid attention module to make the network focus on the non-occluded area of ​​the pedestrian;

[0010] Feature enhancement end: Use the full-view enhancement module to expand the network receptive field, obtain the final identification features of pedestrian images, build a baseline loss function, and iteratively optimize the network parameters;

[0011] Monitoring center: perform similarity measurement on the final identification features of the extracted pedestrian image and each pedestrian image in the image library, and output a matching list.

[0012] Furthermore, the cross-erase strategy is a cross-mask operation.

[0013] Furthermore, the adversarial loss function is two adversarial cross entropy losses.

[0014] Furthermore, the occlusion-aware hybrid attention module consists of spatial attention and channel attention, including 3×3 convolution, 5×5 convolution, convolution block, BN, ReLU, Sigmoid, segmentation, pooling, splicing, compression, decompression, element-level multiplication and element-level addition operations.

[0015] Furthermore, the convolution block is composed of depth-wise separable convolutions, and the specific execution order is 1×1 convolution, lightweight 3×3 convolution, lightweight 3×3 convolution and 1×1 convolution.

[0016] Furthermore, the full field of view enhancement module adopts a multi-branch dilated convolution structure, including 1×1 convolution, 3×3 convolution, dilated convolution with different dilation coefficients, splicing, ReLU and element-level addition operations.

[0017] Furthermore, the dilated convolutions with different dilation coefficients are composed of four 3×3 dilated convolutions with different dilation coefficients, and the dilation coefficients are 1, 3, 5 and 7 respectively.

[0018] Furthermore, the baseline loss function is cross entropy loss and hard sample triplet loss.

[0019] Furthermore, the similarity measurement obtains the final matching result by calculating the cosine similarity between the features of the pedestrian images, and generates a matching list from large to small according to the cosine similarity.

[0020] In a second aspect of the embodiments of the present invention, a monitoring system for obstructed pedestrians is provided, which is used to execute the monitoring method for obstructed pedestrians in the first aspect.

[0021] Compared with the prior art, the present invention has the following beneficial effects:

[0022] 1. The present invention innovatively proposes a feature map cross-erasing module, which simulates the occlusion situation in the real scene by erasing the feature map operation, thereby enhancing the network's ability to handle occlusion.

[0023] 2. The present invention proposes a novel occlusion-aware hybrid attention module, which combines the designed spatial attention and channel attention to enable the network to focus on the non-occluded areas of pedestrians, helping the network to extract the discriminative features of pedestrian images.

[0024] 3. The present invention proposes a novel full-view enhancement module, which expands the network's receptive field by constructing a multi-branch dilated convolution structure, helps the network to more effectively parse the contextual information of pedestrian images, and mines the identification features of pedestrian images from this information.

[0025] 4. The effectiveness of the method of the present invention is verified on two occlusion datasets, Occluded-DukeMTMC and P-DukeMTMC-reID. In addition, to prove the generalization of the method of the present invention, we also conducted corresponding experiments on two common datasets, Market1501 and MSMT17. Comprehensive evaluation indicators are used to evaluate the accuracy and complexity of the network: including mAP and Rank-1. The experimental results under the two indicators fully demonstrate the effectiveness of the method of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Other features, objects and advantages of the present invention will become more apparent from the detailed description of non-limiting embodiments made with reference to the following drawings:

[0027] Figure 1 A flowchart of the steps of a monitoring method for obstructing pedestrians provided by the present invention;

[0028] Figure 2 The overall framework diagram of a monitoring method for occluding pedestrians provided by the present invention;

[0029] Figure 3 A schematic diagram of the structure of an occlusion-aware mixed attention module according to a preferred embodiment of the present invention;

[0030] Figure 4 A schematic structural diagram of a full-viewing enhancement module according to a preferred embodiment of the present invention. DETAILED DESCRIPTION

[0031] In order to enable those skilled in the art to more clearly understand the present invention, the following is described in conjunction with specific embodiments. The specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0032] like Figure 1 As shown, in a first aspect of an embodiment of the present invention, a flowchart of a method for monitoring pedestrians blocking pedestrians is provided, including:

[0033] S1, collection end: collect pedestrian images to be identified;

[0034] S2, feature extraction end: extract features of the collected pedestrian image through the extractor ResNet50 to obtain a pedestrian image feature map;

[0035] S3, feature erasing end: adopt a cross-erasing strategy to erase the feature map, construct an adversarial loss function, and iteratively optimize the parameters of the feature map cross-erasing module;

[0036] S4, feature attention end: the erased feature map is input into the occlusion-aware hybrid attention module to make the network focus on the non-occluded area of ​​the pedestrian;

[0037] S5, feature enhancement end: use the full-view enhancement module to expand the network receptive field, obtain the final identification features of the pedestrian image, build a baseline loss function, and iteratively optimize the network parameters;

[0038] S6, monitoring center: perform similarity measurement on the final identification features of the extracted pedestrian image and each pedestrian image in the image library, and output a matching list.

[0039] like Figure 2 FIG. 1 is an overall framework diagram of a monitoring method for occluded pedestrians according to an embodiment of the present invention, which mainly includes three modules: a feature map cross-erasing module, an occlusion-aware hybrid attention module, and a full-view enhancement module.

[0040] The present invention provides a preferred embodiment to perform S1. The purpose of this embodiment is to collect images of pedestrians to be identified.

[0041] The collection end is generally a surveillance camera installed in different locations. Here we use two occlusion datasets Occluded-DukeMTMC and P-DukeMTMC-reID and two common datasets Market1501 and MSMT17 for experiments.

[0042] The present invention provides a preferred embodiment to perform S2. The purpose of this embodiment is to extract features from the collected pedestrian image through the extractor ResNet50 to obtain a pedestrian image feature map.

[0043] like Figure 2 As shown, for the two input pedestrian images and First, ResNet50 is used as an extractor to generate the corresponding feature map and , where H, W, and C represent the height, width, and number of channels, respectively. Then, we perform a cross-erasing strategy on the two feature maps to simulate the occlusion in real scenes.

[0044] The present invention provides a preferred embodiment to perform S3. The purpose of this embodiment is to simulate the occlusion situation in the real scene by erasing the feature map operation, so as to enhance the ability of the network to handle occlusion.

[0045] S31, cross-erasing strategy: According to practical experience, occlusion often appears in the lower half of the pedestrian image, and the occlusion area often occupies half of the pedestrian image area. To this end, the feature map cross-erasing module we proposed simulates the above situation by cutting out "occlusions" from other pedestrian image feature maps. Pedestrian image feature map and , we randomly cut out an area of Patch, that is and .in , , , , , represents a uniform distribution. Next, we will Zoom to Half of the area is then used as a mask to cover The bottom of ;Will Zoom to Half of the area is then pasted as a mask The bottom of , to simulate the above occlusion situation. Then, and They will be sent to the occlusion-aware hybrid attention module for further feature extraction and training. express.

[0046] S32, construct adversarial loss function: In order to train the feature map cross-erasing module, we borrowed the idea of ​​generative adversarial network and adopted an adversarial training method to simultaneously optimize the extractor and classifier 1. Specifically, we optimize the classifier 1 by maximizing the cross entropy loss and optimize the extractor by minimizing the cross entropy loss. The formula is as follows:

[0047] ,

[0048] ,

[0049] in, N represents the batch size, is the true identity label, Is a class The identity prediction score of and Related, is a constant, set to 0.1. When optimizing classifier 1, we adopted the method of maximizing the cross entropy loss, which enhanced the discrimination ability of classifier 1 and increased its probability of correct classification. In the optimization of the feature extractor, we are committed to minimizing the cross entropy loss to ensure that the feature map can still be accurately recognized by classifier 1 after the erasing operation. Through continuous adversarial training between the extractor and classifier 1, we can continuously mine rich discriminative information and enhance the network's ability to resist misidentification caused by occlusion.

[0050] The present invention provides a preferred embodiment to perform S4. The purpose of this embodiment is to combine the designed spatial attention and channel attention so that the network focuses on the non-occluded area of ​​the pedestrian.

[0051] S41, such as Figure 3 As shown, we will input the features F It is divided into three branches for processing. The first branch directly processes the feature map obtained by the feature map cross-erasing module. Specifically, F After channel attention CA, we get , CA is the result of our optimization of ECA. Compared with ECA, an element-level addition operation is added. This improvement helps the network maintain stability while reducing the risk of overfitting. The formula is as follows:

[0052]

[0053] S42, in order to obtain feature information of different scales and help the network to more accurately identify pedestrian non-occluded areas. F Apply 3×3 convolution and 5×5 convolution respectively to get and , then and Through spatial attention SA and

[0054] ,

[0055]

[0056] Next we will Let's take SA as an example. First, we will Segmentation along the height dimension yields and , the formula is as follows:

[0057] ,

[0058] in Represents the segmentation function. Subsequently, we perform global average pooling and global maximum pooling on the segmented feature map along the channel dimension, and then concatenate along the height dimension to obtain the feature map , the formula is as follows:

[0059] ,

[0060] in represents global average pooling, represents the global maximum pooling, represents the concatenation operation. Then, after each convolution block, we get , and then after BN, ReLU and Sigmoid, we get , the formula is as follows:

[0061] ,

[0062] in, express convolution, Indicates lightweight We will and Perform element-wise multiplication to get , and then Add element by element to get , the formula is as follows:

[0063] ,

[0064] in, represents element-wise addition, Represents element-wise multiplication.

[0065] S43, then we will get the characteristics and First, add element by element to get Then, with the first branch, we get Splice to get , the formula is as follows:

[0066]

[0067] S44, finally, the noise-free features With input features Add element by element to get , the formula is as follows:

[0068] .

[0069] The present invention provides a preferred embodiment to perform S5. The purpose of this embodiment is to expand the receptive field of the network by constructing a multi-branch dilated convolution structure, help the network to more effectively parse the context information of pedestrian images, and construct a baseline loss function to iteratively optimize the parameters of the network.

[0070] S51, such as Figure 4 As shown in the figure, we first made innovative improvements based on RFB and added a dilated convolution branch with a dilation factor of 7. This helps to further expand the network receptive field and enable the network to deeply explore the deeper features of pedestrian images.

[0071] S51, such as Figure 4 As shown in the figure, we first made innovative improvements based on RFB and added a dilated convolution branch with a dilation factor of 7. This helps to further expand the network receptive field and enable the network to deeply explore the deeper features of pedestrian images.

[0072] S52,At the same time, we adopt the technical solution in the updated version of RFB in this newly added branch, that is, replacing one 5×5 convolution with two 3×3 convolutions, which not only increases the depth of the network and helps the network to extract deeper features, but also effectively reduces the amount of parameters.

[0073] S53, Next, we added an additional shortcut branch based on RFB. This branch is different from the shortcut branch in RFB. It is directly built between the previous layer and the next layer. This design helps solve the gradient explosion problem in deep networks and allows the gradient to flow more freely in the network. At the same time, this also helps the network remain stable and reduces the risk of overfitting.

[0074] S54, construct baseline loss function: To train the network, we use cross entropy loss and hard sample triplet loss for supervision, the formula is as follows:

[0075] ,

[0076] Among them, P is a randomly selected pedestrian with different identities, Q is the number of randomly selected images for each identity, and PQ represents the batch size. and are the distances between positive and negative pairs from the same identity, is the boundary, initialized to 0.3.

[0077] Based on the above embodiments, the present invention provides a preferred embodiment S6, which measures the similarity between pedestrian image features by calculating cosine similarity to obtain the final matching result, and generates a matching list from large to small cosine similarity. The cosine similarity calculation is as follows:

[0078] ,

[0079] in, and is the feature vector.

[0080] In this example, we use ResNet50 pre-trained on ImageNet as the extractor, and remove the global average pooling layer and the fully connected layer at the end of the model. The input image is resized to 256×128 pixels. The network is trained for 120 epochs on the four datasets. We randomly selected 16 identities and selected 4 images for each identity in each training batch. Using the hard sample mining method, we determined the relevant positive and negative samples for the selected pedestrian images, a total of three images, and performed two trainings, one for the anchor image and positive samples, and the other for the anchor image and negative samples. The Adam optimizer is used for optimization. The learning rate is increased from 10% after the 10th epoch. Increase linearly to , and decay to and .

[0081] The model accuracy is evaluated by Rank-n and mean Average Precision (mAP) in Cumulative Matching Characteristics (CMC). Rank-n in CMC represents the probability that the first n images in the pedestrian matching result list successfully match the query image. mAP represents the average value of the sum of the average precision AP (Average Precision) in multi-classification tasks.

[0082] Table 1 shows the Rank-1 and mAP accuracy of the proposed method and the SOTA method for solving occlusion on the Occluded-DukeMTMC, P-DukeMTMC-reID, Market1501, and MSMT17 datasets. It can be seen that the proposed method achieves excellent recognition accuracy on the Occluded-DukeMTMC and P-DukeMTMC-reID datasets, which verifies that our proposed method has excellent ability to solve the problem of pedestrian occlusion, and on the common datasets Market1501 and MSMT17, it also achieves recognition accuracy comparable to other SOTA methods, which verifies the generalization of the proposed method.

[0083] Table 1 Comparison of the proposed method with the SOTA method on Occluded-DukeMTMC, P-DukeMTMC-reID, Market1501 and MSMT17 datasets (the best results are shown in bold)

[0084] ,

[0085] Table 1 shows the Rank-1 and mAP accuracy of the proposed method and other SOTA methods on the Occluded-DukeMTMC, P-DukeMTMC-reID, Market1501, and MSMT17 datasets. The following observations can be obtained:

[0086] 1. On Occluded-DukeMTMC, the Rank-1 and mAP accuracies of the proposed method are 69.1% and 56.8% respectively; on P-DukeMTMC-reID, the Rank-1 and mAP accuracies of the proposed method are 91.7% and 79.6% respectively; on Market1501, the Rank-1 and mAP accuracies of the proposed method are 95.0% and 87.3% respectively; on MSMT17, the Rank-1 and mAP accuracies of the proposed method are 80.6% and 58.8% respectively.

[0087] 2. The above results show that the proposed method achieves excellent performance compared with the SOTA on public datasets. This is because the proposed method can effectively solve the occlusion problem of pedestrians in monitoring scenarios.

[0088] In a second aspect of the embodiments of the present invention, a monitoring system for obstructed pedestrians is provided, which is used to execute the monitoring method for obstructed pedestrians according to the first aspect of the embodiments of the present invention.

[0089] A monitoring system for obstructed pedestrians provided by the present invention can implement the steps and effects of the monitoring method for obstructed pedestrians of the first aspect of the above-mentioned embodiment. To avoid repetition, the present invention will not go into details.

[0090] The above describes the specific embodiments of the present invention. It should be understood that the present invention is not limited to the above specific embodiments, and those skilled in the art can make various modifications or variations within the scope of the claims, which does not affect the essence of the present invention. The above preferred features can be used in any combination without conflicting with each other.

Claims

1. A monitoring method for obstructed pedestrians, characterized in that: The method comprises the following steps: Collection end: collects pedestrian images to be identified; Feature extraction end: The collected pedestrian images are subjected to feature extraction through the extractor ResNet50 to obtain the pedestrian image feature map; Feature erasing end: adopt a cross-erasing strategy to erase the feature map, construct an adversarial loss function, and iteratively optimize the parameters of the feature map cross-erasing module; The cross-erasing strategy is to randomly crop patches of area a=h×w, i.e., p1 and p2, from two pedestrian image feature maps f1 and f2 with an area of ​​A=H×W, where H and W represent height and width respectively; then, p1 is enlarged to half the area of ​​f2 and then used as a mask to cover the bottom of f2, thereby obtaining f′1; similarly, p2 is enlarged to half the area of ​​f1 and then used as a mask to cover the bottom of f1, thereby obtaining f′2; Feature attention end: The erased feature map is input into the occlusion-aware hybrid attention module to make the network focus on the non-occluded area of ​​the pedestrian; The occlusion-aware hybrid attention module is composed of spatial attention and channel attention. The specific steps are as follows: the input feature F is divided into three branches for processing; the first branch directly processes the feature map obtained by the feature map cross-erasing module; specifically, F is obtained after the channel attention CA c , CA is the result of optimizing ECA. Compared with ECA, it adds an element-level addition operation; the formula is as follows: F c =CA(F) Next, 3×3 convolution and 5×5 convolution are applied to the input feature F to obtain F 3×3 and F 5×5 , then F 3×3 and F 5×5 Through spatial attention SA and The formula is as follows: Next, with F 3×3 Take SA as an example; first, F 3×3 Segmentation along the height dimension yields and The formula is as follows: Where Φ(·) represents the segmentation function; then, the segmented feature map is globally averaged and globally max-pooled along the channel dimension, and then concatenated along the height dimension to obtain the feature map The formula is as follows: Where avgp(·) represents global average pooling, maxp(·) represents global maximum pooling, and C represents the concatenation operation; then, after each convolution block, we get After BN, ReLU and Sigmoid, we get The formula is as follows: Among them, C 1×1 Represents 1×1 convolution, LC 3×3 represents a lightweight 3×3 convolution; and F 3×3 Perform element-wise multiplication to get Then with F 3×3 Perform element-wise addition to get The formula is as follows: in, represents element-wise addition, represents element-wise multiplication; then, the resulting features and First perform element-wise addition to get F′, then add it to the first branch to get F c After splicing, we get F″, the formula is as follows: F″=C(F′,F c ) Finally, the noise-free feature F″ is added element by element to the input feature F to obtain F″′, as follows: Feature enhancement end: Use the full-view enhancement module to expand the network receptive field, obtain the final identification features of pedestrian images, build a baseline loss function, and iteratively optimize the network parameters; The full view enhancement module adopts a multi-branch dilated convolution structure, including 1×1 convolution, 3×3 convolution, dilated convolution with different dilation coefficients, splicing, ReLU and element-level addition operations; wherein the dilated convolution with different dilation coefficients is composed of four 3×3 dilated convolutions with different dilation coefficients, and the dilation coefficients are 1, 3, 5 and 7 respectively; Monitoring center: perform similarity measurement on the final identification features of the extracted pedestrian image and each pedestrian image in the image library, and output a matching list.

2. The monitoring method for obstructed pedestrians according to claim 1, characterized in that: The adversarial loss function is two adversarial cross entropy losses.

3. The monitoring method for obstructed pedestrians according to claim 1, characterized in that: The baseline loss functions described are cross entropy loss and hard sample triplet loss.

4. The monitoring method for obstructed pedestrians according to claim 1, characterized in that: The similarity measurement obtains the final matching result by calculating the cosine similarity between the features of the pedestrian images, and generates a matching list from large to small according to the cosine similarity.

5. A monitoring system for obstructed pedestrians, characterized in that: Used to execute the monitoring method for obstructed pedestrians as described in any one of claims 1 to 4.

Images