A pedestrian search method based on cascade regression and angle relationship supervision

Abstract

The present invention proposes a pedestrian search method based on cascade regression and angular relationship supervision, which belongs to the image retrieval direction in the field of computer vision and aims to solve the problem of pedestrian matching in panoramic pictures taken across cameras. The method adopts cascade regression as a resampling mechanism, and realizes a coarse-to-fine strategy by increasing the threshold of the proposal box stage by stage. A pedestrian search framework SwinCascador is formed to reduce the potential conflict between the detection task and the pedestrian re-identification task and achieve the goal of task balance. An end-to-end architecture multi-stage detection strategy is adopted, and each stage further refines the results of the previous stage, and improves the overall performance through gradual improvement. The angular relationship supervision (AOIM) loss between feature vectors is introduced to suppress the influence between similar identities and reduce the difference in feature vectors under the same identity. Through this innovation, the method effectively improves the discrimination of different pedestrian identities and shows good performance.

Description

Technical Field

[0001] The present invention belongs to the field of computer vision, solves the problem of pedestrian matching in panoramic pictures taken by cross-cameras, and particularly relates to a pedestrian search method based on cascade regression and angle relationship supervision. Technical Background

[0002] Public safety issues have always received widespread social attention. In order to ensure public safety, surveillance cameras are installed in every corner of living places. Since the coverage of a single camera is limited, it is necessary to deploy multiple cameras to form a camera network to achieve complete coverage of the target area. Based on the information captured by the surveillance camera, we can better find the missing person or suspect, and the massive surveillance video also brings challenges to data processing. The traditional method of manually completing video analysis consumes a lot of time and manpower, so it is necessary to use deep learning technology to solve this problem. There is still a big gap between the pedestrian re-identification problem and real-world applications. The re-identification task processes static images and is a cropped pedestrian image. However, in practical applications, the camera captures panoramic images, so it is necessary to add pedestrian detection technology to find the location of pedestrians from the panoramic image, and then crop the image block containing pedestrians to form pedestrian search. Pedestrian search technology is an important research direction after pedestrian re-identification. Pedestrian search is to crop and adjust the original image to a fixed ratio, which solves the problem of identifying people without any prior knowledge of position and ratio. It is a sub-direction of image retrieval, which is the task of searching and matching target pedestrians in the entire scene image captured by a large number of cameras. It is applied to intelligent video surveillance, intelligent security, intelligent transportation and other fields. It includes a process of searching (target detection) and matching (pedestrian re-identification). There are two main methods to achieve it, namely, a two-stage model that considers two tasks independently and an end-to-end model that combines multiple tasks. Because the task of detection is to learn generalized features in the crowd and to detect people as much as possible from the picture, and re-identification is mainly to facilitate better distinction of human features, there is a conflict for the network, and there will be Re-ID priority and detection priority. However, when relying on any one strategy, it is difficult to balance the importance of the two subtasks in different situations. The existing pedestrian re-identification algorithm needs to rely on the results of target detection. Accurately detecting pedestrians is an important prerequisite for the re-identification task. However, target detection and pedestrian re-identification are trained separately and sequentially, and both the detection and re-identification parts require image feature extraction, which is computationally expensive. In addition, the two stages require a complex cascade process, which may be affected by the error propagation between the two stages. Therefore, more and more methods are being developed to realize pedestrian search in an end-to-end manner, integrating object detection and pedestrian re-identification into a framework to achieve high efficiency. The end-to-end method aims to directly retrieve images of people with similar features to the query person from the input image without obvious stage segmentation. Detection and feature extraction are realized in an end-to-end manner. This method can learn the detection and feature matching of people at the same time. At present, how to design an optimal end-to-end architecture and find an efficient and fast metric learning method are urgent problems to be solved in pedestrian search.

[0003] Since deep learning, the current mainstream end-to-end method is mainly based on the parallel structure of Faster R-CNN as the framework, but because detection and Re-ID are processed simultaneously, accurate bounding boxes cannot be obtained before extracting Re-ID features. Sharing features between detection and re-identification will also accumulate the errors of each feature. Sharing features means that these features extracted by the network come from low-quality proposals. Although these inferior features have little effect on coarse-grained classification tasks, they have a greater impact on fine-grained re-identification tasks. The Transformer-based end-to-end method converts images or image regions into sequence data and uses the Transformer model for feature extraction and matching. Images are usually segmented into a series of overlapping or non-overlapping regions, and each region is regarded as a position in the sequence. The Transformer model models each position through a self-attention mechanism, which can well capture the dependencies between regions and global context information. The information of multiple positions can be considered simultaneously in the entire image, which helps to better understand the characteristics and contextual relationships of the person. However, the mainstream still uses CNN as the feature extraction network, inputs high-dimensional features into Transformer and uses its attention mechanism to capture fine-grained features. However, for pedestrian search tasks, which involve multi-target detection and the diversity and complexity of pedestrians, the model needs to be able to understand the appearance, posture, and background characteristics of different pedestrians. The Swin Transformer can better handle global relationships and capture contextual information between pedestrians through its self-attention mechanism and hierarchical structure, and has good modeling capabilities for long-distance dependencies between pedestrians. In addition, the scale of pedestrians may vary greatly due to their distance from the camera. The hierarchical attention mechanism of Swin Transformer enables it to effectively handle targets of different scales and is suitable for detecting pedestrians with varying scales in images.

[0004] We propose a new end-to-end cascade framework (SwinCascador) based on SwinTransformer to solve the end-to-end pedestrian search task. The framework adopts a coarse-to-fine strategy, first focusing on optimizing the detection results, and then focusing on improving the Re-ID matching effect to balance the requirements between the detection task and the Re-ID task, effectively mitigating the potential conflict between the two. By capturing the angle information between feature vectors and better measuring the similarity between them, an inverse cosine online instance matching loss function is proposed. Summary of the invention

[0005] This paper proposes a pedestrian search method based on cascade regression and angular relationship supervision. The method uses the deep attention transformation network Swin Transformer as the backbone and uses the weights pre-trained on the LUPerson dataset as the initialization of the entire network. The designed cascade end-to-end network (SwinCascadorNet) is guided in the training phase. We propose a loss function that suppresses the influence between similar identities and reduces the difference in feature vectors under the same identity. The main idea is to implement a coarse-to-fine detection strategy through three stages. The goal of this strategy is to reduce the conflict between pedestrian detection and pedestrian re-identification, and gradually improve pedestrian detection and identity feature tasks through different focus points in each stage. In the first stage, we focus on accurately detecting the presence of people from the background. Through this preliminary detection, we can determine the possible pedestrians in the image and generate corresponding candidate regions. The second stage further refines the candidate regions of the first stage, focusing on improving the accuracy and precision of detection. At the same time, the second and third stages introduce the task of identity embedding to gain a deeper understanding of the differences between pedestrians and improve the performance of Re-ID. This phased strategy helps resolve the conflict between pedestrian detection and Re-ID. Using a coarse-to-fine strategy, the first phase improves the comprehensiveness of pedestrian detection, the second phase emphasizes detection accuracy and uses Re-ID for preliminary matching, and the third phase focuses on Re-ID feature matching to improve accurate identity recognition. This strategy helps balance the needs between detection tasks and Re-ID tasks, improves overall performance through gradual improvement, reduces potential conflicts between the two tasks, and provides a more effective solution for pedestrian search tasks.

[0006] The technical solution is:

[0007] A pedestrian search method based on cascade regression and angle relationship supervision, the method comprising:

[0008] In step 1), we first select images and use the defined general data transformer to ensure that the input images undergo a series of standardized resizing before being fed into the model. For each training batch, we randomly select P images, each of which is accompanied by a corresponding pedestrian detection label and pedestrian re-identification label.

[0009] Step 2) Design a cascade network feature extraction module, which consists of a miniature version of the Swin Transformer (Swin-T) as the backbone network and a region generation network (RPN). RPN generates region proposals, determines whether the anchors are positive or negative through softmax, and then uses bounding box regression to correct the anchors to obtain accurate proposals. The RoI-Align operation is used to pool a 14×14 region as a proposal feature map.

[0010] Step 3) Design the head module for pedestrian search, including detection head, re-identification head and deep re-identification head, and use a multi-level cascade structure to learn features for pedestrian detection and Re-ID. The features are passed through the box head to obtain the final proposal feature maps, and then the accurate regression detection results are passed to the second stage through classification and regression, and so on. This staged strategy helps to resolve the conflict between pedestrian detection and Re-ID.

[0011] Step 4), in the loss function design of the deep re-identification head, in order to solve the problem caused by the limitation of the shared feature space between the background and the instance, the norm and angle of the feature vector are decomposed. Among them, the norm is used for the confidence of pedestrian detection, and the angle is used for the cosine similarity of the pedestrian re-identification task. An angle term m is introduced to increase the contribution of the angle by increasing θ by m angles, thereby reducing the difference of feature vectors within the same identity.

[0012] Step 5), the training phase, the proposed network is trained end-to-end for pedestrian detection and pedestrian re-identification. The pedestrian detection loss contains regression and classification loss terms in three stages. Different thresholds are used for adjustment in each stage. In the initial stage, the proposals proposed by RPN may have low quality, but with each stage of training, its quality gradually improves to ensure that there is no serious mismatch problem with subsequent detectors using higher IoU thresholds.

[0013] Step 6), in the test phase, the RPN network proposes about 300 proposals, which are directly fed into RoI Pooling for category classification and box regression, and pedestrian detection and re-identification of new images are performed using the trained model. Cumulative matching features (CMC) and mean average precision (mAP) are used to evaluate algorithm performance. CMC is used to search for queries from gallery images and calculate top-K accuracy. mAP calculates the average precision of each query through the precision-recall curve, and then averages the average precision scores of all queries.

[0014] The step 1) randomly selects P images in each training batch so that each batch contains diverse images, which can make the model better adapt to complex situations in real scenes, thereby improving the performance and application scope of the pedestrian search system.

[0015] The backbone of step 2) uses the pre-trained parameters provided by SOLIDER obtained by pre-training on the LUPerson dataset, and uses a miniature version of Swin Transformer (Swin-Tiny) as the backbone network. The input size is adjusted to 1504×928.

[0016] Step 3) uses cascade regression as a resampling mechanism to increase the IoU value of proposals stage by stage, so that the proposals resampled in the previous stage can adapt to the higher IoU threshold in the next stage. Because an inappropriate threshold may filter out a large number of positive samples, resulting in mismatch problems and a reduction in the Top-1 index.

[0017] In step 4), the classic online instance matching (OIM) loss is used in supervised person re-identification, which can effectively calculate the cosine similarity between small batch samples and LUT / CQ samples. In addition, an angle threshold m is added based on the category, which is multiplied by the cosine value of the corresponding label to strengthen the similarity between feature vectors of the same identity and reduce the similarity with other identities.

[0018] In step 5), the model is trained using the stochastic gradient descent (SGD) method. For the PRW dataset, the model was trained for 18 epochs with a batch size of 3; for the CUHK-SYSU dataset, the model was trained for 22 epochs with a batch size of 2. The weight decay and momentum are set to 5×10^(-4) and 0.9, respectively. The learning rate is set to 0.003. For the CUHK-SYSU / PRW datasets, the loop queue size of OIM is set to 5000 / 500. During testing, the thresholds of 0.5 / 0.6 / 0.7 are used to remove the redundant boxes detected by the first / second / third head.

[0019] The step 6) is different from the training stage in that these proposals cannot be sampled in the reasoning stage because the ground truth information cannot be obtained in the reasoning stage, and therefore the IoU cannot be calculated.

[0020] In the whole method, the network is implemented end-to-end during training and testing. After large-scale data training, a higher retrieval rate can be achieved.

[0021] By using cascade regression as a resampling mechanism, the present invention achieves the ability to focus on different tasks at each stage, thereby alleviating the potential conflict between the two tasks. An angular relationship supervision (AOIM) loss is introduced to suppress the influence between similar identities and reduce the difference in feature vectors under the same identity. It has significant advantages over other methods on the standard pedestrian search benchmark. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Attached Figure 1 It is a schematic diagram of the end-to-end cascade framework pedestrian search network model based on Swin Transformer constructed by the present invention.

[0023] Attached Figure 2 Schematic diagram of the deep re-identification head constructed by the present invention.

[0024] Attached Figure 3 It is a schematic diagram of the loss function designed by the present invention. DETAILED DESCRIPTION

[0025] A pedestrian search method based on cascade regression and angle relationship supervision includes the following steps:

[0026] 1) In the specific example of the pedestrian search method based on cascade regression and angle relationship supervision, cascade regression is used as a resampling mechanism to improve the IoU value of proposals stage by stage, so that the proposals resampled in the previous stage can adapt to the higher IoU threshold of the next stage.

[0027] 2) In the specific example of the pedestrian search method based on cascade regression and angular relationship supervision, deep re-identification focuses on Re-ID feature matching to further improve accurate identity recognition. Each stage uses a different head, so that different stages of cascadercnn can adapt to different distributions. This staged strategy helps to resolve the conflict between pedestrian detection and Re-ID. Using cascade regression as a resampling mechanism, the IoU value of the proposal is improved stage by stage, so that the proposals resampled in the previous stage can adapt to the next stage with a higher threshold. The coarse-to-fine strategy allows each stage to focus on different tasks, thereby balancing the needs between detection tasks and Re-ID tasks.

[0028] In the specific example of the pedestrian search method based on cascade regression and angle relationship supervision, a method is designed to better distinguish pedestrians with similar appearance but different identities. This method decomposes the norm and angle of the feature vector, where the norm is used for the confidence of pedestrian detection and the angle is used for the cosine similarity of the pedestrian re-identification task. Global average pooling and fully connected layers are applied to the original features to obtain a 256-dimensional feature vector x. Then, x is decomposed into the norm r and the angle θ through explicit decomposition in polar coordinates.

[0029] x=r·θ

[0030] In order to interpret the norm r as the detection confidence, its magnitude is compressed to the range of [0, 1] by a monotonic mapping. Where σ is the sigmoid activation function for batch normalization.

[0031]

[0032] However, since there are too many categories of identities in the pedestrian search task, the effective distinction of appearance similarity is extremely high. In order to enhance the differences between different identities, we added a supervision term on the basis of NAE. By introducing an additional angle term m, the difference in feature vectors under the same identity is reduced, emphasizing the differences between different identities. At the same time, OIM (Object Instance Mining) loss is used for joint optimization of labeled and unlabeled identities. Let the input cosine be the cosine value after projection, label i be the label of the sample, cosine be the cosine weight in the model parameter, and m be a matrix with a shape of (batch_size, 6000). Among them, the special values ​​without labels in the detection are filtered out (ignore_index=5554).

[0033]

[0034] In the design of the loss function, we introduced an angle term m, which increases the contribution of the angle by increasing θ by m angles, thereby reducing the difference in feature vectors within the same identity. The calculation of the loss function includes the projection of the cosine value, the cosine weight of the label, the calculation of the arccosine of the cosine value, the increase of the angle, and the final cross entropy loss. The purpose of this design is to reduce the intra-class distance and increase the inter-class distance to better distinguish pedestrians with similar appearance but different identities. Among them, i represents the index of the sample and j represents the index of the category. Next, `θ` is calculated, where θ is the arccosine (arccos) of the cosine value:

[0035] θ ij =arccos(cosine ij )

[0036] Then, `m` is added to θ to get θ+m, where the clamp function ensures that it is in the range (1e-3, π-1e-3):

[0037] θ m =clamp(θ+m,min=1e-3,max=Π-1e-3)

[0038] Among them, the clamp function ensures that θm is in the interval (1e-3, Π―1e-3). Finally, the cosine function is applied, multiplied by oim_scalar, and the cross entropy loss is calculated. By adjusting the angle between the centerline vector C and the edge feature vector x, the similarity between the feature vectors is strengthened, and the difference between similar identities is minimized by the cross entropy loss. The stability of the whole method is guaranteed, especially by adjusting the angle and cosine value, the problem of gradient explosion is prevented.

[0039] For the OIM loss function, where x is considered as class-id, the probability of i is calculated using the softmax function:

[0040]

[0041] Similarly, the probability of being recognized as the i-th unlabeled identity is:

[0042]

[0043] The OIM objective is to maximize the expected log likelihood:

[0044]

[0045] The derivative for x is:

[0046]

[0047] Combining the OIM loss with the feature angle supervision forms the AOIM loss, λOIM and λAC are the balance parameters:

[0048] L AOIM =λ scalar L OIM +μ scalar L AC

[0049] Then the detection head and the re-identification head are jointly optimized. T represents the level, which defaults to 3. The total loss of our method is: The total loss of our method is:

[0050]

[0051] In this paper, we propose an end-to-end cascade framework based on Swin Transformer, called SwinCascador. By using cascade regression as a resampling mechanism, we achieve the ability to focus on different tasks at each stage, thereby mitigating the potential conflict between the two tasks. We also introduce angular relationship supervision (AOIM) loss to suppress the influence between similar identities and reduce the difference in feature vectors under the same identity. SwinCascador has significant advantages over other methods on standard person search benchmarks.

[0052] The technical contents not described in the above methods can be realized by adopting or drawing on existing technologies. It should be noted that, under the guidance of this specification, those skilled in the art can also make such or such easy variations, such as equivalent methods or obvious deformation methods. The above variations should all be within the protection scope of the present invention.

Claims

1. A pedestrian search method based on cascade regression and angle relationship supervision, the method comprising: Step 1), first select the image, and use the defined general data transformer to ensure that the input image undergoes a series of standardized resizing before being sent to the model. For each training batch, P images are randomly selected, and each image is accompanied by the corresponding pedestrian detection label and pedestrian re-identification label; Step 2), design a cascade network feature extraction module, which consists of a micro-network based on Swin Transformer as the backbone network and RPN. RPN generates candidate regions, determines whether the anchor box belongs to a positive sample or a negative sample through softmax, and then uses bounding box regression to correct the anchor box to obtain an accurate candidate box. The RoI-Align operation is used to pool a 14×14 region as a proposed feature map; Step 3), design the head module of pedestrian search, including detection head, re-identification head and deep re-identification head, use the cascade network feature extraction module to learn the features for pedestrian detection and Re-ID, the first stage of the cascade network feature extraction module passes the features through the box head to obtain the candidate area, and then passes the accurate regression detection results to the second stage of the cascade network feature extraction module through classification and regression, and further refines the generated candidate area. Then the second stage uses the refined regression detection results as input in the same way as the first stage, and passes them to the third stage of the cascade network feature extraction module. This staged strategy helps to resolve the conflict between pedestrian detection and Re-ID; Step 4), in the loss function design of the deep re-identification head, in order to solve the problem caused by the limitation of the shared feature space between the background and the instance, the norm and angle of the feature vector are decomposed, where the norm is used for the confidence of pedestrian detection and the angle is used for the cosine similarity of the pedestrian re-identification task. An angle term m is introduced to increase the contribution of the angle by increasing θ by m angles, thereby reducing the difference of feature vectors within the same identity; Step 5), in the training phase, the proposed network is trained end-to-end for pedestrian detection and pedestrian re-identification. The pedestrian detection loss contains three stages of regression and classification loss terms, and each stage is adjusted with different thresholds. In the initial stage, the candidate boxes proposed by RPN have low quality, but with each stage of training, their quality gradually improves to ensure that there is no serious mismatch problem with the subsequent detectors using higher IoU thresholds; Step 6), in the testing phase, the RPN network proposes 300 candidate boxes, which directly enter RoIPooling for category classification and box regression. The trained model is used to detect and re-identify pedestrians in new images. CMC and mAP are used to evaluate the algorithm performance. CMC is used to search queries from gallery images and calculate the top-K accuracy, while mAP calculates the average precision of each query through the precision-recall curve, and then averages the average precision scores of all queries.

2. The pedestrian search method based on cascade regression and angle relationship supervision according to claim 1 is characterized in that: Step 1) For each training batch, P images are randomly selected from the image set after data transformation. Each randomly selected image needs to be accompanied by a corresponding pedestrian detection label and pedestrian re-identification label. The pedestrian detection label is used to indicate the position of the pedestrian in the image, while the pedestrian re-identification label is used to identify the differences between different pedestrians, ensuring that the label corresponds to the image one by one to establish correct supervised learning.

3. The pedestrian search method based on cascade regression and angle relationship supervision according to claim 1 is characterized in that: In step 2), a micro-network based on Swin Transformer is used as the backbone network, and each position is modeled through the self-attention mechanism to capture the dependencies between regions and the global context information.

4. The pedestrian search method based on cascade regression and angle relationship supervision according to claim 1, characterized in that: Step 3) Cascade regression is used as a resampling mechanism to increase the IoU threshold of the candidate box stage by stage, so that the candidate box resampled in the previous stage can adapt to the higher IoU threshold of the next stage, because an inappropriate threshold will filter out a large number of positive samples, resulting in mismatch problems and a reduction in the Top-1 index.

5. The pedestrian search method based on cascade regression and angle relationship supervision according to claim 1, characterized in that: Step 4) In supervised person re-identification, the classic online instance matching loss is used as the basis, which can effectively calculate the cosine similarity between small batch samples and LUT / CQ samples. In addition, an angle threshold m is added based on the category. By multiplying it with the cosine value of the corresponding label, the similarity between feature vectors of the same identity is strengthened, while the similarity with other identities is reduced.

6. The pedestrian search method based on cascade regression and angle relationship supervision according to claim 1, characterized in that: In step 5), the pre-trained parameters obtained by LU Person pre-training are used, and the input size is adjusted to 1504×928. Due to the limited video memory, the half-precision method is used to reduce the usage of video memory, and the stochastic gradient descent method is used for model training.

7. The pedestrian search method based on cascade regression and angle relationship supervision according to claim 1, characterized in that: The entire network uses an end-to-end architecture to implement a multi-stage detection strategy during training and testing. Each stage can resample to adapt to the next stage with a higher threshold. This strategy ensures that the detector at each stage will not overfit, and that the samples are of sufficient quality to meet the threshold conditions. It also gradually improves the quality of the candidate boxes during inference, avoiding serious mismatch problems.

Images