A pedestrian search method based on two-branch interaction

Abstract

The application provides a pedestrian search method based on two-branch interaction, and the pedestrian search network used includes a backbone network, a connection network, a detection prediction network and a re-identification prediction network, and contains the following steps: preparing a pedestrian image dataset; training the pedestrian search network, which is divided into the following sub-steps: using the parameters of an ImageNet pre-training model to initialize the weights of the backbone network; inputting the training set image into the pedestrian search network, and outputting a feature map through the backbone network; inputting the feature map into the connection network to output a candidate box, performing RoI feature extraction on the candidate box, and outputting an RoI feature; inputting the RoI feature into the prediction network, generating a pedestrian detection feature and a detection result through the detection prediction network; inputting the RoI feature and the pedestrian detection feature into the re-identification prediction network; inputting the static re-identification feature and the dynamic re-identification feature into a cross attention module to calculate the attention weight between the two re-identification features, and outputting the final pedestrian re-identification feature and a re-identification result.

Description

Technical Field

[0001] This invention relates to the field of pedestrian search technology in computer vision, and is mainly used for pedestrian search methods in fields such as intelligent security. Background Technology

[0002] In today's society, people are increasingly concerned about safety, and relevant agencies are particularly focused on the security of public areas such as airports and shopping malls. Providing security for the public requires significant human and financial resources. In manual video surveillance systems, network cameras continuously record video every day, generating a large amount of video that needs analysis. Monitoring operators need to analyze any specific event or anomaly simultaneously, which is a challenging and tedious task. Intelligent video surveillance systems aim to automatically monitor and analyze video from camera networks to help monitoring operators process and understand the video acquired by the camera network, making it one of the most popular and challenging research areas in computer vision.

[0003] To achieve this goal, pedestrian search technology plays a crucial role. Today, applying pedestrian search technology to intelligent security systems is becoming an effective tool for solving various security problems, such as forensic investigations, crime prevention, and protecting restricted areas. Pedestrian search is a challenging computer vision task aimed at detecting and identifying a queried pedestrian from a set of images in a library, or retrieving images of pedestrians of interest from video frames acquired from different camera views. Figure 1 The image shown is an example of the results of pedestrian search on an actual captured scene image.

[0004] Pedestrian search is generally considered a joint task of pedestrian detection and pedestrian re-identification [1, 2, 3, 4]. Current pedestrian search methods can be mainly divided into two-step methods [5, 6, 7] and single-step methods [8, 9, 10]. Two-step methods use two independent networks for pedestrian detection and re-identification, respectively. Compared to two-step methods, single-step methods use a single network to perform detection and re-identification simultaneously. For example... Figure 2 The diagram shows the network framework of the single-step pedestrian search method, which mainly consists of a backbone network, a connection network, and a prediction network. The prediction network includes a pedestrian detection prediction network and a pedestrian re-identification prediction network, which output the detection results and re-identification results, respectively. Due to its simple and efficient design, the single-step method has become the mainstream approach for pedestrian search.

[0005] In single-walker person search methods, candidate box-based methods have achieved good performance. However, most candidate box-based single-walker person search methods typically pass the extracted RoI (Region of Interest) features through several fixed fully connected layers to extract person re-identification features. This simple re-identification prediction network structure ignores the need to utilize rich instance information to improve the accuracy of person search.

[0006] References

[0007] [1] Gaikwad B, Karmakar A. Real-time distributed video analytics for privacy-aware person search[J]. Computer Vision and Image Understanding, 2023: 103749.

[0008] [2] Liu Z, Mu X, Lu Y, et al. Learning transformer-based attention region with multiple scales for occluded person re-identification[J]. Computer Vision and Image Understanding, 2023, 229: 103652.

[0009] [3] Li X, Li Q, Liang F, et al. Multi-granularity Pseudo-label Collaboration for unsupervised person re-identification[J]. Computer Vision and Image Understanding, 2023, 227: 103616.

[0010] [4] Cao J, Pang Y, et al. From handcrafted to deep features for pedestrian detection: A survey[J]. IEEE transactions on pattern analysis and machine intelligence, 2021, 44(9): 4913-4934.

[0011] [5] Chen D, Zhang S, Ouyang W, et al. Person search via a mask-guided two-stream cnn model[C]. Proceedings of the european conference on computer vision (ECCV). 2018: 734-750.

[0012] [6]Lan X,Zhu X,Gong S.Person search by multi-scale matching[C].

[0013] Proceedings of the European conference on computer vision(ECCV).2018:536-552.

[0014] [7]Han C,Ye J,Zhong Y,et al.Re-id driven localization refinement for person search[C].Proceedings of the IEEE / CVF International Conference on Computer Vision.2019:9814-9823.

[0015] [8]Xiao T,Li S,Wang B,et al.Joint detection and identification feature learning for person search.Proceedings of the IEEE conference on computer vision and pattern recognition.2017:3415-3424.

[0016] [9]Chen D,Zhang S,Yang J,et al.Norm-aware embedding for efficient person search[C].Proceedings of the IEEE / CVF conference on computer vision and pattern recognition.2020:12615-12624.

[0017]

[10] Yan Y,Li J,Qin J,et al.Anchor-free person search[C].Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition.2021:7690-7699. Summary of the Invention

[0018] To address the shortcomings of existing technologies, this invention proposes a pedestrian search method based on two-branch interaction. The traditional re-identification feature extraction method is used as one branch of the re-identification prediction network, and a designed dynamic re-identification module is connected in parallel with it as another branch. This fully utilizes instance information to extract more discriminative re-identification features, thereby improving the performance of pedestrian search.

[0019] This invention is achieved through the following technical solution, which is specifically as follows:

[0020] A pedestrian search method based on two-branch interaction is proposed. It employs a single-walk pedestrian search network framework and constructs a pedestrian search network based on a deep convolutional neural network, comprising a backbone network, a connection network, and a prediction network. The prediction network is further divided into a detection prediction network and a re-identification prediction network, and includes the following steps:

[0021] Step 1: Prepare a pedestrian image dataset;

[0022] Step 2: Divide the dataset into a training set and a test set;

[0023] Step 3: Set parameters such as the total number of training cycles and the training learning rate;

[0024] Step 4: Train the pedestrian search network, which consists of the following sub-steps:

[0025] Sub-step 1: Initialize the weights of the backbone network;

[0026] Sub-step 2: Input the training set images into the pedestrian search network, and output feature maps through the backbone network;

[0027] Sub-step 3: Input the feature map into the connection network to output candidate boxes, extract RoI features from the candidate boxes, and output the RoI features;

[0028] Sub-step 4: Input the RoI features into the prediction network, and generate pedestrian detection features and detection results through the detection prediction network; input the RoI features and pedestrian detection features into the re-identification prediction network;

[0029] The re-identification prediction network consists of two parallel branches: a static re-identification module and a dynamic re-identification module; among them,

[0030] The static re-identification module consists of two fully connected layers connected in series. RoI features are input into the static re-identification module to generate static re-identification features.

[0031] The dynamic re-identification module includes a cascaded dynamic convolutional layer and a fully connected layer. Pedestrian detection features are input into the fully connected layer to generate the parameter matrix required for dynamic convolution calculation. This parameter matrix is ​​then divided into two parts: one part serves as the first sub-parameter matrix, and the other part, after transpose, serves as the second sub-parameter matrix. Subsequently, dynamic convolution calculation is performed in the dynamic convolutional layer. The process involves sequentially convolving the RoI features with the two sub-parameter matrices to output intermediate features. These intermediate features are then input into the fully connected layer cascaded with the dynamic convolutional layer to output dynamic re-identification features.

[0032] The static and dynamic re-identification features are input into the cross-attention module to calculate the attention weight between the two re-identification features, and the final pedestrian re-identification features and re-identification results are output.

[0033] Sub-step 5: Set the loss function for model training, which includes the loss function for pedestrian detection and the loss function for pedestrian re-identification; update the weight parameters of the network through the backpropagation algorithm in the deep convolutional neural network;

[0034] Sub-step 6: Complete model training.

[0035] Furthermore, the backbone network uses ResNet50.

[0036] Furthermore, the weights of the backbone network are initialized using the parameters of the ImageNet pre-trained model.

[0037] Furthermore, Faster R-CNN was used as the detection and prediction network.

[0038] Furthermore, the pedestrian re-identification loss function adopts the OIM loss.

[0039] The beneficial effects of this invention are as follows: Based on traditional candidate box-based pedestrian search methods, this invention proposes a dynamic re-identification module. Through its dynamic convolutional layers, it fully extracts rich instance information, adding a dynamic mechanism for processing pedestrian images, while retaining the original static re-identification module, thus improving the diversity of extracted re-identification features. Furthermore, by utilizing a cross-attention module to complement the two re-identification features generated by the two branches, more discriminative pedestrian re-identification features can be generated, thereby effectively improving the performance of the pedestrian search method. Attached Figure Description

[0040] Figure 1 This is an example of the results of pedestrian search on actual captured scene images.

[0041] Figure 2 A network framework for a single-walker human search method.

[0042] Figure 3This is a network framework diagram of the present invention, namely a pedestrian search method based on two-branch interaction.

[0043] Figure 4 This is a flowchart illustrating a specific implementation method of the method proposed in this invention. Detailed Implementation

[0044] This invention addresses the problem in traditional candidate box-based pedestrian search methods that only use a few fully connected layers to extract pedestrian re-identification features without fully utilizing instance information to enhance these features. It proposes a two-branch interaction-based pedestrian search method that effectively extracts and utilizes rich instance information, and complements the re-identification features generated by the two branches to generate interactive pedestrian re-identification features, thereby improving the performance of the pedestrian search method. Figure 4 The specific implementation of the method proposed in this invention is given, and the main steps are as follows:

[0045] Step 1: Prepare an image dataset containing pedestrians of different identities, such as the PRW dataset, and obtain the annotations of pedestrians in each image of the dataset, including pedestrian bounding box annotations and pedestrian identity annotations;

[0046] Step 2: Divide the dataset into a training set and a test set;

[0047] Step 3: The total number of training cycles is set to 36, the learning rate is set to 0.00003, the number of candidate boxes N during training is set to 100, and the feature dimension is set to 256.

[0048] Step 4: Train the pedestrian search network, which consists of the following sub-steps:

[0049] Sub-step 1: Initialize the weights of the deep convolutional neural network using the parameters of the ImageNet pre-trained model;

[0050] Sub-step 2: Input the training set image I into the pedestrian search network, and output the feature map F through the backbone network. The backbone network adopts ResNet50.

[0051] Sub-step 3: Input the feature map F into the connected network to output candidate boxes P. The connected network uses RPN. Then, RoI features are extracted from the candidate boxes P through the RoI align operation to obtain the RoI features F. RoI ∈R 7×7×256 ;

[0052] Sub-step 4: Extract RoI features F RoI The input is fed into the prediction network, which first generates pedestrian detection features F through the detection prediction network. det The detection results were analyzed, and Faster R-CNN was used as the detection and prediction network. Then, the RoI features F... RoIPedestrian detection features F det ∈R 256 Input re-identification prediction network.

[0053] The proposed re-identification prediction network consists of two parallel branches: a static re-identification module and a dynamic re-identification module. The static re-identification module comprises two fully connected layers connected in series, which integrate RoI features F... RoI The input to the static re-identification module is flattened to one dimension, then reduced to 1024 dimensionality through the first fully connected layer, and finally passed through the second fully connected layer to generate the static re-identification feature F. rp ∈R 256 This branch can capture direct, shallow features of pedestrians. The dynamic re-identification module consists of a dynamic convolutional layer and a fully connected layer connected in series. First, it processes the pedestrian detection features F... det Input a fully connected layer and generate the parameter matrix D∈R required for dynamic convolution computation. 256×64×2 , where 64 is the parameter dimension. The parameter matrix D is then divided into two parts, with the first half serving as the first sub-parameter matrix D1∈R. 256×64 The transposed second half serves as the second subparameter matrix D2∈R 64×256 Then, dynamic convolution calculation is performed in the dynamic convolution layer, that is, the RoI feature F... RoI The two sub-parameter matrices mentioned above are convolved sequentially to output the intermediate feature F. out =F RoI *D1*D2,F out With F RoI Having the same dimension R 7×7×256 The intermediate features are flattened into one dimension and then fed into a fully connected layer connected in series with dynamic convolutional layers, outputting dynamic re-identification features F. rd ∈R 256 Because dynamic convolution calculation is added to the dynamic re-identification module, the re-identification prediction network can extract rich instance information. It adds a dynamic extraction mechanism for changes in pedestrian images and pedestrian context information, effectively making up for the shortcomings of the re-identification prediction network in traditional candidate box-based pedestrian search methods.

[0054] After outputting the re-identification features of each of the two branches, the static re-identification features F with the same dimensions are... rp and dynamic re-identification features F rd The input is fed into the Cross Attention module to calculate the attention weights between the two re-identification features, which are the final person re-identification features. The output is the re-identification features and the re-identification result. The calculation method for the re-identification features is as follows: In the formula, Q = F rp W Q K = F rd W KV = F rd W V W Q W K and W V The matrix consists of three learnable parameter matrices, d = 256, representing the feature dimension. The attention mechanism allows the model to focus more on the more important parts of the vast amount of information contained in the features. Employing a cross-attention module to fuse the re-identification features from the two branches enables the complementary alignment of static and dynamic re-identification features, resulting in generated pedestrian re-identification features with richer pedestrian identity information, thus enhancing discriminability.

[0055] Sub-step 5: Define the loss function for model training, which includes a loss function for pedestrian detection and a loss function for pedestrian re-identification, wherein the pedestrian re-identification loss function adopts OIM loss. Update the network weight parameters using the backpropagation algorithm in the deep convolutional neural network;

[0056] Sub-step 6: When all training cycles are completed, the learned weight parameters are the final network parameters, and the model training is complete;

[0057] Step 5: Input the test set images into the trained model for testing;

[0058] Step 6: Apply the model to real-world application scenarios such as intelligent video surveillance for pedestrian search, and use the results of the pedestrian search to assist in prediction.

[0059] This invention uses the re-identification prediction network of traditional candidate box-based pedestrian search methods as a static re-identification module, preserving direct and shallow pedestrian features. Building upon this, a proposed dynamic re-identification module is connected in parallel as a second branch. Through its dynamic convolutional layers, rich instance information is fully extracted, adding a dynamic mechanism for processing pedestrian images and improving the diversity of extracted re-identification features. Furthermore, a cross-attention module is used to complementarily fuse the two re-identification features generated by the two branches, generating more discriminative pedestrian re-identification features through feature alignment, thereby effectively improving the performance of the pedestrian search method.

Claims

1. A pedestrian search method based on two-branch interaction, employing a single-walk pedestrian search network framework, constructing a pedestrian search network based on a deep convolutional neural network, comprising three parts: a backbone network, a connection network, and a prediction network. The prediction network is divided into a detection prediction network and a re-identification prediction network, and includes the following steps: Step 1: Prepare a pedestrian image dataset; Step 2: Divide the dataset into a training set and a test set; Step 3: Set the total number of training cycles and the training learning rate parameter; Step 4: Train the pedestrian search network, which consists of the following sub-steps: Sub-step 1: Initialize the weights of the backbone network; Sub-step 2: Input the training set images into the pedestrian search network, and output feature maps through the backbone network; Sub-step 3: Input the feature map into the connection network to output candidate boxes, extract RoI features from the candidate boxes, and output the RoI features; Sub-step 4: Input the RoI features into the prediction network, and generate pedestrian detection features and detection results through the detection prediction network; input the RoI features and pedestrian detection features into the re-identification prediction network; The re-identification prediction network consists of two parallel branches: a static re-identification module and a dynamic re-identification module. The static re-identification module is composed of two fully connected layers connected in series. RoI features are input into the static re-identification module to generate static re-identification features. The dynamic re-identification module includes a cascaded dynamic convolutional layer and a fully connected layer. Pedestrian detection features are input into the fully connected layer to generate the parameter matrix required for dynamic convolution calculation. This parameter matrix is ​​then divided into two parts: one part serves as the first sub-parameter matrix, and the other part, after transpose, serves as the second sub-parameter matrix. Subsequently, dynamic convolution calculation is performed in the dynamic convolutional layer. The process involves sequentially convolving the RoI features with the two sub-parameter matrices to output intermediate features. These intermediate features are then input into the fully connected layer cascaded with the dynamic convolutional layer to output dynamic re-identification features. The static and dynamic re-identification features are input into the cross-attention module to calculate the attention weight between the two re-identification features, and the final pedestrian re-identification features and re-identification results are output. Sub-step 5: Set the loss function for model training, which includes the loss function for pedestrian detection and the loss function for pedestrian re-identification; update the weight parameters of the network through the backpropagation algorithm in the deep convolutional neural network; Sub-step 6: Complete model training.

2. The pedestrian search method according to claim 1, characterized in that, Prepare an image dataset containing pedestrians of different identities, and obtain the annotations of pedestrians in each image of the dataset, including pedestrian bounding box annotations and pedestrian identity annotations.

3. The pedestrian search method according to claim 1, characterized in that, The backbone network uses ResNet50.

4. The pedestrian search method according to claim 1, characterized in that, The weights of the backbone network are initialized using the parameters of the ImageNet pre-trained model.

5. The pedestrian search method according to claim 1, characterized in that, Faster R-CNN was used as the detection and prediction network.

6. The pedestrian search method according to claim 1, characterized in that, The pedestrian re-identification loss function adopts OIM loss.

Images