Pedestrian search method and device, equipment and storage medium
By constructing a pedestrian search model based on deformed convolutional layer and pyramid visual transformer network, combining multi-scale deformable attention and shared decoder, the efficiency and accuracy of pedestrian search in complex scenarios are solved, and efficient pedestrian detection and Re-identify.
Patent Information
- Application Number
- CN202411050069.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-01
- Publication Date
- 2024-11-22
AI Technical Summary
The existing pedestrian search technology is poor in complex scenarios and variable pedestrian appearance, lacks feature extraction ability, poor adaptability, and long training time, making it difficult to meet the real-time application needs.
Using a pedestrian search model based on deformable convolutional layer, pyramid visual transformer network and deformable mapper, combined with multi-scale deformable attention and shared decoder, it is used to re-identify through input layer, output layer, encoder, detector and discriminant re-identification The decoder builds the initial model and optimizes the model through the training set and test set to achieve efficient pedestrian detection and re-identification.
The efficiency and accuracy of pedestrian searches are achieved in complex scenarios, the model adaptability and practicality of complex scenarios is improved, the accuracy of detection and identification is optimized, and the efficiency and reliability of the system are enhanced.
Smart Images

Figure CN119007243A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of computer vision technology, and in particular to a pedestrian search method, device, equipment and storage medium. Background Art
[0002] The development of pedestrian search technology has been widely used in security and intelligent monitoring. With the increase of complex scenes and variable pedestrian appearance, traditional feature extraction techniques such as Haar features, HOG features and SIFT features are no longer sufficient to meet the needs of efficient recognition. Deep learning-based methods such as convolutional neural networks (CNN), recurrent neural networks (RNN) and Transformer have made significant progress in improving pedestrian detection and re-identification performance. However, these methods still face many challenges in dealing with complex scenes, lighting changes and scale changes.
[0003] Traditional methods mainly rely on manually designed features (such as Haar features, HOG features, and SIFT features), which perform well in simple scenes, but have poor results in complex scenes and with changing pedestrian appearances. Algorithm models include sliding window-based DPM detectors, YOLO, SSD, and Fast R-CNN. These methods are inefficient when processing large-scale data and have difficulty adapting to changes in different scenes. Deep learning-based methods use network structures such as CNN, RNN, and Transformer to significantly improve the performance of pedestrian detection and re-identification through automatic feature extraction and multi-level feature learning. However, these methods still face problems such as insufficient feature expression capabilities, poor adaptability, long training time, and inconsistency between detection and re-identification tasks.
[0004] Traditional methods rely on hand-crafted features, such as Haar features, HOG features, and SIFT features. Although they perform well in simple scenes, they do not work well in complex scenes and with changing pedestrian appearances. Although deep learning-based methods have improved feature expression capabilities, they still have several key problems: unstable performance and poor adaptability when dealing with pedestrian targets of different scales, especially poor performance under conditions of illumination changes, occlusion, and scale changes. In addition, existing methods usually require a large amount of training data and a long training process, which consumes a lot of computing resources and is difficult to meet the needs of real-time applications. Most importantly, the traditional two-step method processes pedestrian detection and re-identification separately, resulting in inconsistent feature extraction and affecting overall performance. Therefore, how to efficiently search for pedestrians in complex scenes has become an urgent problem to be solved.
[0005] The above contents are only used to assist in understanding the technical solution of the present application and do not constitute an admission that the above contents are prior art. Summary of the invention
[0006] The purpose of this application is to provide a pedestrian search method, device, equipment and storage medium, aiming to solve the technical problem of how to efficiently search for pedestrians in complex scenarios.
[0007] To achieve the above objectives, the present application proposes a pedestrian search method, the method comprising:
[0008] Construct an initial pedestrian search model based on the input layer, output layer, deformable convolution layer, pyramid vision transformer network, deformable mapper, first feedforward layer, and pedestrian search module;
[0009] Obtaining a pedestrian search data set, and dividing the pedestrian search data set into a training set and a test set;
[0010] Using the training set and the test set to train and test the initial pedestrian search model to obtain a target pedestrian search model;
[0011] The target pedestrian search model is used to perform pedestrian search on pedestrian images.
[0012] In one embodiment, the pedestrian search module includes an encoder, a detector, and a discriminative re-identification decoder;
[0013] The step of constructing an initial pedestrian search model according to the input layer, the output layer, the deformable convolution layer, the pyramid visual transformer network, the deformable mapper, the first feedforward layer and the pedestrian search module comprises:
[0014] Construct a backbone network based on the input layer, deformable convolutional layer, pyramid vision transformer network, and deformable mapper;
[0015] constructing a neck network based on the first feed-forward layer and the encoder;
[0016] constructing a head network based on the detector, the discriminative re-ID decoder, and the output layer;
[0017] An initial pedestrian search model is constructed according to the backbone network, the neck network and the head network.
[0018] In one embodiment, the step of performing pedestrian search on a pedestrian image using the target pedestrian search model includes:
[0019] Inputting a pedestrian picture into the target pedestrian search model, and extracting a multi-scale feature map through the deformable convolution layer and the pyramid visual transformer network;
[0020] Mapping the multi-scale feature map to different feature layers through the deformable mapper to obtain a first layer feature map, a second layer feature map, and a third layer feature map;
[0021] The first layer feature map, the second layer feature map, and the third layer feature map are processed by the neck network and the head network to obtain pedestrian search results.
[0022] In one embodiment, the encoder includes a first multi-scale deformable attention module and a second feed-forward layer, and the discriminative re-ID decoder includes a third multi-scale deformable attention module;
[0023] The step of obtaining pedestrian search results by processing the first layer feature map, the second layer feature map, and the third layer feature map through the neck network and the head network includes:
[0024] Processing the first-layer feature map, the second-layer feature map, and the third-layer feature map through the first feed-forward layer to obtain a feed-forward feature map;
[0025] The feedforward feature map is processed by the first multi-scale deformable attention module and the second feedforward layer, and normalized to obtain an encoded feature map;
[0026] Processing the encoding feature map by the detector to obtain a preliminary decoding feature map;
[0027] Processing and normalizing the preliminary decoding feature map through the third multi-scale deformable attention module to obtain a target decoding feature map;
[0028] The target decoding feature map is processed through the output layer to obtain category prediction, bounding box coordinates and re-identification features, and the category prediction, bounding box coordinates and re-identification features are used as pedestrian search results.
[0029] In one embodiment, the detector includes a second multi-scale deformable attention module, a multi-head attention module, and a third feed-forward layer;
[0030] The step of processing the encoding feature map by the detector to obtain a preliminary decoding feature map comprises:
[0031] Processing and normalizing the encoded feature map by the second multi-scale deformable attention module to obtain a multi-scale attention feature map;
[0032] Processing and normalizing the multi-scale attention feature map through the multi-head attention module to obtain a multi-head attention feature map;
[0033] The multi-head attention feature map is processed by the third feed-forward layer to obtain a preliminary decoding feature map.
[0034] In one embodiment, the step of using the training set and the test set to train and test the initial pedestrian search model to obtain a target pedestrian search model includes:
[0035] Using the training set to train the initial pedestrian search model to obtain a reference pedestrian search model;
[0036] Using the test set to test the reference pedestrian search model, to obtain a test result;
[0037] When the test result does not meet the preset requirements, return to the step of using the training set to train the initial pedestrian search model to obtain a reference pedestrian search model, until the test result meets the preset requirements to obtain a target pedestrian search model.
[0038] In one embodiment, the step of using the training set to train the initial pedestrian search model to obtain a reference pedestrian search model includes:
[0039] Inputting the training set into the initial pedestrian search model to obtain pedestrian detection results, wherein the pedestrian detection results include classification scores, bounding boxes, and re-identification features;
[0040] Calculate a classification loss, a bounding box loss, and an online instance matching loss according to the classification score, the bounding box, and the re-identification feature;
[0041] Obtaining a total loss based on the classification loss, the bounding box loss, and the online instance matching loss;
[0042] The initial pedestrian search model is trained using a back propagation strategy until the total loss is less than or equal to a preset loss value, thereby obtaining a reference pedestrian search model.
[0043] In addition, to achieve the above purpose, the present application also proposes a pedestrian search device, the device comprising:
[0044] A construction module, for constructing an initial pedestrian search model according to an input layer, an output layer, a deformable convolution layer, a pyramid vision transformer network, a deformable mapper, a first feed-forward layer, and a pedestrian search module;
[0045] A data module, used for obtaining a pedestrian search data set and dividing the pedestrian search data set into a training set and a test set;
[0046] A training module, used to train and test the initial pedestrian search model using the training set and the test set to obtain a target pedestrian search model;
[0047] The search module is used to perform pedestrian search on pedestrian images using the target pedestrian search model.
[0048] In addition, to achieve the above-mentioned purpose, the present application also proposes a pedestrian search device, which includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the pedestrian search method described above.
[0049] In addition, to achieve the above-mentioned purpose, the present application also proposes a storage medium, which is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, the steps of the pedestrian search method described above are implemented.
[0050] One or more technical solutions proposed in this application have at least the following technical effects:
[0051] The present application constructs an initial pedestrian search model according to an input layer, an output layer, a deformable convolution layer, a pyramid visual transformer network, a deformable mapper, a first feedforward layer, and a pedestrian search module; obtains a pedestrian search data set, and divides the pedestrian search data set into a training set and a test set; uses the training set and the test set to train and test the initial pedestrian search model to obtain a target pedestrian search model; uses the target pedestrian search model to perform pedestrian search on pedestrian images. The present application first performs a pedestrian search task by constructing an initial pedestrian search model, including an input layer, which receives original image data or feature representation; an output layer, which generates a bounding box and classification result for pedestrian detection, as well as a feature vector for pedestrian re-identification; a deformable convolution layer and a pyramid visual transformer network (PVT), which are used to process multi-scale information and extract global and local features; a deformable mapper, which enhances the model's ability to handle irregular shapes; a first feedforward layer, which performs feature conversion and nonlinear mapping; a pedestrian search module, which combines multi-scale deformable attention and a shared decoder. Secondly, a pedestrian search dataset is obtained and divided into a training set and a test set. The dataset includes image data annotated with pedestrian bounding boxes and identity information. Reasonable division of the dataset is conducive to the independence and generalization of model training and evaluation. Then, the initial pedestrian search model is trained using the training set, and the model parameters are optimized through back propagation to adapt it to the pedestrian search task. In the test phase, the performance of the model is evaluated using the test set, including indicators such as pedestrian detection accuracy and re-identification accuracy, and the model performance is gradually optimized. Finally, the trained target pedestrian search model is used to search for new pedestrian images. The model will output the classification results, bounding box positions, and re-identification feature vectors of each detected pedestrian, realizing efficient pedestrian search in complex scenes, providing effective support for fields such as surveillance security, improving the model's adaptability to complex scenes, optimizing the accuracy of detection and recognition, and enhancing the efficiency and reliability of the system in practical applications. BRIEF DESCRIPTION OF THE DRAWINGS
[0052] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.
[0053] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.
[0054] Figure 1 A flowchart of the first embodiment of the pedestrian search method of the present application is provided;
[0055] Figure 2 The MDAT model architecture diagram provided for the first embodiment of the pedestrian search method of the present application;
[0056] Figure 3 A flowchart of the second embodiment of the pedestrian search method of the present application is provided;
[0057] Figure 4 A comparison diagram of the discriminative re-identification decoder design solutions provided in the second embodiment of the pedestrian search method of the present application;
[0058] Figure 5 Schematic diagram of a multi-scale deformable attention module and a deformable attention layer provided for the second embodiment of the pedestrian search method of the present application;
[0059] Figure 6 This is a schematic diagram of the module structure of a pedestrian search device according to an embodiment of the present application;
[0060] Figure 7 This is a schematic diagram of the device structure of the hardware operating environment involved in the pedestrian search method in the embodiment of the present application.
[0061] The purpose, features and advantages of this application will be further described in conjunction with the embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION
[0062] It should be understood that the specific embodiments described herein are only used to explain the technical solutions of the present application and are not used to limit the present application.
[0063] In order to better understand the technical solution of the present application, a detailed description will be given below in conjunction with the accompanying drawings and specific implementation methods.
[0064] The development of pedestrian search technology is widely used in security and intelligent monitoring. Traditional methods such as Haar features, HOG features and SIFT features have limited effects in complex scenes and changing pedestrian appearances. Although deep learning-based methods such as CNN, RNN and Transformer have made progress in pedestrian detection and re-identification performance, they still face challenges such as handling different scales, lighting changes, occlusion and long training time. Traditional methods rely on manually designed features, while modern methods improve performance through automatic feature learning, but the consistency and adaptability of the two still need to be further improved to meet the needs of real-time applications.
[0065] The main solution of the embodiment of the present application is: first, build an initial pedestrian search model, including an input layer to receive image data, an output layer to generate detection results and feature vectors, use deformable convolution, PVT and deformable mappers to process multi-scale and irregular shapes, and finally combine multi-scale attention for pedestrian search. Secondly, obtain a pedestrian search data set, divide it into training sets and test sets to ensure model independence and generalization ability. Then, use the training set to train the model and evaluate the performance through the test set, and gradually optimize the model. Finally, use the trained target model to search for new pedestrian images, output classification, bounding box and re-identification features, and improve the adaptability and practicality of the model in complex scenarios.
[0066] It should be noted that the execution subject of the embodiments of the present application may be a computing service device with data processing, network communication and program running functions, such as a tablet computer, a personal computer, a mobile phone, etc., or an electronic device capable of realizing the above functions, a camera, etc. The following takes a camera as an example to illustrate this embodiment and the following embodiments.
[0067] Based on this, the present application embodiment provides a pedestrian search method, referring to Figure 1 , Figure 1 This is a flow chart of the first embodiment of the pedestrian search method of the present application.
[0068] In this embodiment, the pedestrian search method includes steps S10 to S40:
[0069] Step S10, constructing an initial pedestrian search model according to the input layer, the output layer, the deformable convolution layer, the pyramid visual transformer network, the deformable mapper, the first feedforward layer and the pedestrian search module;
[0070] It should be noted that the input layer refers to the initial input of the original image data or feature representation received by the model. The output layer refers to the final output of the model, including the bounding box position and classification results of pedestrian detection, as well as the feature vector of pedestrian re-identification. Deformable Convolution is a special layer in a convolutional neural network that allows the convolution kernel to shift position within the local receptive field. Traditional convolution operations are performed on a fixed grid, while deformable convolution enables the convolution kernel to adapt to irregular shapes and structures in the input feature map by learning deformable offsets. The Pyramid Vision Transformer (PVT) is a visual transformer (Transformer) architecture that combines the ideas of visual pyramids and transformers. It introduces multi-scale feature expressions and processes inputs of different resolutions in layers to better capture global and local information in the image. PVT performs well in computer vision tasks, especially when dealing with multi-scale inputs and large-scale image datasets. Deformable Mapper is a module for computer vision tasks that allows local feature maps to interact with global features. This method enhances the model's ability to handle irregular shapes and spatial changes by allowing local regions of the mapper to deform spatially. In a neural network, a feed-forward layer refers to the process by which data is passed from the input layer to the output layer after a series of calculations of weights and activation functions. The first feed-forward layer is the first feed-forward layer in an encoder or decoder, located between the mapper and the pedestrian search module, for further feature transformation and nonlinear mapping. The Person Search Module (PSM) is a module used for pedestrian detection and recognition tasks in computer vision. It usually combines target detection and tracking techniques to accurately detect and locate pedestrians in complex scenes. It also includes identity recognition or pedestrian re-identification functions. In this embodiment, PSM is a module specially designed for pedestrian search tasks in the Multi-Scale Deformable Attention Transformer (MDAT) model. It combines a multi-scale deformable attention mechanism and multi-level supervision of a shared decoder for simultaneous pedestrian detection and pedestrian re-identification.The initial pedestrian search model refers to a preliminary model architecture for pedestrian search tasks built based on the above components. In this embodiment, it is MDAT. This model enhances the feature representation capability by introducing a multi-scale deformable attention mechanism and improves the model's adaptability to pedestrians of different scales and shapes. This model extracts features from the image data received by the input layer through a pyramid visual transformer network, and then further processes the features using a deformable mapper and a feedforward layer, and finally generates the prediction results of the output layer through the pedestrian search module.
[0071] It can be understood that, first, the model receives the original image data as the input layer. Through the pyramid visual transformer network (PVT), the feature expression of the input image is extracted at multiple scales to capture global and local information. Next, the deformable mapper module is applied to allow the local feature map to be deformed in space, enhancing the model's ability to handle irregular shapes and spatial changes. Between the feature mapper and the pedestrian search module (PSM), the first feed-forward layer (Feed-Forward Layer) performs further feature transformation and nonlinear mapping to optimize the feature expression. Finally, the pedestrian search module (PSM) uses the multi-scale deformable attention transformer (MDAT) model, combined with multi-level supervision and shared decoder technology, to accurately detect and identify pedestrians, and generate output layer prediction results including pedestrian detection bounding boxes, classification results, and pedestrian re-identification feature vectors. Overall, the initial pedestrian search model achieves effective positioning, recognition, and re-identification functions of pedestrians in complex scenes through the above steps.
[0072] Please refer to Figure 2 , Figure 2The MDAT model architecture diagram provided for the pedestrian search method embodiment 1 of the present application shows the architecture flow of the MDAT model, starting from the input layer, which receives the original pedestrian image data. Subsequently, these data are fed into the backbone network for extracting feature representations of the image at different scales. These multi-scale feature maps are then fed into the deformable mapper, which enhances the model's adaptability to pedestrian targets of different shapes and sizes. After that, the feature map enters the neck network, where the first feedforward layer further processes the feature map to extract deeper features. The encoder then uses these features for encoding to generate encoded feature maps, which are processed by the first multi-scale deformable attention module and the second feedforward layer and normalized. In the head network, the encoded feature map is first fed into the detector, which includes a second multi-scale deformable attention module, a multi-head attention module, and a third feedforward layer, which is used to generate a preliminary decoded feature map, including the bounding box and classification information of the pedestrian. The preliminary decoded feature map is then fed into the discriminative re-ID decoder, where the third multi-scale deformable attention module further processes and normalizes the feature map to extract the re-ID features of the pedestrian. Finally, the output layer combines the results of the detector and the re-ID decoder to output the final result of the pedestrian search, which includes category predictions, bounding box coordinates, and pedestrian feature vectors for re-ID. In this way, the MDAT model can perform pedestrian detection and re-ID at the same time, providing an efficient and accurate pedestrian search solution.
[0073] Step S20, obtaining a pedestrian search data set, and dividing the pedestrian search data set into a training set and a test set;
[0074] It should be noted that pedestrian search datasets refer to datasets specifically used for pedestrian detection, recognition, and re-identification tasks. These datasets usually contain annotated pedestrian images or video sequences, and each sample may include information such as the pedestrian's bounding box position, identity, and pedestrian attributes. The purpose of these datasets is to train and evaluate the performance of pedestrian search models. The training set refers to the subset data used to train the pedestrian search model. The model learns feature representation and pattern recognition through these data, and optimizes parameters to improve performance on pedestrian search tasks. The test set is a subset of data used to evaluate the generalization ability of the pedestrian search model after training. The performance of the model on the test set directly reflects its predictive ability and performance on unseen data.
[0075] It is understandable that first, it is necessary to select a public or private dataset suitable for pedestrian detection, recognition, and re-identification tasks, such as Market-1501, CUHK03, CityPersons, etc. After obtaining the dataset, data preprocessing is required, including reading images or video sequences and obtaining annotation information for each pedestrian, such as bounding box location, identity identification, etc. Next, the dataset is usually divided into a training set and a test set by random partitioning or cross-validation. The training set (usually 70% to 80% of the total dataset) is used for parameter training and optimization of the model, and the test set (usually 20% to 30% of the total dataset) is used to evaluate the performance and generalization ability of the model on unseen data to ensure the effectiveness and stability of the model in real scenes. In this embodiment, the CUHK-SYSU and PRW datasets are used. CUHK-SYSU is a large-scale person search dataset. There are a total of 18,184 images, covering a variety of real-world challenges, including perspective changes, lighting changes, and different backgrounds. It has 96,143 annotated pedestrians and 8,432 different identities. The training set includes 11206 images, 55272 pedestrians, and 5532 identities. The test set contains 6978 images, 40871 pedestrians, and 2900 identities. During reasoning, the dataset defines gallery sets of different sizes, ranging from 50 to 4000. This example uses the standard setting of gallery size 100 for experiments. In addition, the performance when changing the gallery size is analyzed. PRW is a challenging person search dataset collected by 6 static cameras. The training set contains 5704 images, 18048 pedestrians, and 482 identities. The test set has 6112 images, 25062 pedestrians, and 450 identities. The following is a comparison of the CUHK-SYSU and PRW datasets:
[0076]
[0077] Step S30, using the training set and the test set to train and test the initial pedestrian search model to obtain a target pedestrian search model;
[0078] It should be noted that training refers to using the training set to adjust the parameters of the model so that it can better fit the data, which usually includes the process of backpropagation and optimizer updating parameters. Testing refers to using the test set to evaluate the performance of the trained model on unseen data to evaluate the generalization ability and performance of the model. The target pedestrian search model refers to the model that performs best in a given task (such as pedestrian detection and re-identification) after training and testing, that is, the model that achieves the expected performance indicators on the test set.
[0079] It can be understood that, first, the obtained pedestrian search dataset is divided into a training set and a test set according to a predetermined ratio, ensuring that the training set is used for model parameter optimization and learning, while the test set is used to evaluate the generalization ability and performance of the model. Then, the initial pedestrian search model is loaded and initialized using the experimental environment supported by the MindSpore framework. In the training phase, the model iteratively optimizes its own parameters through multiple epochs to minimize the loss function on the training set. This process involves the use of the AdamW (Adaptive Moment Estimation with Weight Decay) optimizer, which can effectively adjust the learning rate to improve the training effect. After the training is completed, the test set is used to evaluate the performance of the model on unknown data, and its performance is evaluated by measuring its accuracy in terms of pedestrian detection bounding box position, classification results, and pedestrian re-identification feature vectors. Finally, the model with the best performance on the test set is selected as the target pedestrian search model, which can provide accurate and reliable pedestrian detection and recognition functions in practical applications.
[0080] As an example, the initial pedestrian search model is trained and tested using the training set and the test set to obtain a target pedestrian search model, including the steps of: training the initial pedestrian search model using the training set to obtain a reference pedestrian search model; testing the reference pedestrian search model using the test set to obtain a test result; when the test result does not meet the preset requirements, returning to the step of training the initial pedestrian search model using the training set to obtain a reference pedestrian search model, until the test result meets the preset requirements to obtain the target pedestrian search model.
[0081] The reference pedestrian search model refers to the intermediate model version obtained by training with the training set, that is, the state of the model at a certain point in time during the training process. It is used as an intermediate stage model to evaluate the performance on the training data and guide subsequent optimization. The test result refers to the performance of the reference pedestrian search model evaluated using the test set, which usually includes indicators such as pedestrian detection accuracy, precision, and recall rate, as well as the evaluation results of pedestrian re-identification, such as the similarity calculation of feature vectors. The preset requirement refers to a pre-defined model performance indicator or threshold used to determine whether the test result meets the requirements of practical applications. For example, the accuracy of pedestrian detection can be set to be above 90%, and the similarity of feature vectors of pedestrian re-identification can be set to be above 0.8. For example, if the preset requirement is that the accuracy of pedestrian detection is not less than 90%, when the reference pedestrian search model is evaluated with the test set for the first time, the accuracy is found to be 85%, then it is necessary to return to the step of training the initial pedestrian search model with the training set, and continue to optimize the model until the accuracy reaches or exceeds 90%, thereby obtaining a target pedestrian search model that meets the preset requirements.
[0082] First, the initial pedestrian search model is trained using the training set. During the training process, the model learns and adjusts parameters according to the image data and label information of the training set to optimize the model's pedestrian detection and re-identification capabilities. Next, the trained model is used to generate a reference pedestrian search model, which represents a certain state during the training process on the training set and can be used to evaluate the performance of the model on the training data. Subsequently, the reference pedestrian search model is tested using the test set. During the test process, the model performs pedestrian detection and re-identification on the images in the test set to obtain test results. These results include performance indicators of the model, such as the accuracy, precision and recall rate of pedestrian detection, and the feature vector similarity of pedestrian re-identification. If the test results do not meet the preset requirements, such as the accuracy of pedestrian detection is lower than the expected threshold, then it is necessary to return to the step of further training the initial model using the training set. This process may include adjusting the model's hyperparameters, increasing the amount of training data, or adjusting the training strategy to improve the model's performance. The training and testing iterations are repeated until the performance of the model on the test set meets or exceeds the preset requirements, and then the target pedestrian search model is obtained. This final model can be used for pedestrian detection and re-identification tasks in real applications.
[0083] As an example, the steps of using the training set to train the initial pedestrian search model to obtain a reference pedestrian search model include: inputting the training set into the initial pedestrian search model to obtain a pedestrian detection result, wherein the pedestrian detection result includes a classification score, a bounding box, and a re-identification feature; calculating the classification loss, the bounding box loss, and the online instance matching loss based on the classification score, the bounding box, and the re-identification feature; obtaining the total loss based on the classification loss, the bounding box loss, and the online instance matching loss; and training the initial pedestrian search model using a back propagation strategy until the total loss is less than or equal to a preset loss value to obtain a reference pedestrian search model.
[0084] The pedestrian detection result refers to the pedestrian information detected by the model in the image, which usually includes the pedestrian's position bounding box, the pedestrian's classification score (indicating the probability of whether the bounding box contains a pedestrian), and the pedestrian's re-identification feature vector. Classification score: refers to the pedestrian detection model's confidence score for whether each detected bounding box contains a pedestrian, which is usually a probability value. Bounding box refers to the location area of the pedestrian detected by the model in the image, usually represented by the coordinates of the upper left corner and the lower right corner. Re-identification feature refers to the feature vector used for pedestrian re-identification, which is usually a high-dimensional vector that represents the visual features of the pedestrian and is used to compare the similarity of the same pedestrian in different images. Classification loss refers to the loss function used to measure the difference between the pedestrian detection model's classification score and the true label, helping the model optimize the prediction of whether there is a pedestrian in the bounding box. Bounding box loss refers to the loss function used to measure the difference between the bounding box predicted by the model and the true bounding box, helping the model optimize the position and size of the bounding box. Online instance matching loss (OIM loss) refers to a loss function used for pedestrian re-identification tasks, which is used to measure the similarity of the re-identification feature vectors of the same pedestrian in different images. The total loss refers to the loss value that combines the classification loss, bounding box loss, and online instance matching loss, and is used to evaluate the performance of the model on the training set. The backpropagation strategy refers to the process of calculating the gradient of the loss function and updating the model parameters based on the gradient, which is used to optimize the model training process. The preset loss value refers to the target loss value threshold set during the training process. When the total loss of the model reaches or is lower than this value, it means that the performance of the model on the training set meets the expected requirements, and the reference pedestrian search model is obtained. For example, suppose that when training the initial pedestrian search model, the preset loss value is set to 0.5. This means that during the training process, the total loss of the model (composed of classification loss, bounding box loss, and online instance matching loss, etc.) needs to reach or be lower than 0.5 before the training is stopped. Such a setting helps to control the training time and ensure that the model achieves a sufficient performance level on the training set.
[0085] MDAT predicts the classification score, location, and re-ID features for each detection query. The detection features are classified and localized through two layers of multi-layer perceptrons (MLP), while the re-ID features are directly used for the re-ID task. During the training process, a lookup table V and a circular queue U are established to guide the re-ID feature learning. The re-ID features of all labeled identities are stored in V, while the re-ID features of the most recently unlabeled identities are stored in U. In each iteration, the re-ID features in the current mini-batch (e.g., F r1 ) with all the features in V and U. Then, the online instance matching (OIM) loss is calculated based on the similarity. The OIM loss maximizes the expected log-likelihood of each re-identified feature in the current mini-batch, that is:
[0086]
[0087] Among them, θ represents the current feature, φ(y) represents the real identity feature, and φ(k) represents the candidate feature.
[0088] During the back-propagation process, if the re-identification feature belongs to the real identity i, the i-th entry of V is updated by popping out the old identity and pushing the re-identification feature of the new unlabeled identity into U. The final total loss function is:
[0089] L=λ1L cls +λ2L iou +λ3L l1 +λ4L oim
[0090] Among them, L cls represents the classification loss, L iou represents the bounding box IoU loss, L l1 represents the bounding box l1 cost, L oim represents OIM loss. λ1,λ2,λ3,λ4 are hyperparameters for balancing different losses and are set to 1.0, 4.0, 3.0, and 0.7 respectively.
[0091] During inference, an annotated (bounding box) query person is searched for in a given query image from a set of gallery images. First, multiple predictions of the query image are generated using MDAT, where each prediction includes a classification score, a bounding box, and a re-ID feature. Then, the re-ID feature of the query person is set to the re-ID feature of the prediction with the largest overlap with the query person's bounding box; finally, predictions for all gallery images are generated, and the re-ID feature similarity between the query person and the predictions in the gallery images is calculated to identify the matching person in the gallery images.
[0092] First, the training set is input into the initial pedestrian search model, and the classification score, bounding box location, and re-identification features of each pedestrian are inferred by the model. Subsequently, the classification loss, bounding box loss, and online instance matching loss are calculated based on these outputs. These losses are used to measure the model's errors in classification accuracy, bounding box positioning, and re-identification accuracy. The total loss is obtained by weighted summation of these losses, and the model parameters are adjusted using the back-propagation algorithm to minimize the total loss. During the training process, the model is continuously optimized until the total loss is less than or equal to the preset training requirements. This process ensures the model's adaptability and performance improvement on the training set. Finally, the reference pedestrian search model is obtained, which has been trained to have the pedestrian detection ability and feature representation learned on the training set.
[0093] This example uses mean average precision (mAP) and top-1 accuracy to evaluate the performance of person search, using a backbone network pre-trained on ImageNet: ResNet-50. MDAT is trained on a single NVIDIA GeForce RTX 4090 GPU using the AdamW optimizer. During training, a multi-scale training scheme was used, and the focal online instance matching loss (OIM) was used for alignment. During inference, the test images were scaled to a fixed size of 1500×900 pixels. The model was trained for a total of 24 epochs, using a mini-batch size of 2. The initial learning rate was set to 0.0001, and the learning rate was reduced by a factor of 10 at the 19th and 23rd epochs. The MindSpore framework was used to support training. With this setup, the method can be efficiently trained and tested in complex real-world scenarios, and achieves excellent performance on the CUHK-SYSU dataset (Xiao T, Li S, Wang B, et al. Joint Detection and Identification Feature Learning for PersonSearch[J]. 2016. DOI: 10.48550 / arXiv.1604.01850.) and the person re-identification dataset (PRW, Zheng L, Zhang H, Sun S, et al. Person re-identification in the wild[C] / / Proceedings of the IEEE conference on computer vision and pattern recognition. 2017: 1367-1376).
[0094] Step S40: Use the target pedestrian search model to perform pedestrian search on the pedestrian picture.
[0095] It should be noted that pedestrian images refer to input images used for pedestrian search tasks, which usually contain pedestrian targets in various scenes. In pedestrian search tasks, the model needs to accurately detect the location of pedestrians (bounding boxes), classify the presence or absence of pedestrians (classification scores), and possible re-identification features from these images for subsequent identity verification or tracking.
[0096] It can be understood that, first, the pedestrian image to be searched is input into the trained target pedestrian search model; the model will process and analyze the input image to extract the feature information therein; through the pedestrian detection function of the model, the position and bounding box of the pedestrians that may exist in the image are determined; further, the model re-identifies based on the feature vector of the pedestrian to ensure that multiple detections of the same pedestrian can be correctly matched; finally, the model will output the classification results and bounding box information of each detected pedestrian to achieve accurate search and identification of pedestrian images.
[0097] The present embodiment provides a pedestrian search method, which constructs an initial pedestrian search model according to an input layer, an output layer, a deformable convolution layer, a pyramid visual transformer network, a deformable mapper, a first feedforward layer, and a pedestrian search module; obtains a pedestrian search data set, and divides the pedestrian search data set into a training set and a test set; uses the training set and the test set to train and test the initial pedestrian search model to obtain a target pedestrian search model; and uses the target pedestrian search model to perform pedestrian search on pedestrian images. The present embodiment first performs a pedestrian search task by constructing an initial pedestrian search model, including an input layer, which receives original image data or feature representation; an output layer, which generates a bounding box and classification result for pedestrian detection, as well as a feature vector for pedestrian re-identification; a deformable convolution layer and a pyramid visual transformer network (PVT), which are used to process multi-scale information and extract global and local features; a deformable mapper, which enhances the model's ability to process irregular shapes; a first feedforward layer, which performs feature conversion and nonlinear mapping; and a pedestrian search module, which combines multi-scale deformable attention and a shared decoder. Secondly, a pedestrian search dataset is obtained and divided into a training set and a test set. The dataset includes image data annotated with pedestrian bounding boxes and identity information. Reasonable division of the dataset is conducive to the independence and generalization of model training and evaluation. Then, the initial pedestrian search model is trained using the training set, and the model parameters are optimized through back propagation to adapt it to the pedestrian search task. In the test phase, the performance of the model is evaluated using the test set, including indicators such as pedestrian detection accuracy and re-identification accuracy, and the model performance is gradually optimized. Finally, the trained target pedestrian search model is used to search for new pedestrian images. The model will output the classification results, bounding box positions, and re-identification feature vectors of each detected pedestrian, realizing efficient pedestrian search in complex scenes, providing effective support for fields such as surveillance security, improving the model's adaptability to complex scenes, optimizing the accuracy of detection and recognition, and enhancing the efficiency and reliability of the system in practical applications.
[0098] Based on the first embodiment of the present application, in the second embodiment of the present application, the same or similar contents as those in the above-mentioned embodiment 1 can refer to the above introduction, and will not be repeated later. Figure 3 , Figure 3 This is a flow chart of the second embodiment of the pedestrian search method of the present application. The pedestrian search module includes an encoder, a detector, and a discriminative re-identification decoder. Step S10 of the pedestrian search method includes steps S11 to S14:
[0099] Step S11, constructing a backbone network according to the input layer, the deformable convolution layer, the pyramid visual transformer network and the deformable mapper;
[0100] It should be noted that the encoder is the part of the deep learning model used to extract features. Usually in the Transformer architecture, the encoder and detector are stacked by multiple identical layers (or "blocks"), each of which contains a self-attention mechanism and a feedforward network. In the PSM module, the encoder is based on Deformable DETR, which contains three layers of encoders and three layers of detectors. It uses the feature map P from the backbone network to extract features. i As input, each encoder contains a deformable self-attention layer and a feed-forward layer. The output features of each encoder layer are represented as F e1 , F e2 , F e3 , the first detector uses F e3 The features and N detection queries are taken as input, and each detector contains a standard self-attention layer, a deformable cross-attention layer, and a feed-forward layer. The output features of each layer of detector are represented as F d1 , F d2 , F d3 A feature length of 256 is used for all three encoder layers and detectors in this module. The detector features are used in the prediction head for bounding box classification and regression, and are further used to obtain re-ID feature queries for the discriminative re-ID decoder.
[0101] During the detection process, the detector features are used in the prediction head to perform bounding box classification and regression. Specifically, the regression of the bounding box can be expressed as the following formula:
[0102]
[0103] in, is the predicted bounding box, σ is the activation function, and W b and b b are the weight and bias of regression, F d is the output feature of the detector.
[0104] At the same time, the classification task can be expressed as:
[0105]
[0106] in, is the predicted classification probability distribution, softmax is the softmax function, W c and b care the weights and biases of the classification, respectively. In the MDAT model, the encoder is responsible for processing the input image and extracting its feature representation. The detector refers to the part of the model that is responsible for identifying the location and category of the target in the image. In the MDAT model, the detector is combined with the decoder to generate the bounding box and category information of the pedestrian. The Discriminative Re-ID Decoder is a component in the MDAT model used to extract pedestrian re-ID features. It uses a multi-scale deformable attention mechanism to enhance the discriminability of features, thereby improving the performance of pedestrian re-ID. MDAT introduces a discriminative re-ID decoder to generate discriminative re-ID features for each pedestrian. The discriminative re-ID decoder uses multi-level supervision designed with a shared decoder. To this end, the discriminative re-ID decoder uses features F from different detection decoders. d1 , F d2 , F d3 As a re-identification feature query, it can improve the diversity of re-identification feature queries and box positions (sampling positions) during training. In the inference process, the feature F is used d3 As the re-identification feature query, a discriminative re-identification feature is obtained. The discriminative re-identification decoder obtains the re-identification query from the detection decoder and directly performs i This architecture design is more accurate for the re-ID subtask than the standard encoder-decoder based design. Multi-level supervision of shared decoder design: A straightforward way to design a re-ID decoder is to use the last detection feature F d3 As the re-ID feature query, and using the re-ID decoder for feature prediction, this design achieves suboptimal performance, probably due to the lack of discriminative re-ID features learned from a single layer of supervision. This embodiment proposes two schemes that use multi-level (intermediate) supervision in the re-ID decoder to better learn re-ID features. This embodiment refers to the two proposed schemes as parallel re-ID decoder and shared re-ID decoder. The parallel re-ID decoder uses the features from each detection decoder as the re-ID feature query and generates the re-ID feature F using three parallel decoder layers. r1 , F r2 , F r3 Here, the re-identification feature F r1 , F r2 , which is only used to provide multi-level (intermediate) supervision during training. Unlike the parallel re-ID decoder scheme, the shared re-ID decoder scheme adopts a twin structure. Each of the three re-ID feature queries has a shared decoder to generate three re-ID features. Similar to the parallel re-ID decoder, the shared re-ID decoder scheme also only uses feature F during training. r1 , F r2. As mentioned earlier, the two subtasks of pedestrian detection and re-identification in pedestrian search have different goals (object-level recognition and instance-level matching). Based on this, MDAT directly uses backbone features as the input of the discriminative re-identification decoder instead of using the features of the detection encoder. It has been empirically verified that this can lead to superior performance compared to using the features of the detection encoder. The Backbone Network is a feature extraction network that constitutes the core of the deep learning model. It provides input features for other parts of the model (such as the detector and re-identification decoder). In the MDAT model, the Backbone Network consists of an input layer, a deformable convolutional layer, a pyramid visual transformer network, and a deformable mapper, which is responsible for extracting and processing multi-scale information and global-local features of the image.
[0107] It can be understood that, first, the input layer receives the original image data or feature representation; then, the deformable convolution layer is used to process multi-scale information and extract local features; then, the pyramid visual transformer network is responsible for processing the extraction and integration of global and local features; finally, the deformable mapper enhances the model's ability to handle irregular shapes. These components together constitute the backbone network, which is used to extract multi-scale information and global and local features of the image, providing rich feature representation and processing capabilities for subsequent pedestrian search tasks.
[0108] Please refer to Figure 4 , Figure 4A comparison diagram of the discriminative re-identification decoder design schemes provided for the second embodiment of the pedestrian search method of the present application, which shows in detail the design methods of three discriminative re-identification decoders and compares their structural differences and potential performance. In (a), the design of a "single-layer re-identification decoder" is presented, where there is only one re-identification decoder (decoder), which receives features (Fdl, Fd2, Fd3) from three detection decoders (labeled as decoder1, decoder2, decoder3), which represent detection information at different levels. The re-identification decoder outputs a single feature vector (Fr) for the pedestrian re-identification task. (b) describes the "parallel re-identification decoder" scheme, in which the output features of each detection decoder are independently fed into the corresponding three parallel re-identification decoders (labeled as decoder2, decoder3), and each decoder generates its own unique re-identification features (Fr2, Fr3). This design allows the model to learn parallel re-identification features from detection features at different levels, increasing the diversity of features and potentially improving recognition accuracy. Finally, (c) illustrates the design of a “shared re-ID decoder” where the output features of the three detection decoders (Fdl, Fd2, Fd3) are aggregated and fed into a shared re-ID decoder. This shared decoder is responsible for generating the three re-ID features (Fr3). The shared decoder design allows the model to utilize multi-level supervision signals during training, which helps improve the discriminative power of the re-ID features while maintaining the efficiency and accuracy of the model. Overall, the figure highlights the impact that different design schemes may have on the performance of pedestrian re-ID, and highlights the importance of key concepts such as multi-scale feature processing, multi-level supervision, and feature fusion in improving pedestrian search technology.
[0109] Step S12, constructing a neck network according to the first feed-forward layer and the encoder;
[0110] It should be noted that in the MDAT model, the neck network consists of the first feedforward layer and the encoder. This part of the network structure takes over the multi-scale features extracted by the backbone network and further processes and refines them. The first feedforward layer performs a preliminary transformation of the features through linear transformation and nonlinear activation, while the encoder enhances the expressiveness of the features through deep feature extraction. This design of the neck network is intended to provide a richer and more discriminative feature representation for the head network of the model, namely the detection and re-identification module, thereby improving the performance of the pedestrian search task. In short, the neck network is the key part of the MDAT model that connects the backbone network and the head network, and is responsible for the deep extraction and optimization of features to meet the needs of specific tasks.
[0111] It can be understood that in the MDAT model, the process of building the neck network involves the synergy of the first feedforward layer and the encoder. The first feedforward layer is responsible for receiving multi-scale features from the backbone network and performing initial linear transformation and nonlinear activation. These features come from the input layer, deformation convolution layer, pyramid visual transformer network, and deformable mapper. Its main purpose is to further transform the features into higher-level representations to enhance the model's abstract ability and understanding of the input image information. The encoder takes on a deeper responsibility, extracting and encoding the semantic information of the input features through stacked self-attention mechanisms and feedforward network layers. The encoder is designed to enable the neck network to generate richer and more discriminative feature representations, which are critical for the detection and re-identification modules in the pedestrian search task. Therefore, as a key part connecting the backbone network and the head network (detection and re-identification module), the neck network significantly improves the performance and efficiency of the MDAT model in the pedestrian search task through deep feature extraction and optimization.
[0112] Step S13, constructing a head network according to the detector, the discriminative re-identification decoder and the output layer;
[0113] It should be noted that the head network refers to the part of the MDAT model responsible for generating the final output, including the detector, the discriminative re-identification decoder, and the output layer. The detector is responsible for identifying the location and category information of pedestrians in the input image, generating the bounding box and classification results of pedestrians; the discriminative re-identification decoder focuses on extracting the re-identification feature vector of pedestrians, and enhances the discriminative power of features through a multi-scale deformable attention mechanism to support the re-identification task of pedestrians; the output layer integrates the information generated by the detector and the discriminative re-identification decoder, and finally outputs the classification results, bounding box locations, and re-identification feature vectors of each detected pedestrian. In summary, the head network in the MDAT model is responsible for converting the deeply extracted features into the final pedestrian detection and re-identification results, thereby completing the entire pedestrian search task.
[0114] It can be understood that the step of constructing the head network according to the detector, the discriminative re-identification decoder and the output layer includes integrating and configuring the key components in the MDAT model. The detector is responsible for identifying and locating pedestrians in the input image, generating a bounding box containing the pedestrian's position and category information; the discriminative re-identification decoder further extracts the pedestrian's re-identification features, which are enhanced by a multi-scale deformable attention mechanism to ensure the uniqueness and distinguishability of each pedestrian; the output layer combines the results of the detector and the re-identification decoder, and finally outputs the classification information, position and re-identification feature vector of each detected pedestrian, so that the entire head network can effectively support the pedestrian search task and improve the performance of the model in complex environments.
[0115] Step S14, constructing an initial pedestrian search model according to the backbone network, the neck network and the head network.
[0116] It can be understood that, first, the backbone network is responsible for extracting multi-scale visual features from the input image, and processes global and local features with the support of deformable convolutional layers and pyramid visual transformer networks. Secondly, the neck network receives the feature representation extracted by the backbone network, and further features are processed and refined through the first feedforward layer and encoder to enhance the expressiveness of the features. Finally, the head network consists of a detector, a discriminative re-identification decoder, and an output layer. The detector locates and classifies pedestrians in the input image, and the discriminative re-identification decoder extracts unique re-identification features for each pedestrian, and outputs the final pedestrian detection results through the output layer, including the pedestrian's location information, classification label, and re-identification feature vector. The initial pedestrian search model is constructed by integrating these network components, which is designed to improve the detection and re-identification accuracy of the model in complex scenes and provide effective support for pedestrian search tasks.
[0117] As an example, the step of using the target pedestrian search model to perform pedestrian search on pedestrian pictures includes: inputting the pedestrian picture into the target pedestrian search model, and extracting a multi-scale feature map through the deformable convolution layer and the pyramid visual transformer network; mapping the multi-scale feature map to different feature layers through the deformable mapper to obtain a first layer feature map, a second layer feature map, and a third layer feature map; processing the first layer feature map, the second layer feature map, and the third layer feature map through the neck network and the head network to obtain pedestrian search results.
[0118] Multi-scale feature maps refer to image feature maps extracted by deformable convolutional layers and pyramid visual transformer networks in pedestrian search tasks, capturing local and global information of images at different scales. Feature layers refer to different levels or dimensions to which the deformable mapper maps multi-scale feature maps. These feature layers are different expressions in feature space, aiming to more accurately describe different features and structures in the image. The first-layer feature map, the second-layer feature map, and the third-layer feature map are feature maps of different feature layers obtained after mapping by the deformable mapper. These hierarchical feature maps represent different levels of abstraction and expression of image features, usually containing a combination of local and global information. Pedestrian search results refer to the final output obtained after processing by the neck network and the head network. These results include the detected pedestrian location (bounding box), the prediction of pedestrian class, and the feature vector for pedestrian re-identification. Pedestrian search results are the final output information of the entire pedestrian search model, which is used to identify and track pedestrian targets in the image.
[0119] First, the pedestrian image to be searched is input into the target pedestrian search model, which extracts multi-scale feature maps from the input image through deformable convolutional layers and pyramid visual transformer networks. These feature maps can capture local and global information of the image. Next, the deformable mapper is used to map these multi-scale feature maps to different feature layers to obtain the first layer feature map, the second layer feature map, and the third layer feature map. These hierarchical feature maps differ in the dimension and abstraction level of feature representation to more accurately describe the characteristics and structure of pedestrians in the image. Finally, the first layer, second layer, and third layer feature maps are processed and integrated by the neck network and the head network to obtain the final pedestrian search results. These results include the detected pedestrian location (bounding box), the prediction of pedestrian class, and the feature vector for pedestrian re-identification, which effectively achieves the goal of accurately searching and identifying pedestrians in complex scenes.
[0120] As an example, the encoder includes a first multi-scale deformable attention module and a second feedforward layer, and the discriminative re-identification decoder includes a third multi-scale deformable attention module; the step of processing the first layer feature map, the second layer feature map and the third layer feature map through the neck network and the head network to obtain a pedestrian search result includes: processing the first layer feature map, the second layer feature map and the third layer feature map through the first feedforward layer to obtain a feedforward feature map; processing the feedforward feature map through the first multi-scale deformable attention module and the second feedforward layer, and normalizing it to obtain an encoded feature map; processing the encoded feature map through the detector to obtain a preliminary decoding feature map; processing and normalizing the preliminary decoding feature map through the third multi-scale deformable attention module to obtain a target decoding feature map; processing the target decoding feature map through the output layer to obtain category prediction, bounding box coordinates and re-identification features, and using the category prediction, bounding box coordinates and re-identification features as pedestrian search results.
[0121] The first multi-scale deformable attention module: This module is part of the encoder and applies a deformable attention mechanism at multiple scales. Deformable attention allows the model to dynamically adjust the shape and size of the attention region when processing features to better adapt to objects of different scales and shapes. The second feed-forward layer is a component of the encoder and is responsible for further transformation and feature extraction of the feature map processed by the first multi-scale deformable attention module. It usually includes linear transformations and nonlinear activation functions to enhance the expressiveness and discrimination of the features. The third multi-scale deformable attention module is part of the discriminative re-ID decoder and is specifically used to process feature maps during the decoding process. It is similar to the first multi-scale deformable attention module, but in the re-ID task, it pays special attention to improving the discrimination and accuracy of the re-ID features. The feed-forward feature map is the feature map processed by the first feed-forward layer. The first feed-forward layer usually includes convolution operations, activation functions, etc., which are used to extract preliminary feature representations of the input image. In deep learning, normalization is a common technique used to adjust the numerical range of features to a specific interval (such as 0 to 1 or -1 to 1) to accelerate the training process and improve the stability of the model, making it easier to train and optimize. Normalization usually includes batch normalization or feature normalization operations. The encoded feature map is a feature map obtained after processing by the first multi-scale deformable attention module and the second feedforward layer. Compared with the feedforward feature map, the encoded feature map has a higher level of semantics and abstract representation, and is suitable for subsequent recognition and classification tasks. The preliminary decoded feature map is a feature map obtained after the detector processes the encoded feature map. The preliminary decoded feature map is usually used for preliminary target recognition and location positioning, such as detecting the presence and approximate location of pedestrians. The target decoded feature map refers to the feature map obtained after further processing and normalization by the third multi-scale deformable attention module. The target decoded feature map has a more refined feature representation and contains detailed information for the final prediction. It is usually used for subsequent re-identification tasks. Category prediction refers to one of the outputs of the pedestrian search model, which is used to predict the category of the pedestrian, such as the type of pedestrian, identity information, or industry label. Bounding box coordinates refer to the position and size information of the rectangular bounding box of the pedestrian's position, which is used to accurately locate the specific position of the pedestrian in the image. Re-identification features refer to feature vectors or feature descriptions used to uniquely identify pedestrians, and are usually used to identify and track the same pedestrian in different scenarios.
[0122] First, the first, second, and third feature maps of the input are preliminarily processed through the first multi-scale deformable attention module and the second feedforward layer of the encoder to obtain the feedforward feature map. The first feedforward layer is responsible for linear transformation and nonlinear activation of these feature maps to generate the feedforward feature map. Next, the first multi-scale deformable attention module and the second feedforward layer in the encoder perform multi-scale feature extraction and processing on the feedforward feature map, and obtain the encoded feature map through normalization operation. Then, the detector processes the encoded feature map to identify the preliminary decoded feature map of the target, including the preliminary position and category information of the target. Next, the preliminary decoded feature map is further processed and normalized through the third multi-scale deformable attention module of the discriminative re-identification decoder to obtain the target decoded feature map. This step further enhances the discriminative power of the feature. Finally, the target decoded feature map is processed through the output layer to generate the final pedestrian search result, including category prediction, bounding box coordinates, and re-identification features, thereby achieving accurate search and recognition of pedestrian images. The entire process greatly improves the accuracy and robustness of pedestrian search through multi-level and multi-scale feature processing.
[0123] In MDAT, the multi-scale deformable attention module is integrated in the encoder, detector, and discriminative re-ID decoder. The input feature map is first extracted from the backbone network and processed at multiple scales to generate feature maps of different scales. At each scale, these feature maps are processed by multiple deformable attention layers, each of which includes calculating sampling offsets, calculating attention weights, calculating sampling positions, and performing attention calculations. Finally, the processing results of each scale are fused to generate the final output feature map. The multi-scale deformable attention module is described by the following formula:
[0124] (1) Calculate sampling offset:
[0125] Δ pi,j =W Δp F i,j +b Δp
[0126] Among them, Δ pi,j represents the sampling offset at position (i, j), W Δp and b Δp are weights and biases, respectively, and F i,j is the feature of the feature map at position (i, j).
[0127] (2) Calculate attention weight:
[0128] A i,j =softmax(W A F i,j +bA )
[0129] Among them, A i,j represents the attention weight at position (i, j), W A F i,j and b A are weight and bias respectively.
[0130] (3) Calculate the sampling position:
[0131] p i,j =p0+Δ pi,j
[0132] Among them, p i,j represents the sampling position at position (i, j), p0 is the initial position, Δ pi,j is the calculated sampling offset.
[0133] (4) Perform attention calculation:
[0134]
[0135] Among them, F i ' ,j Represents the output feature at position (i, j), A i,j,k is the attention weight, F(p i,j,k ) is at the sampling position p i,j,k features at different scales. In the encoder and detector, the attention mechanism enables the model to aggregate information on feature maps of different scales, thereby generating more refined detection queries. In the discriminative re-ID decoder, the attention mechanism is used to capture the changes in pedestrian features at different scales, thereby enhancing the discriminative power of re-ID features. The multi-scale deformable attention module improves the robustness of the model in dealing with pedestrians of different sizes and scales by flexibly selecting sampling points and fusing features at different scales. This mechanism effectively improves the performance of MDAT in dealing with complex scenes and pedestrians of different scales, enabling the model to achieve better results in pedestrian detection and re-ID tasks.
[0136] Please refer to Figure 5 , Figure 5A schematic diagram of a multi-scale deformable attention module and a deformable attention layer provided for the pedestrian search method embodiment 1 of the present application, which shows the internal structure and workflow of the multi-scale deformable attention module and the deformable attention layer, which are key components of the MDAT model and are used to enhance the model's ability to characterize pedestrian features. In (a), the "multi-scale deformable attention block" consists of multiple sub-layers, including a deformable attention layer and a multi-layer perceptron (MLP). The input features first pass through the deformable attention layer, which calculates the sampling offset and then applies it to the input features to generate new sampling positions. These sampling positions weight the input features through the attention mechanism to generate weighted features. Then, the weighted features are further extracted and transformed through the multi-layer perceptron, and finally the processing results of the attention block are output. (b) The internal operation of the "multi-scale deformable attention layer" is described in detail, including the steps of multi-scale processing of the input features. In this process, the feature map is decomposed into sub-feature maps of different scales, and each sub-feature map is processed by the deformable attention mechanism. This involves calculating the sampling offsets at each position and then adjusting the sampling points of the original feature map according to these offsets to capture detailed information at different scales. The calculation of attention weights determines the contribution of each sampling point, and these weights are multiplied and added with the sampled features to generate the final output feature map. Overall, the multi-scale deformable attention modules and layers capture the characteristics of pedestrians at different scales by flexibly adjusting the sampling points and attention weights, thereby improving the model's recognition and re-identification capabilities for pedestrian targets.
[0137] As an example, the detector includes a second multi-scale deformable attention module, a multi-head attention module and a third feed-forward layer; the step of processing the encoded feature map through the detector to obtain a preliminary decoding feature map includes: processing and normalizing the encoded feature map through the second multi-scale deformable attention module to obtain a multi-scale attention feature map; processing and normalizing the multi-scale attention feature map through the multi-head attention module to obtain a multi-head attention feature map; processing the multi-head attention feature map through the third feed-forward layer to obtain a preliminary decoding feature map.
[0138] The second multi-scale deformable attention module refers to a module in the detector that processes feature maps. It uses a deformable attention mechanism to adjust the weights of different positions in the feature map, thereby improving the model's adaptability to different scales and posture changes. This module performs normalization operations to ensure that the generated multi-scale attention feature maps remain consistent and effective in different situations. The multi-head attention module is another key component. It receives and processes feature maps from the second multi-scale deformable attention module. This module uses a multi-head mechanism, that is, processing multiple attention heads in parallel, to capture and integrate detailed information at different positions. The output generated by each head is normalized to form multi-head attention feature maps, which contain more detailed and comprehensive feature expressions. The third feed-forward layer is a part of the detector that receives multi-head attention feature maps as input and performs further feature extraction and processing. The main task of this layer is to convert the processed feature maps into preliminary decoded feature maps, which are ready for subsequent object detection and recognition tasks. The multi-scale attention feature map is a collection of feature maps generated by the second multi-scale deformable attention module. The feature map of each scale reflects the features and attention distribution at different scales. The multi-head attention feature map is a collection of feature maps generated by the multi-head attention module, and the feature map of each head captures detailed feature information at different positions and directions.
[0139] First, the second multi-scale deformable attention module receives the encoded feature map as input, uses the deformable attention mechanism to adjust the weights of different positions in the feature map, and performs normalization to generate a multi-scale attention feature map. This step ensures that the model can effectively capture feature information and spatial relationships at different scales. Next, the multi-head attention module receives the multi-scale attention feature map as input, and captures and integrates detailed information at each position in the feature map by processing multiple attention heads in parallel. The output generated by each attention head is normalized to form a multi-head attention feature map. This step improves the model's perception of diverse features through the multi-head attention mechanism. Finally, the third feed-forward layer receives the multi-head attention feature map as input, performs further feature extraction and processing, and obtains a preliminary decoded feature map. These feature maps are ready for subsequent target decoding and feature analysis tasks, providing a key feature representation and processing basis for the final pedestrian search results. Through the above steps, the multi-scale deformable attention and multi-head attention mechanisms are comprehensively utilized to effectively process and optimize the encoded feature map, thereby improving the performance and robustness of the pedestrian search model in complex scenarios.
[0140] This embodiment constructs a backbone network based on the input layer, the deformable convolution layer, the pyramid visual transformer network and the deformable mapper; constructs a neck network based on the first feedforward layer and the encoder; constructs a head network based on the detector, the discriminative re-identification decoder and the output layer; and constructs an initial pedestrian search model based on the backbone network, the neck network and the head network. This embodiment first uses the input layer to receive pedestrian images, and constructs a backbone network through a deformable convolution layer, a pyramid visual transformer network and a deformable mapper. These steps extract features through convolution operations, and use a multi-scale feature extraction mechanism to enhance the model's perception of information at different scales, dynamically adjust and map feature maps to improve the model's adaptability and accuracy. Then, the neck network is constructed in combination with the first feedforward layer and the encoder to further enhance the feature representation capability and provide the model with a deeper and more semantic feature representation. Subsequently, the detector, discriminative re-identification decoder and output layer are integrated to build the head network. The detector is used to locate and identify pedestrian targets. The discriminative re-identification decoder enhances the feature expression and differentiation capabilities through multi-scale attention and multi-head attention mechanisms. The output layer generates category predictions, bounding box coordinates and re-identification features, and comprehensively provides pedestrian search results. Finally, the backbone network, neck network and head network are organically combined to form an end-to-end pedestrian search model, ensuring that the model has efficient pedestrian detection and recognition capabilities in different tasks and scenarios. Integrating the feature processing capabilities of each step improves the overall performance and application value of the model.
[0141] It should be noted that the above examples are only used to understand the present application and do not constitute a limitation on the pedestrian search method of the present application. More simple transformations based on this technical concept are all within the scope of protection of the present application.
[0142] This application also provides a pedestrian search device, please refer to Figure 6 , the pedestrian search device comprises:
[0143] A construction module 10 is used to construct an initial pedestrian search model according to the input layer, the output layer, the deformable convolution layer, the pyramid visual transformer network, the deformable mapper, the first feedforward layer and the pedestrian search module;
[0144] A data module 20 is used to obtain a pedestrian search data set and divide the pedestrian search data set into a training set and a test set;
[0145] A training module 30, configured to train and test the initial pedestrian search model using the training set and the test set to obtain a target pedestrian search model;
[0146] The search module 40 is used to perform pedestrian search on the pedestrian picture using the target pedestrian search model.
[0147] The pedestrian search device provided by the present application adopts the pedestrian search method in the above embodiment, which can solve the technical problem of how to efficiently search for pedestrians in complex scenes. Compared with the prior art, the beneficial effects of the pedestrian search device provided by the present application are the same as the beneficial effects of the pedestrian search method provided by the above embodiment, and the other technical features of the pedestrian search device are the same as the features disclosed in the above embodiment method, which will not be repeated here.
[0148] The present application provides a pedestrian search device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the pedestrian search method in the above-mentioned embodiment one.
[0149] Reference below Figure 7 , which shows a schematic diagram of the structure of a pedestrian search device suitable for implementing the embodiment of the present application. The pedestrian search device in the embodiment of the present application may include but is not limited to mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Descriptions), PMPs (Portable Media Players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Figure 7 The pedestrian search device shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.
[0150] like Figure 7As shown, the pedestrian search device may include a processing device 1001 (e.g., a central processing unit, a graphics processor, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM: Read Only Memory) 1002 or a program loaded from a storage device 1003 to a random access memory (RAM: Random Access Memory) 1004. In RAM1004, various programs and data required for the operation of the pedestrian search device are also stored. The processing device 1001, ROM1002, and RAM1004 are connected to each other through a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Generally, the following systems can be connected to the I / O interface 1006: an input device 1007 including, for example, a touch screen, a touch pad, a keyboard, a mouse, an image sensor, a microphone, an accelerometer, a gyroscope, etc.; an output device 1008 including, for example, a liquid crystal display (LCD: Liquid Crystal Display), a speaker, a vibrator, etc.; a storage device 1003 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 1009. The communication device 1009 can allow the pedestrian search device to communicate with other devices wirelessly or by wire to exchange data. Although the figure shows a pedestrian search device with various systems, it should be understood that it is not required to implement or have all the systems shown. More or fewer systems can be implemented or have alternatively.
[0151] In particular, according to the embodiments disclosed in the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present application include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device, or installed from a storage device 1003, or installed from a ROM 1002. When the computer program is executed by the processing device 1001, the above-mentioned functions defined in the method of the embodiment disclosed in the present application are executed.
[0152] The pedestrian search device provided by the present application adopts the pedestrian search method in the above embodiment, which can solve the technical problem of how to efficiently search for pedestrians in complex scenes. Compared with the prior art, the beneficial effects of the pedestrian search device provided by the present application are the same as the beneficial effects of the pedestrian search method provided by the above embodiment, and the other technical features in the pedestrian search device are the same as the features disclosed in the method of the previous embodiment, which will not be repeated here.
[0153] It should be understood that the various parts disclosed in this application can be implemented by hardware, software, firmware or a combination thereof. In the description of the above embodiments, specific features, structures, materials or characteristics can be combined in any one or more embodiments or examples in a suitable manner.
[0154] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.
[0155] The present application provides a computer-readable storage medium having computer-readable program instructions (ie, computer programs) stored thereon, and the computer-readable program instructions are used to execute the pedestrian search method in the above-mentioned embodiment.
[0156] The computer-readable storage medium provided in the present application may be, for example, a USB flash drive, but is not limited to electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, systems or devices, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, system or device. The program code contained on the computer-readable storage medium may be transmitted using any appropriate medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination of the above.
[0157] The computer-readable storage medium may be included in the pedestrian search device; or may exist independently without being assembled into the pedestrian search device.
[0158] The above-mentioned computer-readable storage medium carries one or more programs. When the above-mentioned one or more programs are executed by the pedestrian search device, the pedestrian search device: constructs an initial pedestrian search model according to the input layer, the output layer, the deformable convolution layer, the pyramid visual transformer network, the deformable mapper, the first feedforward layer and the pedestrian search module; obtains a pedestrian search data set, and divides the pedestrian search data set into a training set and a test set; uses the training set and the test set to train and test the initial pedestrian search model to obtain a target pedestrian search model; and uses the target pedestrian search model to perform pedestrian search on pedestrian pictures.
[0159] Computer program code for performing the operations of the present application may be written in one or more programming languages or a combination thereof, including object-oriented programming languages such as Java, Smalltalk, C++, and conventional procedural programming languages such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).
[0160] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present application. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.
[0161] The modules involved in the embodiments described in this application may be implemented by software or hardware, wherein the name of the module does not constitute a limitation on the unit itself in some cases.
[0162] The readable storage medium provided in the present application is a computer-readable storage medium, which stores computer-readable program instructions (i.e., computer programs) for executing the above-mentioned pedestrian search method, and can solve the technical problem of how to efficiently search for pedestrians in complex scenes. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided in the present application are the same as the beneficial effects of the pedestrian search method provided in the above-mentioned embodiment, and will not be elaborated here.
[0163] The present application also provides a computer program product, including a computer program, which implements the steps of the above-mentioned pedestrian search method when executed by a processor.
[0164] The computer program product provided by this application can solve the technical problem of how to efficiently search for pedestrians in complex scenes. Compared with the prior art, the beneficial effects of the computer program product provided by this application are the same as the beneficial effects of the pedestrian search method provided by the above embodiment, which will not be repeated here.
[0165] The above descriptions are only some embodiments of the present application, and are not intended to limit the patent scope of the present application. All equivalent structural changes made using the contents of the present application specification and drawings under the technical concept of the present application, or direct / indirect applications in other related technical fields are included in the patent protection scope of the present application.
Claims
1. A pedestrian search method, It is characterized in that The method comprises: Construct an initial pedestrian search model based on the input layer, output layer, deformable convolution layer, pyramid vision transformer network, deformable mapper, first feedforward layer, and pedestrian search module; Obtaining a pedestrian search data set, and dividing the pedestrian search data set into a training set and a test set; Using the training set and the test set to train and test the initial pedestrian search model to obtain a target pedestrian search model; The target pedestrian search model is used to perform pedestrian search on pedestrian images.
2. The method according to claim 1, It is characterized in that The pedestrian search module includes an encoder, a detector and a discriminative re-identification decoder; The step of constructing an initial pedestrian search model according to the input layer, the output layer, the deformable convolution layer, the pyramid visual transformer network, the deformable mapper, the first feedforward layer and the pedestrian search module comprises: Construct a backbone network based on the input layer, deformable convolutional layer, pyramid vision transformer network, and deformable mapper; constructing a neck network based on the first feed-forward layer and the encoder; constructing a head network based on the detector, the discriminative re-ID decoder, and the output layer; An initial pedestrian search model is constructed according to the backbone network, the neck network and the head network.
3. The method according to claim 2, It is characterized in that The step of using the target pedestrian search model to search for pedestrians in a pedestrian picture comprises: Inputting a pedestrian picture into the target pedestrian search model, and extracting a multi-scale feature map through the deformable convolution layer and the pyramid visual transformer network; Mapping the multi-scale feature map to different feature layers through the deformable mapper to obtain a first layer feature map, a second layer feature map, and a third layer feature map; The first layer feature map, the second layer feature map, and the third layer feature map are processed by the neck network and the head network to obtain pedestrian search results.
4. The method according to claim 3, It is characterized in that The encoder comprises a first multi-scale deformable attention module and a second feed-forward layer, and the discriminative re-ID decoder comprises a third multi-scale deformable attention module; The step of obtaining pedestrian search results by processing the first layer feature map, the second layer feature map, and the third layer feature map through the neck network and the head network includes: Processing the first-layer feature map, the second-layer feature map, and the third-layer feature map through the first feed-forward layer to obtain a feed-forward feature map; The feedforward feature map is processed by the first multi-scale deformable attention module and the second feedforward layer, and normalized to obtain an encoded feature map; Processing the encoding feature map by the detector to obtain a preliminary decoding feature map; Processing and normalizing the preliminary decoding feature map through the third multi-scale deformable attention module to obtain a target decoding feature map; The target decoding feature map is processed through the output layer to obtain category prediction, bounding box coordinates and re-identification features, and the category prediction, bounding box coordinates and re-identification features are used as pedestrian search results.
5. The method according to claim 4, It is characterized in that The detector includes a second multi-scale deformable attention module, a multi-head attention module and a third feed-forward layer; The step of processing the encoding feature map by the detector to obtain a preliminary decoding feature map comprises: Processing and normalizing the encoded feature map by the second multi-scale deformable attention module to obtain a multi-scale attention feature map; Processing and normalizing the multi-scale attention feature map through the multi-head attention module to obtain a multi-head attention feature map; The multi-head attention feature map is processed by the third feed-forward layer to obtain a preliminary decoding feature map.
6. The method according to claim 1, It is characterized in that The step of using the training set and the test set to train and test the initial pedestrian search model to obtain a target pedestrian search model comprises: Using the training set to train the initial pedestrian search model to obtain a reference pedestrian search model; Using the test set to test the reference pedestrian search model, to obtain a test result; When the test result does not meet the preset requirements, return to the step of using the training set to train the initial pedestrian search model to obtain a reference pedestrian search model, until the test result meets the preset requirements to obtain a target pedestrian search model.
7. The method according to claim 6, It is characterized in that The step of using the training set to train the initial pedestrian search model to obtain a reference pedestrian search model comprises: Inputting the training set into the initial pedestrian search model to obtain pedestrian detection results, wherein the pedestrian detection results include classification scores, bounding boxes, and re-identification features; Calculate a classification loss, a bounding box loss, and an online instance matching loss according to the classification score, the bounding box, and the re-identification feature; Obtaining a total loss based on the classification loss, the bounding box loss, and the online instance matching loss; The initial pedestrian search model is trained using a back propagation strategy until the total loss is less than or equal to a preset loss value, thereby obtaining a reference pedestrian search model.
8. A pedestrian search device, It is characterized in that The device comprises: A construction module, for constructing an initial pedestrian search model according to an input layer, an output layer, a deformable convolution layer, a pyramid vision transformer network, a deformable mapper, a first feed-forward layer, and a pedestrian search module; A data module, used for obtaining a pedestrian search data set and dividing the pedestrian search data set into a training set and a test set; A training module, used to train and test the initial pedestrian search model using the training set and the test set to obtain a target pedestrian search model; The search module is used to perform pedestrian search on pedestrian images using the target pedestrian search model.
9. A pedestrian search device, It is characterized in that The device comprises: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the pedestrian search method according to any one of claims 1 to 7.
10. A storage medium, It is characterized in that The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, the steps of the pedestrian search method according to any one of claims 1 to 7 are implemented.
Citation Information
Cited By
End-to-end pedestrian re-identification method based on index prediction
CN120126181A
