An attention-based method for anchor-free pedestrian search

Abstract

This invention discloses an anchorless person search method based on an attention mechanism, comprising the following steps: S1, selecting the PRW dataset as training samples; S2, constructing an anchorless person search network based on an attention mechanism; S3, transmitting the training samples to the anchorless person search network based on an attention mechanism, backpropagating the loss, and repeating this step until training is complete; S4, validating the trained anchorless person search network based on an attention mechanism. If the person search accuracy is lower than a set accuracy, return to step S3; otherwise, proceed to step S5; S5, performing anchorless person search using the trained anchorless person search network based on an attention mechanism. Compared with existing technologies, this invention features a simpler network structure design, faster inference speed, improved person search accuracy, and uses a random erasure data augmentation strategy, which better handles occlusion situations.

Description

Technical Field

[0001] This invention relates to the field of computer vision application technology, specifically including an anchor-free pedestrian search method based on an attention mechanism. Background Technology

[0002] With the widespread use of video surveillance equipment, a vast amount of video surveillance data has been generated. How to utilize this data to make video surveillance more useful to people is a crucial question. Currently, the most common method for video surveillance retrieval is manual searching, which is inefficient and tedious.

[0003] Video surveillance often operates under poor visual conditions, failing to capture the entire person or resulting in an image that is too small, rendering facial recognition ineffective. Pedestrian re-identification emerged to address this issue. Pedestrian re-identification refers to identifying a person after they have already been identified. It often utilizes features such as clothing and body shape, calculating feature similarity to determine if it's the same person—a process similar to manual retrieval. However, a crucial prerequisite for pedestrian re-identification is that the person must be detected first; that is, the data for pedestrian re-identification is cropped from panoramic images. Therefore, re-identification from complete video images requires two steps: pedestrian detection and pedestrian re-identification. These two steps necessitate two separate networks, significantly increasing the computational load.

[0004] Pedestrian search refers to performing pedestrian detection and pedestrian re-identification within a single network. Typically, pedestrian search networks add branches to the object detection network as the output for re-identification. Depending on the object detection network, pedestrian search networks can be categorized into one-stage object detection-based pedestrian search networks and two-stage object detection-based pedestrian search networks. The difference between one-stage and two-stage networks lies in the source of the predefined bounding boxes: in one-stage networks, they are manually specified, while in two-stage networks, they are learned. Therefore, one-stage networks are faster, while two-stage networks are more accurate. Anchor-free networks are a special form of one-stage networks where no predefined bounding boxes are specified. Anchor-bound networks predict the final result based on fine-tuning of the predefined bounding boxes, such as moving them 5 pixels to the left. Anchor-free networks predict the bounding boxes directly from anchor points, for example, the left and right boundaries of the bounding box are 10 pixels away from the anchor points. Because anchor-free networks have fewer predefined bounding boxes, the number of parameters and computational cost are significantly reduced.

[0005] Existing pedestrian search methods are all based on improvements to two-stage object detection, and are basically based on Faster R-CNN as the main structure. Seq directly superimposes an object detection prediction head and a re-identification head after the complete object detection network. Although the accuracy is improved, the computational load is significantly increased, and the network structure is very bloated. Summary of the Invention

[0006] To address the aforementioned shortcomings in existing technologies, this invention provides an anchor-free pedestrian search method based on an attention mechanism, which solves the problems of high computational load and bloated network structure in existing pedestrian search technologies.

[0007] To achieve the above-mentioned objectives, the technical solution adopted by the present invention is as follows: an anchor-free pedestrian search method based on attention mechanism, comprising the following steps: S1, selecting the PRW dataset as training samples;

[0008] S2. Using an improved ResNet50 network as the backbone network, construct an anchor-free pedestrian search network based on an attention mechanism.

[0009] S3. Transfer the training samples to the attention-based, anchor-free pedestrian search network, calculate the loss based on the label values ​​and loss function, backpropagate the loss, and repeat this step until training ends.

[0010] S4. Validate the trained attention-based, anchor-free pedestrian search network. If the pedestrian search accuracy is lower than the set accuracy, return to step S3; otherwise, proceed to step S5.

[0011] S5. An anchor-free pedestrian search network based on the attention mechanism is used for anchor-free pedestrian search after training.

[0012] Furthermore, the attention-based anchor-free pedestrian search network includes an attention-based backbone network, an improved feature fusion neck, a target detection head, and a re-identification head; the output of the attention-based backbone network is connected to the input of the improved feature fusion neck; the output of the improved feature fusion neck is connected to the input of the target detection head and the re-identification head, respectively.

[0013] The method for constructing the backbone network based on the attention mechanism is as follows: on the basis of the ResNet50 network, the convolutional layer of the second stage is replaced with a deformable convolution, and an attention mechanism is added after the third convolutional layer of each bottleneck structure in the three stages after the second stage.

[0014] The improved method for constructing the feature fusion neck is as follows: replace the horizontal convolutional layer and the feature aggregation convolutional layer in the feature fusion neck with deformable convolutions, and perform channel concatenation product on the features of each level during top-down fusion.

[0015] The target detection head is used to further extract improved features and fuse neck features to output the category and predicted bounding box.

[0016] The head is re-identified, and the improved features are fused with the neck output features and normalized to output the re-identified features.

[0017] Furthermore, the object detection header contains two sub-modules:

[0018] The classification submodule is used to perform four 3*3 convolutions and one 1*1 convolution on each layer output of the improved feature fusion neck layer to obtain a 1-channel classification result, which is the probability that the training sample is a human; the last 3*3 convolution uses deformable convolution.

[0019] The regression submodule is used to perform four 3*3 convolutions on each layer output of the improved feature fusion neck to obtain a 4-channel regression result, which corresponds to the distance between the four bounding boxes and the anchor points; the last convolution uses deformable convolution.

[0020] The input of the classification submodule, the output of the regression submodule, and the input of the re-identification head are connected;

[0021] The attention-based, anchor-free pedestrian search network ultimately outputs a 261-dimensional feature vector; where 1 dimension is the pedestrian detection classification score, 4 dimensions represent the predicted bounding box of the pedestrian, and the remaining 256 dimensions are the re-identification features of the pedestrian; the output of the object detection head is connected to the input of the re-identification head.

[0022] Furthermore, the specific implementation method of step S3 is as follows:

[0023] S3-1. Preprocess the training samples and normalize the image data of the training samples using mean and variance.

[0024] S3-2. Randomly scale the normalized image to a specified size, randomly flip the image with a probability of 0.5, and randomly erase the image to obtain the processed image.

[0025] S3-3. The processed image is fed into the attention-based backbone network to obtain feature maps C2, C3, C4 and C5 output from the last four stages of the attention-based backbone network. Feature maps C3, C4 and C5 are then fed into the improved feature fusion neck to obtain outputs P3, P4 and P5. Output P5 is downsampled and convolved to obtain output P6. Output P6 is downsampled and convolved to obtain output P7.

[0026] S3-4. Input the outputs P3, P4, P5, P6, and P7 into the two sub-modules of the target detection head and the re-identification head to obtain the probability, predicted bounding box, and re-identification features of the object corresponding to output P3 as a pedestrian; obtain the probability, predicted bounding box, and re-identification features of the object corresponding to output P4 as a pedestrian; obtain the probability, predicted bounding box, and re-identification features of the object corresponding to output P5 as a pedestrian; obtain the probability, predicted bounding box, and re-identification features of the object corresponding to output P6 as a pedestrian; obtain the probability, predicted bounding box, and re-identification features of the object corresponding to output P7 as a pedestrian.

[0027] S3-5. Unify the label and output format, and convert the label format into an anchor point map that is consistent with the output format; establish the relationship between the anchor point position and the actual position based on the feature map size and the original image size;

[0028] Read the truth labels and create category anchor plots, bounding box anchor plots, and identity anchor plots;

[0029] S3-6. According to the formula:

[0030]

[0031] To obtain the centerness of a point relative to the bounding box, create a centerness anchor point map; where l, r, t, and b represent the distances between the anchor point and the left, right, top, and bottom bounding boxes, respectively, min(·) represents the minimum value, and max(·) represents the maximum value;

[0032] S3-7. Distinguish between positive and negative samples. Record the set of non-background anchor points in the category anchor point map as positive samples. Perform center point sampling on the anchor points. Only anchor points within 1.5 anchor points away from the center of the prediction box are considered as positive samples. There are D anchor points in the center of the prediction box, where D≤9. Calculate the position regression loss, re-identification loss, and centrality loss only for positive samples.

[0033] S3-8. According to the formula:

[0034] L(p t )=-(1-p t ) γ log(p t )

[0035] The classification loss L is obtained, where pt represents the predicted positive sample probability, i.e. the probability that the object belongs to the pedestrian, and log represents the logarithm to the base e.

[0036] S3-9. Calculate the centrality loss based on the centrality and the cross-entropy loss function CrossEntropyLoss.

[0037] S3-10. According to the formula:

[0038]

[0039]

[0040] The regression loss GIOU is obtained, and the centrality loss is used to weight the regression loss GIOU to obtain the weighted regression loss; where A is the ground truth label of the prediction box, B is the predicted value of the prediction box, C is the minimum bounding box of the union of ground truth label A and predicted value B, and IOU is the intersection-union ratio of A and B.

[0041] S3-11. Determine if this is the first iteration. If so, construct and initialize lists LTU and V. List LTU is initially set to 0 and is used to store the features of anchor points for labeled people. List V is used to store the features of anchor points for unlabeled people. Otherwise, randomly sort the anchor point features of a person.

[0042] According to the formula:

[0043]

[0044] The probability p that the k-th anchor point belongs to someone k , make the largest p k The corresponding person identity label is used as the output of the attention-based, anchor-free pedestrian search network; and through probability p k The OIM re-identification loss is calculated using the focal loss function along with the real labels; where X k Let represent the characteristics of the k-th anchor point, and ... T Indicates transpose;

[0045] S3-12. Backpropagate the classification loss, weighted regression loss, and OIM re-identification loss to update the network parameters;

[0046] S3-13. Repeat steps S3-1 to S3-12 according to the pre-set number of iterations to obtain the trained attention-based anchor-free pedestrian search network.

[0047] Furthermore, the proportions of the width and height of the erased area in the normalized image to the width and height of the overall image area are as follows:

[0048] (0.05, 0.05), (0.1, 0.1), (0.15, 0.15), (0.02, 0.06), (0.05, 0.15), (0.07, 0.21), (0.08, 0.24), (0.09, 0.27).

[0049] Furthermore, the specific implementation method of step S4 is as follows:

[0050] S4-1. Use the attention-based, anchor-free pedestrian search network trained in step S3 to perform pedestrian search on the validation samples.

[0051] S4-2. Based on the probability that the verification sample is a human obtained from the pedestrian search, non-maximum suppression is applied to it to obtain the target detection results and target detection performance indicators.

[0052] S4-3. Set the threshold for IOU, calculate the intersection-union ratio (IOU) between the ground truth bounding box of the person being searched in a certain query sample image and the predicted bounding box detected by the target in the image. If the calculated IOU meets the set threshold for IOU, then the re-identified feature corresponding to the predicted bounding box is the feature of the person being searched, and proceed to step S4-4; otherwise, reselect the verification sample and proceed to step S4-1.

[0053] S4-4. For each image of all verification samples other than the query sample images mentioned above, obtain each ground truth bounding box and its corresponding identity label, and obtain the prediction result;

[0054] S4-5. In each verification sample, iterate through all unidentified labels in the labels to obtain the corresponding ground truth bounding boxes;

[0055] S4-6. Iterate through all prediction results and denote a prediction result as box_pre. Calculate the IOU (Intersection over Union) between the prediction box with negative identity label and box_pre and denote it as IOU_no_label. Obtain the maximum IOU (Intersection over Union) between all prediction boxes with positive ground truth labels and box_pre and denote it as IOU_else_max. If IOU_no_label is greater than IOU_else_max and IOU_no_label is greater than the set threshold, then the prediction result belongs to an unlabeled person. Remove the corresponding prediction box, classification loss, and re-identification features. Otherwise, put the validation sample corresponding to the prediction box into the dataset composed of people with identity labels.

[0056] S4-7. Iterate through the person features of each query sample. For each query sample's person features, iterate through each verification sample in the dataset consisting of people with identity labels. During the iteration, calculate the cosine score of the re-identification features of the query sample and the verification sample. Determine the accuracy of the pedestrian search based on the obtained cosine score. If the accuracy is higher than the predetermined accuracy, proceed to step S5. If it is lower than the predetermined accuracy, return to step S3 for retraining.

[0057] The beneficial effects of the present invention are: (1) The pedestrian search network of the present invention is based on the anchorless target detection algorithm, rather than the common Faster RCNN target detection algorithm. The pedestrian search network of the present invention has a simple network structure, does not require an RPN network, and has a faster inference speed.

[0058] (2) This invention modifies the backbone network ResNet50. A GC context channel attention mechanism module and deformable convolution are added to the BottleNeck structure. The former can make full use of context features, while the latter can align ROI regions, making the re-identification features more focused on the human body region. The combination of the two greatly improves the re-identification accuracy.

[0059] (3) In order to improve the shortcomings of OIM loss in anchorless networks, this invention takes advantage of the characteristics of OIM backpropagation, and randomly sorts the features of all anchor points for each person with an identity label. This makes the re-identification features more robust to the anchor point position, and ultimately improves the detection accuracy.

[0060] (4) This invention improves the evaluation metrics. Due to large pedestrian displacements between frames or the presence of people with similar clothing, it is impossible to completely determine the identity label of a person. Therefore, there are fuzzy samples in the dataset with an identity label of -1. In the original evaluation metrics, a high score for a fuzzy sample indicates poor network performance. This invention considers fuzzy samples to be of no reference value, and therefore does not consider fuzzy samples when calculating the metrics.

[0061] (5) The present invention uses a random erasure data augmentation strategy in pedestrian search, which can better deal with occlusion. Attached Figure Description

[0062] Figure 1 Network structure diagram;

[0063] Figure 2 An improved backbone network structure diagram;

[0064] Figure 3 Here is a diagram of the attention module structure;

[0065] Figure 4 Preprocessing flowchart selected for improved computational re-identification accuracy;

[0066] Figure 5 Flowchart for calculating OIM loss;

[0067] Figure 6 This is the overall flowchart. Detailed Implementation

[0068] The specific embodiments of the present invention are described below to enable those skilled in the art to understand the present invention. However, it should be understood that the present invention is not limited to the scope of the specific embodiments. For those skilled in the art, various changes are obvious as long as they are within the spirit and scope of the present invention as defined and determined by the appended claims. All inventions utilizing the concept of the present invention are protected.

[0069] like Figure 1 As shown, the anchor-free pedestrian search network includes an attention-based backbone network, an improved feature fusion neck, a target detection head, and a re-identification head; the output of the attention-based backbone network is connected to the input of the improved feature fusion neck; the output of the improved feature fusion neck is connected to the input of the target detection head and the re-identification head, respectively.

[0070] The improved method for constructing the feature fusion neck is as follows: replace the horizontal convolutional layer and the feature aggregation convolutional layer in the feature fusion neck with deformable convolutions, and perform channel concatenation product on the features of each level during top-down fusion.

[0071] The object detection head is used to perform instance normalization on the output of the improved feature fusion neck.

[0072] The object detection header contains two sub-modules:

[0073] The classification submodule is used to perform four 3*3 convolutions and one 1*1 convolution on each layer output of the improved feature fusion neck layer to obtain a 1-channel classification result, which is the probability that the training sample is a human; the last 3*3 convolution uses deformable convolution.

[0074] The regression submodule is used to perform four 3*3 convolutions on each layer output of the improved feature fusion neck to obtain a 4-channel regression result, which corresponds to the distance between the four bounding boxes and the anchor points; the last convolution uses deformable convolution.

[0075] The input of the classification submodule, the output of the regression submodule, and the input of the re-identification head are connected;

[0076] The network ultimately outputs a 261-dimensional feature vector; where 1 dimension is the pedestrian detection classification score, 4 dimensions represent the predicted bounding box of the pedestrian, and the remaining 256 dimensions are the re-identification features of the pedestrian; the output of the object detection head is connected to the input of the re-identification head.

[0077] like Figure 2As shown, the construction method of the backbone network based on the attention mechanism is as follows: on the basis of the ResNet50 network, the convolutional layer of the second stage is replaced with a deformable convolutional layer, and a BN function and a ReLU operation are added after the first convolutional layer and the deformable convolutional layer; an attention mechanism is added after the third convolutional layer of each bottleneck structure in the three stages after the second stage.

[0078] like Figure 3 As shown, the GC attention module performs convolution or normalization operations on the feature map. The input feature map is represented by [NCHW], where N is the batch size, C is the number of channels, H is the height, and W is the width. Conv indicates that the convolution produces a 1-channel feature map; softmax indicates that the feature map is normalized; view1 indicates that the normalized feature map is transformed; view2 indicates that the feature map [NCHW] is transformed; the transformed feature maps are multiplied; the resulting matrix is ​​convolved once by Conv1, the result of the convolution is normalized and ReLU is performed, and the resulting matrix is ​​convolved again by Conv2; the result of Conv2 is added to the feature map [NCHW] and output.

[0079] like Figure 4 As shown, a trained attention-based, anchor-free pedestrian search network is used to perform pedestrian search on the validation samples.

[0080] Based on the probability that the verification sample is a human obtained from the pedestrian search, non-maximum suppression is applied to it to obtain the target detection results and target detection performance indicators;

[0081] Set an IOU threshold, calculate the Intersection over Union (IOU) of the ground truth bounding box of the person being searched in a query sample image and the predicted bounding box of the object detected in the image. If the calculated IOU meets the set IOU threshold, the re-identified feature corresponding to the predicted bounding box is the feature of the person being searched, and proceed to step S4-4; otherwise, reselect a verification sample and proceed to step S4-1.

[0082] For each image of all validation samples other than the query sample images mentioned above, obtain each ground truth bounding box and its corresponding identity label, and obtain the prediction result;

[0083] In each verification sample, iterate through all unidentified labels in the labels to obtain the corresponding ground truth bounding boxes;

[0084] Iterate through all prediction results and denote a prediction result as box_pre; calculate the IOU based on the prediction box with negative identity label and box_pre and denote it as IOU_no_label; obtain the maximum IOU based on all prediction boxes with positive ground truth labels and box_pre and denote it as IOU_else_max; if IOU_no_label is greater than IOU_else_max and IOU_no_label is greater than the set threshold, then the prediction result belongs to an unlabeled person, and the corresponding prediction box, classification loss and re-identification features are removed; otherwise, the validation sample corresponding to the prediction box is put into the dataset composed of people with identity labels.

[0085] S4-7. Iterate through the person features of each query sample. For each person feature of a query sample, iterate through each verification sample in the dataset consisting of people with identity labels. During the iteration, calculate the cosine score of the re-identification features of the query sample and the verification sample. Determine the accuracy of the pedestrian search based on the obtained cosine score.

[0086] like Figure 5 As shown, anchor points for a target are obtained from the anchor point map, typically 9 anchor points; each anchor point has 256-dimensional features. The order of the 9 anchor points in the anchor point map is shuffled and the order of the shuffled anchor points is recorded. The shuffled anchor point order is used to replace the initial anchor point order, and finally the OIM loss is calculated.

[0087] like Figure 6 As shown, an anchor-free person search method based on an attention mechanism includes the following steps:

[0088] S1. Select the PRW dataset as the training sample;

[0089] S2. Using an improved ResNet50 network as the backbone network, construct an anchor-free pedestrian search network based on an attention mechanism.

[0090] S3. Transfer the training samples to the attention-based, anchor-free pedestrian search network, calculate the loss based on the label values ​​and loss function, backpropagate the loss, and repeat this step until training ends.

[0091] S4. Validate the trained attention-based, anchor-free pedestrian search network. If the pedestrian search accuracy is lower than the set accuracy, return to step S3; otherwise, proceed to step S5.

[0092] S5. An anchor-free pedestrian search network based on the attention mechanism is used for anchor-free pedestrian search after training.

[0093] The specific implementation method of step S3 is as follows:

[0094] S3-1. Preprocess the training samples and normalize the image data of the training samples using mean and variance.

[0095] S3-2. Randomly scale the normalized image to a specified size, randomly flip the image with a probability of 0.5, and randomly erase the image to obtain the processed image.

[0096] S3-3. The processed image is fed into the attention-based backbone network to obtain feature maps C2, C3, C4 and C5 output from the last four stages of the attention-based backbone network. Feature maps C3, C4 and C5 are then fed into the improved feature fusion neck to obtain outputs P3, P4 and P5. Output P5 is downsampled and convolved to obtain output P6. Output P6 is downsampled and convolved to obtain output P7.

[0097] S3-4. Input the outputs P3, P4, P5, P6, and P7 into the two sub-modules of the target detection head and the re-identification head to obtain the probability, predicted bounding box, and re-identification features of the object corresponding to output P3 as a pedestrian; obtain the probability, predicted bounding box, and re-identification features of the object corresponding to output P4 as a pedestrian; obtain the probability, predicted bounding box, and re-identification features of the object corresponding to output P5 as a pedestrian; obtain the probability, predicted bounding box, and re-identification features of the object corresponding to output P6 as a pedestrian; obtain the probability, predicted bounding box, and re-identification features of the object corresponding to output P7 as a pedestrian.

[0098] S3-5. Unify the label and output format, and convert the label format into an anchor point map that is consistent with the output format; establish the relationship between the anchor point position and the actual position based on the feature map size and the original image size;

[0099] Read the truth labels and create category anchor plots, bounding box anchor plots, and identity anchor plots;

[0100] S3-6. According to the formula:

[0101]

[0102] To obtain the centerness of a point relative to the bounding box, create a centerness anchor point map; where l, r, t, and b represent the distances between the anchor point and the left, right, top, and bottom bounding boxes, respectively, min(·) represents the minimum value, and max(·) represents the maximum value;

[0103] S3-7. Distinguish between positive and negative samples. Record the set of non-background anchor points in the category anchor point map as positive samples. Perform center point sampling on the anchor points. Only anchor points within 1.5 anchor points away from the center of the prediction box are considered as positive samples. There are D anchor points in the center of the prediction box, where D≤9. Calculate the position regression loss, re-identification loss, and centrality loss only for positive samples.

[0104] S3-8. According to the formula:

[0105] L(p t )=-(1-p t ) γ log(p t )

[0106] We obtain the classification loss L, where p t This represents the predicted positive sample probability, i.e., the probability that the object belongs to a pedestrian, and log represents the logarithm to the base e.

[0107] S3-9. Calculate the centrality loss based on the centrality and the cross-entropy loss function CrossEntropyLoss.

[0108] S3-10. According to the formula:

[0109]

[0110]

[0111] The regression loss GIOU is obtained, and the centrality loss is used to weight the regression loss GIOU to obtain the weighted regression loss; where A is the ground truth label of the prediction box, B is the predicted value of the prediction box, C is the minimum bounding box of the union of ground truth label A and predicted value B, and IOU is the intersection-union ratio of A and B.

[0112] S3-11. Determine if this is the first iteration. If so, construct and initialize lists LTU and V. List LTU is initially set to 0 and is used to store the features of anchor points for labeled people. List V is used to store the features of anchor points for unlabeled people. Otherwise, randomly sort the anchor point features of a person.

[0113] According to the formula:

[0114]

[0115] The probability p that the k-th anchor point belongs to someone k , make the largest p k The corresponding person identity label is used as the output of the attention-based, anchor-free pedestrian search network; and through probability p kThe OIM re-identification loss is calculated using the focal loss function along with the real labels; where X k Let represent the characteristics of the k-th anchor point, and ... T Indicates transpose;

[0116] S3-12. Backpropagate the classification loss, weighted regression loss, and OIM re-identification loss to update the network parameters;

[0117] S3-13. Repeat steps S3-1 to S3-12 according to the pre-set number of iterations to obtain the trained attention-based anchor-free pedestrian search network.

[0118] The proportions of the width and height of the erased area in the normalized image to the width and height of the overall image area are as follows:

[0119] (0.05, 0.05), (0.1, 0.1), (0.15, 0.15), (0.02, 0.06), (0.05, 0.15), (0.07, 0.21), (0.08, 0.24), (0.09, 0.27).

[0120] The trained attention-based, anchor-free pedestrian search network is used for validation. When traversing each dataset consisting of people with identity labels, the cosine score of the re-identified features is calculated. The accuracy of the pedestrian search is judged based on the obtained cosine score. If the accuracy is higher than the predetermined accuracy, proceed to step S5. If it is lower than the predetermined accuracy, return to step S3 for retraining.

[0121] In one embodiment of the present invention, when scaling the normalized training samples of image data to a specified size, five sizes are specified, and the samples are randomly scaled to a specified size for multi-scale training; the length of the LTU list is set to 500; when creating category anchors and identity anchors, all values ​​inside the bounding box are set to the corresponding category and identity; when creating the bounding box anchor map, the values ​​inside the bounding box are set to the distances from the four boundaries of the bounding box to the anchors.

[0122] A trained attention-based, anchor-free pedestrian search neural network is used to perform pedestrian retrieval tasks in scene images.

[0123] (1) Input the image containing the location of the pedestrian to be queried and the image containing the location of the pedestrian to be queried into the trained attention-based anchor-free pedestrian search network; the trained attention-based anchor-free pedestrian search network outputs the feature representation of the pedestrian to be queried.

[0124] (2) Input the set of scene image data to be searched into the trained attention-based anchor-free pedestrian search network. The trained attention-based anchor-free pedestrian search network predicts the location of pedestrians in the scene image and their corresponding re-identification feature representation; store the above data for subsequent pedestrian feature similarity comparison.

[0125] (3) The distance between the features of the target pedestrian and the features of the pedestrian to be searched is measured, and the similarity between the features is calculated. The similarity between the pedestrian features is measured using cosine score.

[0126] (4) For each pedestrian to be queried, display the pedestrians in the scene image to be searched that are similar to the pedestrians in the scene image to be searched, from high to low similarity, and display the relevant information.

[0127] The anchor-free person search network based on the attention mechanism of this invention has a simple structure, does not require an RPN network, and has a faster inference speed. The backbone network ResNet50 has been modified; a GC context channel attention mechanism module and deformable convolutions have been added to the BottleNeck structure. The former can fully utilize contextual features, while the latter can align ROI regions, making the re-identification features more focused on the human body region. The combination of the two greatly improves the re-identification accuracy. To address the shortcomings of OIM loss in anchor-free networks, this invention randomly sorts the features of all anchor points for each person's identity. This makes the re-identification features more robust to anchor point positions, ultimately improving detection accuracy. The evaluation metric has been improved. Due to large pedestrian displacement between frames or similar clothing, the identity of a person cannot be completely determined. Therefore, there are ambiguous samples with an identity label of -1 in the dataset. In the original evaluation metric, a high score for ambiguous samples indicates poor network performance. This invention considers ambiguous samples to be of no reference value and therefore does not consider ambiguous samples when calculating the metric. A random erasure data augmentation strategy is used in the person search, thus better handling occlusion situations.

Claims

1. An anchor-free bounding box pedestrian search method based on an attention mechanism, characterized in that, The method comprises the following steps: S1, selecting a PRW data set as a training sample; S2, using an improved resnet50 network as a backbone network to construct an anchor-free pedestrian search network based on an attention mechanism; S3, transmitting the training sample to the anchor-free pedestrian search network based on the attention mechanism, calculating the loss according to the label value and the loss function, propagating the loss in reverse, and repeating the step until the training is completed; S4, verifying the anchor-free pedestrian search network based on the attention mechanism after training, and if the pedestrian search accuracy is lower than the set accuracy, returning to step S3; Otherwise, go to step S5; S5, using the trained anchor-free pedestrian search network based on the attention mechanism to search for pedestrians; The anchor-free pedestrian search network based on the attention mechanism comprises a backbone network based on the attention mechanism, an improved feature fusion neck, a target detection head and a re-identification head; the output end of the backbone network based on the attention mechanism is connected with the input end of the improved feature fusion neck; The output end of the improved feature fusion neck is connected with the input end of the target detection head and the re-identification head respectively; The construction method of the backbone network based on the attention mechanism is that, on the basis of the resnet50 network, the convolution layer in the second stage is replaced with a deformable convolution, and the attention mechanism is added after the third convolution layer of each bottleneck structure in the three stages after the second stage; The construction method of the improved feature fusion neck is that, the transverse convolution layer and the feature aggregation convolution layer in the feature fusion neck are both replaced with deformable convolution, and the features of each level in the top-down fusion are channel spliced and multiplied; The target detection head is used for further extracting the features of the improved feature fusion neck, and outputs the class and the prediction box; The re-identification head normalizes the output features of the improved feature fusion neck, and outputs the re-identification features.

2. The anchor-free bounding box pedestrian search method based on an attention mechanism according to claim 1, characterized in that, The target detection head comprises two sub-modules: The classification sub-module is used for sequentially performing 4 times of 3*3 convolution and 1 time of 1*1 convolution on each layer output of the improved feature fusion neck, so as to obtain a 1-channel classification result, that is, the probability that the training sample is a person; wherein the last 3*3 convolution uses deformable convolution; The regression sub-module is used for performing 4 times of 3*3 convolution on each layer output of the improved feature fusion neck, so as to obtain a 4-channel regression result corresponding to the distance between the four edge boxes and the anchor point; wherein the last convolution uses deformable convolution; The input end of the classification sub-module, the output end of the regression sub-module and the input end of the re-identification head are connected; The anchor-free pedestrian search network based on the attention mechanism finally outputs a 261-dimensional feature vector; wherein, 1 dimension is the pedestrian detection classification score, 4 dimensions represent the prediction box of the pedestrian, and the remaining 256 dimensions are the re-identification features of the pedestrian; The output end of the target detection head is connected with the input end of the re-identification head.

3. The anchor-free bounding box pedestrian search method based on the attention mechanism according to claim 2, characterized in that, The specific implementation method of step S3 is as follows: S3-1, preprocessing the training sample, and normalizing the image data of the training sample by using the mean and variance; S3-2, the normalized image is randomly scaled to a specified size, and the image is randomly flipped and randomly erased with a probability of 0.5 to obtain a processed image; S3-3, the processed image is sent into the backbone network based on the attention mechanism to obtain feature maps C2, C3, C4 and C5 outputted by the last four stages of the backbone network based on the attention mechanism, and the feature maps C3, C4 and C5 are sent into the improved feature fusion neck to obtain outputs P3, P4 and P5, the output P5 is down-sampling convolution to obtain an output P6, and the output P6 is down-sampling convolution to obtain an output P7; S3-4, the outputs P3, P4, P5, P6 and P7 are sent into two sub-modules of the target detection head and the re-identification head to obtain the probability that the object corresponding to the output P3 belongs to a pedestrian, the prediction box and the re-identification feature; the probability that the object corresponding to the output P4 belongs to a pedestrian, the prediction box and the re-identification feature; the probability that the object corresponding to the output P5 belongs to a pedestrian, the prediction box and the re-identification feature; the probability that the object corresponding to the output P6 belongs to a pedestrian, the prediction box and the re-identification feature; the probability that the object corresponding to the output P7 belongs to a pedestrian, the prediction box and the re-identification feature; S3-5, the labels and output formats are unified, the label format is converted into the form of anchor point map consistent with the output format, and the relationship between the anchor point position and the actual position is established according to the feature map size and the original image size; read the true value label, and make a category anchor point map, a bounding box anchor point map and an identity anchor point map; S3-6, according to the formula: Obtaining the centrality of a point relative to the bounding box Centerness , and making a centrality anchor point map; wherein, l 、 r 、 t and b respectively represent the distance between the anchor point and the left, right, top and bottom bounding boxes, min(·) represents taking the minimum value, and max(·) represents taking the maximum value; S3-7, distinguish between positive and negative samples, and the set of anchor points in the category anchor point map that are not background is recorded as positive samples, and the anchor points are sampled for the center point, only the anchor points within 1.5 anchor point distances from the most central anchor point of the prediction box are taken as positive samples, there are D anchor points in the center of the prediction box, and D≤9; only the positive samples are calculated for the position regression loss and the re-identification loss and the centerness loss; S3-8, according to the formula: obtaining a classification loss L wherein, p t denotes the predicted positive sample probability, i.e. the probability that the object belongs to the class of pedestrians, log denotes the logarithm to the base e 10. S3-9. According to the centrality Centerness and a cross-entropy loss function CrossEntropyLoss to calculate the centrality loss; S3-10, according to the formula: The regression loss GIOU is obtained, and then the centrality loss is used to weight the regression loss GIOU to obtain the weighted regression loss; where... A For the ground truth labels of the predicted bounding boxes, B The predicted value of the prediction box. C For truth labels A and predicted value B The minimum bounding box of the union of rectangles, with IOU being A and B The intersection and union ratio; S3-11, judge whether it is the first iteration, if yes, construct and initialize a list LTU and a list V, wherein the initial value of the list LTU is 0, and the list V is used to store the anchor point features of the person without a label; otherwise, the anchor point features of a certain person are randomly sorted; according to the formula: The probability that the kth anchor point belongs to a certain person The maximum The corresponding person identity label is taken as the output of the pedestrian search network based on the anchor-free attention mechanism; and the probability And the real label is used focal loss The OIM re-identification loss is calculated by the loss function; wherein X k The feature of the kth anchor point is represented by Xk, cat The splicing operation is represented by, The transpose is represented by S3-12, the classification loss, the weighted regression loss and the OIM re-identification loss are back propagated to update the network parameters; S3-13, repeat steps S3-1 to S3-12 according to the preset number of iterations to obtain the trained anchor-free pedestrian search network based on the attention mechanism.

4. The anchor-free bounding box pedestrian search method based on the attention mechanism according to claim 3, characterized in that, The ratio of the width and height of the erasing area of the normalized image to the width and height of the overall area of the image is as follows: (0.05，0.05)、(0.1，0.1)、(0.15，0.15)、(0.02，0.06)、(0.05，0.15)、(0.07，0.21)、(0.08，0.24)、(0.09，0.27)。 5. The anchor-free bounding box pedestrian search method based on the attention mechanism according to claim 3, characterized in that, The specific implementation method of step S4 is as follows: S4-1, performing pedestrian search on the verification sample using the pedestrian search network based on the anchor-free attention mechanism trained in step S3; S4-2, performing non-maximum suppression on the verification sample according to the probability that the verification sample is a human to obtain a target detection result and a target detection performance index; S4-3, setting a threshold of IOU, calculating the intersection over union (IOU) of the real box of the query person in a query sample image and the predicted box detected by the target detection of the image, if the obtained IOU meets the set threshold of IOU, the re-identification feature corresponding to the predicted box is the feature of the query person, and step S4-4 is entered; Otherwise, reselect the verification sample to enter step S4-1; S4-4, obtaining each real box and its corresponding identity label, and obtaining the prediction result for each image of all the verification samples except the query sample image; S4-5, traversing all the unlabeled labels in the label in each verification sample to obtain the corresponding real box; S4-6, traversing all the prediction results, taking a certain prediction result as box_pre, calculating IOU_no_lable according to the prediction box with negative identity label and box_pre, and obtaining IOU_else_max according to all the prediction boxes with positive true value label and box_pre, if IOU_no_lable is greater than IOU_else_max and IOU_no_lable is greater than a set threshold, the prediction result belongs to the unlabeled person, and the corresponding prediction box, classification loss and re-identification feature are removed, otherwise, the verification sample corresponding to the prediction box is put into a dataset composed of persons with identity labels; S4-7, traversing each query sample person feature, traversing each verification sample in the dataset composed of persons with identity labels for each query sample person feature, calculating the cosine score of the re-identification features of the query sample and the verification sample during the traversal, judging the accuracy of the pedestrian search according to the obtained cosine score, if the accuracy is higher than a predetermined accuracy, step S5 is entered, if the accuracy is lower than the predetermined accuracy, step S3 is returned to retrain.

Images