A video pedestrian re-identification method based on attention spatio-temporal graph network

Abstract

The application belongs to the field of artificial intelligence and video pedestrian re-identification, and discloses a video pedestrian re-identification method based on an attention space-time graph network, and proposes a multi-granularity attention space-time graph network method, which aims to pay attention to foreground information without destroying the body structure of pedestrians, reduce background interference, fuse multi-granularity time information, and suppress the feature expression of abnormal frames. The steps are as follows: extracting pedestrian original frame-level features; extracting frame-level pedestrian key region information; extracting global branch features; updating frame-level features by using a multi-granularity space attention graph network; obtaining pedestrian video sequence-level features by using a multi-granularity time attention graph network; and training the network by using a loss function. By using the application, the destruction of the body structure can be reduced, the interference of background noise can be reduced, and the expression of abnormal frame information can be suppressed, so that the performance of video pedestrian re-identification is significantly improved. The application can be widely applied to the field of video pedestrian re-identification as a video pedestrian re-identification method based on an attention space-time graph network.

Description

TECHNICAL FIELD

[0001] The application belongs to the field of artificial intelligence and computer vision, specifically to the field of video pedestrian re-identification technology, and relates to a video pedestrian re-identification method based on an attention spatio-temporal graph network. BACKGROUND

[0002] The pedestrian re-identification task based on videos aims to match video sequences of the same pedestrian individual in non-overlapping camera views. The video pedestrian re-identification technology can be applied to intelligent security and intelligent monitoring fields and has practical significance in searching for criminals, finding missing persons, and intelligent behavior tracking.

[0003] For the pedestrian re-identification task, the algorithm based on video sequences has the advantages of alleviating the problems of limited information provided by single-frame images, great influence of single-frame images by occlusion, light, and background noise, etc. Therefore, the pedestrian re-identification algorithm based on videos has been paid more and more attention, which can effectively reduce the negative effects of occlusion and blur, fully excavate the features in the time and space dimensions, and has great research value and application value in the academic and industrial fields.

[0004] At present, the common research methods in the field of video-based pedestrian re-identification are as follows: (1) using a spatio-temporal weight fusion method to obtain the final pedestrian feature expression of the video sequence level by taking the weighted sum of frame-level features; (2) using the powerful automatic relationship modeling capability of a graph network to capture robust spatio-temporal features; (3) using a three-dimensional convolutional neural network model and its variants to excavate rich time and space information in the video sequence; and (4) using a Transformer to learn long-distance dependencies and obtain global information. The first method defines the video pedestrian re-identification task as a weighted expression of image frame features, which can effectively alleviate the influence of abnormal frames, but often ignores the excavation of spatial structure information of pedestrians in the frame. The second method uses training data labels to supervise message passing in the graph network, thereby generating more accurate feature fusion weights. The third method uses three-dimensional convolution operations on the video sequence to simultaneously extract time clues and spatial features, but the three-dimensional convolution operation is sensitive to the spatial dislocation problem of the time dimension, and the excessive parameters lead to difficulties in model optimization. The fourth method has the advantage of processing long sequence data, but the excessive attention parameters are prone to overfitting. The present application mainly faces the research field of video pedestrian re-identification and proposes a video pedestrian re-identification method based on an attention spatio-temporal graph network.

[0005] In recent years, the key to video-based person re-identification research is to fully utilize the information provided by pedestrian video sequences and mine spatiotemporal cues containing pedestrian information, thereby obtaining discriminative and robust pedestrian feature representations. Graph neural networks have shown good performance in solving the video person re-identification problem. Their superior relation modeling ability can analyze the potential relationships between pedestrian body parts within and between frames, thereby mining spatiotemporal dependency cues in pedestrian video sequences. Yang et al. proposed a spatiotemporal graph convolutional network model. Through joint optimization of spatial and temporal branches, this model can complementaryly extract the appearance information of target pedestrians in video sequences (Yang J, Zheng WS, Yang Q, et al. Spatial-temporal graph convolutional network for video-based person re-identification[C] / / Proceedings of the IEEE / CVF Conference on Computer Vision and PatternRecognition.2020:3289-3299.). Due to factors such as occlusion and changes in shooting angle, pedestrian appearance varies significantly in samples taken from cameras at different angles. Therefore, how to extract spatial body part information with temporal consistency from pedestrian video sequences has become a research hotspot. Yan et al. proposed a multi-granular hypergraph model to model the spatiotemporal dependence in video sequences at multiple granular levels in order to pursue better representation capabilities of video sequences (Yan Y, Qin J, Chen J, et al. Learning multi-granular hypergraphs for video-based person re-identification[C] / / Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition.2020:2899-2908.). Although existing methods are effective, they neglect three important issues. (1) The impact of pedestrian body structure destruction on modeling. Fixedly dividing each image frame in a pedestrian video sequence into multiple horizontal blocks to construct a spatial graph neural network model will destroy the internal information between pedestrian body structures. (2) The impact of background noise. Due to the imperfection of pedestrian image box detection algorithms, some pedestrian image boxes contain too much background information. Using fixed-level blocks as nodes in a graph neural network introduces too much background information, making it difficult to learn pedestrian features. (3) The impact of abnormal frames. Due to occlusion, blurring, changes in lighting, etc., abnormal frames in pedestrian video sequences will affect the final pedestrian video sequence-level features.The purpose of this invention is to solve the above three key problems in the field of video pedestrian re-identification.

[0006] For the first and second questions, this invention proposes a multi-granularity spatial attention map neural network module, which aims to focus on foreground information and reduce background interference without destroying the pedestrian's body structure.

[0007] Regarding the third question, this invention proposes a multi-granularity temporal attention graph neural network module, which aims to enhance the features of discriminative image frames, fuse multi-granularity temporal information, and suppress the feature expression of abnormal frames. Summary of the Invention

[0008] To address the aforementioned problems, the present invention aims to provide a video pedestrian re-identification method based on an attention-based spatiotemporal graph network. This method can explicitly model the spatiotemporal dependencies in pedestrian video sequences and utilizes a multi-granularity spatial attention module to mine different body regions of pedestrians within frames, effectively reducing damage to body structures and minimizing background noise interference. Simultaneously, by transferring and fusing node features through a multi-granularity spatial graph network and a multi-granularity temporal graph network, the expression of abnormal frame information is suppressed, thereby improving the performance of video pedestrian re-identification.

[0009] A video person re-identification method based on attention-based spatiotemporal graph networks includes the following steps:

[0010] Step 1: Use a CNN feature extractor based on an ImageNet pre-trained model to extract image features from each frame of the pedestrian video sequence;

[0011] Step 2: Extract the pedestrian sequence features obtained in Step 1 and input them into the multi-granularity spatial attention module to find key pedestrian region information at each frame level;

[0012] Step 3: Use the fused normalized spatial attention map obtained in Step 2 and the pedestrian video sequence features obtained in Step 1 to extract global branch features;

[0013] Step 4: Extract the regularized spatial attention map constructed in Step 2 to obtain the extended key region, and use the frame feature map obtained in Step 1 to construct spatial graph neural network nodes;

[0014] Step 5: Use the spatial graph neural network nodes obtained in Step 4 to combine with global image features to construct a spatial graph neural network module;

[0015] Step 6: Using the frame-level features and temporal self-attention module output from Step 5, construct multi-granularity attention temporal graph network nodes;

[0016] Step 7: Extract the multi-granularity attention temporal graph network node features output from Step 6. Similar to Step 5, obtain the temporal graph regularization relation matrix. Update the temporal graph network node features, perform node relationship fusion, and finally obtain the pedestrian video sequence-level features.

[0017] Step 8: Train the network using cross-entropy loss and hard sample triplet loss, and optimize the multi-granularity spatial attention module using the overlap penalty loss from Step 2.

[0018] Furthermore, the specific process of step 1 is as follows:

[0019] The CNN feature extractor is a ResNet50 network pre-trained on ImageNet. After removing its penultimate classification layer, the final stride is set to 1 to increase feature resolution. Each pedestrian video sequence in the training set contains T frames. Given a pedestrian video sequence x... i ={I1,I2,...I T}, input its T-frame images into the ResNet50 network, assuming that the extracted features of the i-th frame image are F i Then pedestrian video sequence x i The features are a set of frame-level features:

[0020]

[0021] In the formula, T represents the number of image frames in the pedestrian video sequence; F represents the pedestrian video sequence features; F i This represents the image features extracted from the i-th frame of the pedestrian video sequence. Its dimensions are C×H×W, where C is the number of channels in the feature map, H is the height of the feature map, and W is the width of the feature map. This feature map will be used as the input to the global branch and the graph branch.

[0022] Furthermore, the specific process of step 2 is as follows:

[0023] The multi-granularity spatial attention module consists of R feature learners based on overlap penalty loss; each feature learner comprises two 1×1 convolutional layers separated by ReLU activation. For the feature map F of the i-th frame in the pedestrian video sequence... i Each feature learner with the same structure learns F respectively. i The discrimination region of the pedestrian image, where the spatial attention map generated by the r-th feature learner is:

[0024] g r (H,W)=L r (F i (2)

[0025] In the formula, L rG represents the learner of the r-th feature; r This represents the corresponding spatial attention map, with dimensions H×W, where H represents the height of the spatial attention map, and W is the height of the feature map F. i The height is consistent, W represents the width of the spatial attention map, and is consistent with the feature map F. i The widths are consistent;

[0026] The spatial attention map g generated by the r-th feature learner r (H,W) is subjected to softmax normalization, and the calculation formula is as follows:

[0027]

[0028] In the formula, This represents the spatial attention graph after [0,1] regularization;

[0029] The spatial attention maps obtained from R feature learners are normalized using formula (3) and then fused. The calculation formula is as follows:

[0030]

[0031] In the formula, This represents the element in row a and column b of the normalized spatial attention graph. This represents the fused normalized spatial attention map;

[0032] Normalized spatial attention maps generated by R feature learners are obtained through formula (3), and then optimized using overlap penalty loss. To increase the distinctiveness of key areas, the calculation formula is as follows:

[0033]

[0034] In the formula, R represents the number of feature learners; ⊙ denotes element-wise multiplication, where any two normalized spatial attention maps are multiplied element-wise to obtain a total of . a choice;

[0035] Furthermore, the specific process of step 3 is as follows:

[0036] Step 2 yields the fused normalized spatial attention map. Extract the image features F of the i-th frame from the pedestrian video sequence obtained in step 1. i The two are then multiplied element-wise along the channel dimension, and the calculation formula is as follows:

[0037]

[0038] In the formula, ⊙ represents element-wise multiplication;

[0039] The feature map obtained from formula (6) With the input image features F i The mixture is then subjected to spatial global average pooling and temporal average pooling operations in sequence, calculated using the following formula:

[0040]

[0041] In the formula, s globsl Represents global branch video sequence features; TAP() represents temporal average pooling; GAP() represents spatial global average pooling.

[0042] Furthermore, the specific process of step 4 is as follows:

[0043] Extending the spatial attention map g in step 2 r (H,W), calculate the extended critical region using the following formula:

[0044]

[0045] In the formula, W max Representation of spatial attention graph g r (H,W) represents the spatial x-coordinate of the maximum response value; H max Representation of spatial attention graph g r (H,W) represents the spatial ordinate of the maximum response value; r represents the r-th feature learner; This represents the extended spatial attention mask, where y1 represents the x-coordinate and y2 represents the y-coordinate;

[0046] Extract the image features F of the i-th frame obtained in step 1. i The spatial graph neural network node encoding is constructed, and the calculation formula is as follows:

[0047]

[0048] In the formula, the mapping operation ⊙ represents element-wise multiplication; F i Represents the image features of the i-th frame; m r This represents the expanded spatial attention mask, where r is the r-th feature learner, and there are a total of R feature learners; GAP() represents spatial global average pooling. This represents a spatial graph neural network node generated by the r-th feature learner, with the superscript p indicating a node;

[0049] Furthermore, the specific process of step 5 is as follows:

[0050] The image features F obtained in step 1 of the i-th frame i Spatial global average pooling is performed to obtain global spatial graph nodes. The calculation formula is as follows:

[0051] F global =GAP(Fi (10)

[0052] In the formula, F global Represents a global spatial graph node; GAP() represents global average pooling; F i Represents the features of the i-th frame image;

[0053] Number of nodes N in a spatial graph neural network S The calculation formula is as follows:

[0054] N S =R+1 (11)

[0055] In the formula, N S This indicates the number of nodes in the spatial graph neural network; the subscript S represents space; and R represents the number of feature learners.

[0056] The features of each node in a spatial graph neural network are represented as follows:

[0057]

[0058] In the formula, f p N represents the set of initial features for each node in a spatial graph neural network, with the superscript p indicating a node; S This represents the number of nodes in a spatial graph neural network. The superscript 'S' indicates space; z S This represents the node features of a spatial graph neural network, with dimension N. S ×d, where d represents the number of channels in each node of the spatial graph neural network;

[0059] The formula for calculating the pairwise relationships between nodes in a spatial graph neural network is as follows:

[0060]

[0061]

[0062]

[0063] In the formula, and Let represent the l-th and q-th spatial graph neural network nodes, respectively, with a dimension of 1×d, where d represents the number of channels in the spatial graph neural network node. and The superscript S indicates space; Represents a node and The edge relationship between them; K S and Q S Indicates feature space transformation; and Let represent the weight matrix, which has dimensions d×d; (·) T This represents the matrix transpose operation;

[0064] By introducing a normalization operation, the weights of each row of the edge relationships between nodes in the spatial graph neural network are restricted to the range (0,1). The calculation formula is as follows:

[0065]

[0066] In the formula, t represents the temperature parameter; the subscripts l and q represent the l-th and q-th nodes of the spatial graph neural network; A S This represents the regularized relation matrix, with dimension N. S ×N S ;

[0067] Using a regularized relation matrix representing the node-edge relationships in a spatial graph neural network. The feature z of each node in the spatial graph neural network obtained by updating formula (12) S The update method is as follows:

[0068]

[0069] In the formula, l represents the l-th spatial graph neural network node; This represents the feature of the original l-th spatial graph neural network node; This represents the updated spatial graph neural network node features; This represents a learnable weight matrix with dimensions d×d, where d is the number of channels, and the subscript V is... S Representation space;

[0070] Using the multi-granularity spatial attention module and spatial graph neural network module from step 2, frame-level features are output based on the graph relationship model guided by spatial attention. The calculation formula is as follows:

[0071]

[0072] In the formula, MaxPool() represents the max pooling operation; This is the updated set of features for the spatial graph neural network nodes obtained by formula (17); The frame-level features output by the spatial attention-guided graph relation model have a dimension of 1×d.

[0073] Furthermore, the specific process of step 6 is as follows:

[0074] Extract the frame-level features output from step 5. Construct pedestrian video sequence features from the time dimension set:

[0075]

[0076] In the formula, x represents the frame-level feature obtained after step 5. The set of x, where the dimension of x is T×d; T represents the number of image frames contained in the video sequence;

[0077] The frame-level feature set obtained by formula (19) is transformed into a feature space, and a temporal attention mask is constructed. The calculation formula is as follows:

[0078] W clip =K A (x)Q A (x) T (20)

[0079]

[0080]

[0081] In the formula, W clip Let be the temporal attention mask for the pedestrian video sequence; x be the pedestrian video sequence feature in formula (19); K A (x) and Q A (x) represents the feature space, which has a dimension of T×d, where T is the number of image frames in the video sequence, d is the number of channels, and the subscript A is the attention. and K A (x) and Q A The weight parameter matrix of (x); (·) T This represents the matrix transpose operation;

[0082] The output of formula (20) is extracted and normalized along the time dimension to obtain a regularized temporal attention mask. Then, an averaging operation is performed along the first dimension to output the final temporal attention mask of the pedestrian video sequence. Its dimension is 1×T, where T represents the number of image frames contained in the pedestrian video sequence; based on The multi-granularity attention temporal graph network nodes are constructed using the following calculation formula:

[0083]

[0084] In the formula, Sort() represents sorting the pedestrian video sequence image frames in descending order according to the temporal attention weight; top k () represents the frame-level features of the pedestrian video sequence output by the first k (k = 2, 4, ..., T) formulas (19); Mean() represents the use of averaging to build the temporal graph network node features; Represents the feature at the k-th time point, with the superscript indicating the feature. This indicates that the current variable belongs to the time plot parameters;

[0085] Furthermore, the specific process of step 8 is as follows:

[0086] The network is trained using cross-entropy loss and hard sample triplet loss, and the multi-granular spatial attention module is optimized using the overlap penalty loss from step 2. The total loss function is calculated as follows:

[0087]

[0088] In the formula, the subscript xent represents cross-entropy loss; tri represents hard sample triplet loss; the superscript global represents global branch output features; the superscript graph represents graph branch output features; the superscript frame represents frame-level features; L OP Indicates overlapping penalty loss;

[0089] The beneficial effects of this invention are as follows: By designing a multi-granularity spatial attention map neural network module, this invention mines structural information between different body regions of a pedestrian within a frame, effectively reducing damage to body structure and mitigating background noise interference. By designing a multi-granularity temporal attention map neural network module, this invention effectively enhances discriminative image frame features, fuses multi-granular temporal information, and suppresses feature representation of abnormal frames. Finally, by comprehensively optimizing model parameters through cross-entropy loss, hard-sample triplet loss, and overlap penalty loss, the performance of the video pedestrian re-identification algorithm is improved. Attached Figure Description

[0090] Figure 1 This is a flowchart of the video pedestrian re-identification method based on attention spatiotemporal graph networks of the present invention;

[0091] Figure 2 This is a schematic diagram of a video pedestrian re-identification task; where (a) is the query pedestrian video sequence, (b) is the video retrieval result, and the circle in the lower right corner of (b) indicates a correct match;

[0092] Figure 3 This is a flowchart of the training process in a specific implementation of the present invention; Detailed Implementation

[0093] The specific embodiments of the present invention will be further described below with reference to the accompanying drawings and technical methods.

[0094] like Figure 2 As shown, this invention provides a video person re-identification method based on attention-based spatiotemporal graph networks. The implementation method of this invention is described in detail below with reference to the accompanying drawings, specifically including the following steps:

[0095] 1. Video-based pedestrian re-identification task and feature extraction of each frame image

[0096] like Figure 1As shown, the application of this invention does not limit the source of pedestrian video sequence data. This invention uses the datasets MARS (Zheng L, Bie Z, Sun Y, et al. Mars: A video benchmark for large-scale person re-identification[C] / / Proceedings of the Conference on European Conference on Computer Vision.2016:868-884.) and iLIDS-VID (Hirzer M, Beleznai C, Roth PM, et al. Person re-identification by descriptive and discriminative classification[C] / / Proceedings of the Conference on Scandinavian Conference on Image Analysis.2011:91-102.) as examples. The MARS dataset collected 1,261 pedestrians from six cameras across different viewpoints, containing 20,715 pedestrian video sequences. The MARS dataset is divided into a training set and a test set. The training set contains 625 different pedestrians, and the test set contains 636 different pedestrians, excluding pedestrians who appeared in the training set. The iLIDS-VID dataset contains 600 video sequence samples from 300 different pedestrians, collected from two cameras. The video-based person re-identification task is to match video sequences of the same pedestrian individual across non-overlapping camera views. In this embodiment, we train the model using the MARS dataset and the iLIDS-VID dataset.

[0097] like Figure 3 As shown, a CNN feature extractor based on an ImageNet pre-trained model is used to extract image features from each frame of a pedestrian video sequence;

[0098] The CNN feature extractor is a ResNet50 network pre-trained on ImageNet. After removing its penultimate classification layer, the final stride is set to 1 to increase feature resolution. Each pedestrian video sequence in the training set contains T frames. Given a pedestrian video sequence x... i ={I1,I2,...I T}, input its T-frame images into the ResNet50 network, assuming that the extracted features of the i-th frame image are F i Then pedestrian video sequence x i The features are a set of frame-level features:

[0099]

[0100] In the formula, T represents the number of image frames in the pedestrian video sequence; F represents the pedestrian video sequence features; F i This represents the image features extracted from the i-th frame of the pedestrian video sequence. Its dimensions are C×H×W, where C is the number of channels in the feature map, H is the height of the feature map, and W is the width of the feature map. This feature map will be used as the input to the global branch and the graph branch.

[0101] 2. Locate pedestrian key region information at each frame level

[0102] The pedestrian sequence features obtained in step 1 The information is input into a multi-granular spatial attention module to find key pedestrian regions at each frame level;

[0103] The multi-granularity spatial attention module consists of R feature learners based on overlap penalty loss; each feature learner comprises two 1×1 convolutional layers separated by ReLU activation. For the feature map F of the i-th frame in the pedestrian video sequence... i Each feature learner with the same structure learns F respectively. i The discrimination region of the pedestrian image, where the spatial attention map generated by the r-th feature learner is:

[0104] g r (H,W)=L r (F i (2) In formula, L r G represents the learner of the r-th feature; r This represents the corresponding spatial attention map, with dimensions H×W, where H represents the height of the spatial attention map, and W is the height of the feature map F. i The height is consistent, W represents the width of the spatial attention map, and is consistent with the feature map F. i The widths are consistent;

[0105] The spatial attention map g generated by the r-th feature learner r (H,W) is subjected to softmax normalization, and the calculation formula is as follows:

[0106]

[0107] In the formula, This represents the spatial attention map after [0,1] regularization;

[0108] The spatial attention maps obtained from R feature learners are normalized using formula (3) and then fused. The calculation formula is as follows:

[0109]

[0110] In the formula, This represents the element in row a and column b of the normalized spatial attention graph. This represents the fused normalized spatial attention map;

[0111] Normalized spatial attention maps generated by R feature learners are obtained through formula (3), and then optimized using overlap penalty loss. To increase the distinctiveness of key areas, the calculation formula is as follows:

[0112]

[0113] In the formula, R represents the number of feature learners; ⊙ denotes element-wise multiplication, where any two normalized spatial attention maps are multiplied element-wise to obtain a total of . a choice;

[0114] 3. Extract global branch features

[0115] Use the fused normalized spatial attention map obtained in step 2 Global branch features are extracted from the pedestrian video sequence features F obtained in step 1;

[0116] Step 2 yields the fused normalized spatial attention map. Extract the image features F of the i-th frame from the pedestrian video sequence obtained in step 1. i The two are then multiplied element-wise along the channel dimension, and the calculation formula is as follows:

[0117]

[0118] In the formula, ⊙ represents element-wise multiplication;

[0119] The feature map obtained from formula (6) With the input image features F i The mixture is then subjected to spatial global average pooling and temporal average pooling operations in sequence, calculated using the following formula:

[0120]

[0121] In the formula, s global Represents global branch video sequence features; TAP() represents temporal average pooling; GAP() represents spatial global average pooling.

[0122] 4. Constructing spatial graph neural network nodes

[0123] The regularized spatial attention map obtained using formula (3) The extended key region is obtained, and the frame feature map F obtained in step 1 is used. i Constructing nodes for a spatial graph neural network;

[0124] Extending the spatial attention map g in step 2 r (H,W), calculate the extended critical region using the following formula:

[0125]

[0126] In the formula, W max Representation of spatial attention graph g r (H,W) represents the spatial x-coordinate of the maximum response value; H max Representation of spatial attention graph g r (H,W) represents the spatial ordinate of the maximum response value; r represents the r-th feature learner; This represents the extended spatial attention mask, where y1 represents the x-coordinate and y2 represents the y-coordinate;

[0127] Extract the image features F of the i-th frame obtained in step 1. i The spatial graph neural network node encoding is constructed, and the calculation formula is as follows:

[0128]

[0129] In the formula, the mapping operation ⊙ represents element-wise multiplication; F i Represents the image features of the i-th frame; m r This represents the expanded spatial attention mask, where r is the r-th feature learner, and there are a total of R feature learners; GAP() represents spatial global average pooling. This represents a spatial graph neural network node generated by the r-th feature learner, with the superscript p indicating a node;

[0130] 5. Construct a spatial graph neural network module

[0131] The spatial graph neural network modules are constructed by combining the spatial graph neural network nodes obtained in step 4 with global image features.

[0132] The image features F obtained in step 1 of the i-th frame i Spatial global average pooling is performed to obtain global spatial graph nodes. The calculation formula is as follows:

[0133] F global =GAP(F i (10)

[0134] In the formula, F global Represents a global spatial graph node; GAP() represents global average pooling; F i Represents the features of the i-th frame image;

[0135] Number of nodes N in a spatial graph neural network S The calculation formula is as follows:

[0136] N S =R+1 (11) where N S This indicates the number of nodes in the spatial graph neural network; the subscript S represents space; and R represents the number of feature learners.

[0137] The features of each node in a spatial graph neural network are represented as follows:

[0138]

[0139] In the formula, f p N represents the set of initial features for each node in a spatial graph neural network, with the superscript p indicating a node; S This represents the number of nodes in a spatial graph neural network. The superscript 'S' indicates space; z S This represents the node features of a spatial graph neural network, with dimension N. S ×d, where d represents the number of channels in each node of the spatial graph neural network;

[0140] The formula for calculating the pairwise relationships between nodes in a spatial graph neural network is as follows:

[0141]

[0142]

[0143]

[0144] In the formula, and Let represent the l-th and q-th spatial graph neural network nodes, respectively, with a dimension of 1×d, where d represents the number of channels in the spatial graph neural network node. and The superscript S indicates space; Represents a node and The edge relationship between them; K S and Q S Indicates feature space transformation; and Let represent the weight matrix, which has dimensions d×d; (·) T This represents the matrix transpose operation;

[0145] By introducing a normalization operation, the weights of each row of the edge relationships between nodes in the spatial graph neural network are restricted to the range (0,1). The calculation formula is as follows:

[0146]

[0147] In the formula, t represents the temperature parameter; the subscripts l and q represent the l-th and q-th nodes of the spatial graph neural network; AS This represents the regularized relation matrix, with dimension N. S ×N S ;

[0148] Using a regularized relation matrix representing the node-edge relationships in a spatial graph neural network. The feature z of each node in the spatial graph neural network obtained by updating formula (12) S The update method is as follows:

[0149]

[0150] In the formula, l represents the l-th spatial graph neural network node; This represents the feature of the original l-th spatial graph neural network node; This represents the updated spatial graph neural network node features; This represents a learnable weight matrix with dimensions d×d, where d is the number of channels, and the subscript V is... s Representation space;

[0151] Using the multi-granularity spatial attention module and spatial graph neural network module from step 2, frame-level features are output based on the graph relationship model guided by spatial attention. The calculation formula is as follows:

[0152]

[0153] In the formula, MaxPool() represents the max pooling operation; This is the updated set of features for the spatial graph neural network nodes obtained by formula (17); The frame-level features output by the spatial attention-guided graph relation model have a dimension of 1×d.

[0154] 6. Construct multi-granularity attention temporal graph network nodes

[0155] Use the frame-level features output from step 5 With the temporal self-attention module, construct multi-granularity attention temporal graph network nodes;

[0156] Extract the frame-level features output from step 5. Construct pedestrian video sequence features from the time dimension set:

[0157]

[0158] In the formula, x represents the frame-level feature obtained after step 5. The set of x, where the dimension of x is T×d; T represents the number of image frames contained in the video sequence;

[0159] The frame-level feature set obtained by formula (19) is transformed into a feature space, and a temporal attention mask is constructed. The calculation formula is as follows:

[0160] W clip =K A (x)Q A (x) T (20)

[0161]

[0162]

[0163] In the formula, W clip Let be the temporal attention mask for the pedestrian video sequence; x be the pedestrian video sequence feature in formula (19); K A (x) and Q A (x) represents the feature space, which has a dimension of T×d, where T is the number of image frames in the video sequence, d is the number of channels, and the subscript A is the attention. and K A (x) and Q A The weight parameter matrix of (x); (·) T This represents the matrix transpose operation;

[0164] The output of formula (20) is extracted and normalized along the time dimension to obtain a regularized temporal attention mask. Then, an averaging operation is performed along the first dimension to output the final temporal attention mask of the pedestrian video sequence. Its dimension is 1×T, where T represents the number of image frames contained in the pedestrian video sequence; based on The multi-granularity attention temporal graph network nodes are constructed using the following calculation formula:

[0165]

[0166] In the formula, Sort() represents sorting the pedestrian video sequence image frames in descending order according to the temporal attention weight; top k () represents the frame-level features of the pedestrian video sequence output by the first k (k = 2, 4, ..., T) formulas (19); Mean() represents the use of averaging to build the temporal graph network node features; Represents the feature at the k-th time point, with the superscript indicating the feature. This indicates that the current variable belongs to the time plot parameters;

[0167] 7. Construct pedestrian video sequence-level features

[0168] Extracting multi-granularity attention temporal graph network node features from step 6 The same process as in step 5 yields the time-map regularized relation matrix. Update the temporal graph network node features; perform inter-node relationship fusion; finally obtain pedestrian video sequence-level features. Its dimension is 1×d;

[0169] 8. Construct the loss function and train the network

[0170] The network is trained using cross-entropy loss and hard sample triplet loss, and the multi-granular spatial attention module is optimized using the overlap penalty loss from step 2. The total loss function is calculated as follows:

[0171]

[0172] In the formula, the subscript xent represents cross-entropy loss; tri represents hard sample triplet loss; the superscript global represents global branch output features; the superscript graph represents graph branch output features; the superscript frame represents frame-level features; L OP Indicates overlapping penalty loss;

[0173] In summary, this invention discloses a video pedestrian re-identification method based on a spatiotemporal attention graph network. This invention designs a multi-granularity spatial attention graph neural network module to mine structural information between different body regions of a pedestrian within a frame, effectively reducing damage to body structure and mitigating background noise interference. Then, by designing a multi-granularity temporal attention graph neural network module, it effectively enhances discriminative image frame features, fuses multi-granular temporal information, and suppresses feature representation of abnormal frames. Finally, by comprehensively optimizing model parameters through cross-entropy loss, hard-sample triplet loss, and overlap penalty loss, the model can learn discriminative and robust video-level pedestrian features, improving the performance of the video pedestrian re-identification algorithm.

[0174] First, a multi-granularity spatial attention graph neural network module is used to mine structural information between different body regions of a pedestrian within a frame using an overlap penalty loss. Then, the graph network is used to fuse node information, which effectively reduces damage to the body structure and reduces the interference of background noise.

[0175] Secondly, a multi-granularity temporal attention map neural network module is adopted to effectively enhance the discriminative image frame features, fuse multi-granularity temporal information, and suppress the feature expression of abnormal frames to obtain pedestrian video sequence-level features.

[0176] Finally, by combining cross-entropy loss, hard sample triplet loss, and overlap penalty loss to optimize the model parameters, the model can learn discriminative and robust pedestrian video-level features, thereby improving the accuracy of the video pedestrian re-identification algorithm.

Claims

1. A video pedestrian re-identification method based on attention-based spatiotemporal graph networks, characterized in that, The steps are as follows: Step 1: Use a CNN feature extractor based on an ImageNet pre-trained model to extract image features from each frame of the pedestrian video sequence; The CNN feature extractor is a ResNet50 network using a pre-trained ImageNet model. After removing its penultimate classification layer, the final stride is set to 1 to increase feature resolution. Each pedestrian video sequence in the training set contains... Frame images, given a sequence of pedestrian videos , and The frame image is input into the ResNet50 network, assuming the first frame is extracted. Frame image features are Then pedestrian video sequence The features are a set of frame-level features: （1）； In the formula, Indicates the number of image frames in a pedestrian video sequence; Represents pedestrian video sequence features; Indicates the number of pedestrian video sequences The feature map extracted from the frame has a dimension of . ,in The number of channels in the feature map. The height of the feature map, This is the width of the feature map; this feature map will serve as the input to both the global branch and the graph branch. Step 2: Extract the pedestrian sequence features from the video obtained in Step 1. The information is input into a multi-granular spatial attention module to find key pedestrian regions at each frame level; The multi-granularity spatial attention module consists of an overlap-penalty loss-based approach. It consists of two feature learners; the feature learner is composed of two... The convolutional layers are separated by ReLU activation, for the first convolutional layer in the pedestrian video sequence. Frame feature map Each feature learner with the same structure learns separately. The discrimination region of the pedestrian image, where the first The spatial attention map generated by the feature learner is as follows: （2）； In the formula, Indicates the first Feature learner; This represents the corresponding spatial attention map, whose dimension is... ,in The height of the spatial attention map is represented by the feature map. Highly consistent The width of the spatial attention map is represented by the feature map. The widths are consistent; The first Spatial attention map generated by a feature learner The softmax normalization process is performed, and the calculation formula is as follows: （3）； In the formula, express Spatial attention map after softmax normalization; Will The normalized spatial attention maps obtained from the feature learners are fused using formula (3) as follows: （4）； In the formula, This indicates the location of the fused normalized spatial attention map. The value at; This represents the spatial attention map generated by the first feature learner. The spatial attention map after softmax normalization is located at... The value at that location, Indicates the first Spatial attention map generated by a feature learner The value of the spatial attention map at (h,w) after softmax normalization; We obtain the results using formula (3). The normalized spatial attention map generated by the feature learner is optimized using overlap penalty loss to increase the discriminative power of key regions. The calculation formula is as follows: （5）； In the formula, The number of feature learners; Element-wise multiplication means that element-wise multiplication of any two normalized spatial attention maps results in a total of a choice; Step 3: Use the fused normalized spatial attention map obtained in Step 2 and pedestrian video sequence features obtained in step 1 Extract global branch features; Step 2 yields the fused normalized spatial attention map. Extract the first pedestrian from the pedestrian video sequence obtained in step 1. Frame Image Features The two are then multiplied element-wise along the channel dimension, and the calculation formula is as follows: （6）； In the formula, This indicates element-wise multiplication; The feature map obtained from formula (6) With input image features The mixture is then subjected to spatial global average pooling and temporal average pooling operations in sequence, calculated using the following formula: （7）； In the formula, Represents global branch video sequence features; Indicates time-averaged pooling; Indicates space-wide average pooling; Step 4: Use the softmax-normalized spatial attention map obtained by formula (3) The extended key region is obtained, and the frame feature map obtained in step 1 is used. Constructing nodes for a spatial graph neural network; Extending the spatial attention map in step 2 The extended critical region is calculated using the following formula: （8）； In the formula, Spatial attention map The horizontal axis of the maximum response value; Spatial attention map The spatial ordinate of the maximum response value; Indicates the first Feature learner; Represents the extended spatial attention mask, where Represents the x-axis, Represents the ordinate; Extract the first step obtained in step 1 Frame Image Features The spatial graph neural network node encoding is constructed, and the calculation formula is as follows: （9）； In the formula, This indicates element-wise multiplication; Indicates the first Frame image features; This represents the extended spatial attention mask. For the first There are feature learners, with a total of _ ... Feature learner; Indicates space-wide average pooling; Indicates by the first The spatial graph neural network nodes generated by the feature learner are labeled with superscripts. Represents a node; Step 5: Use the spatial graph neural network nodes obtained in Step 4 to combine with global image features to construct a spatial graph neural network module; The result obtained in step 1 Frame Image Features Spatial global average pooling is performed to obtain global spatial graph nodes, calculated using the following formula: （10）； In the formula, Represents nodes in the global spatial graph; Indicates space-wide average pooling; Indicates the first Frame image features; Number of nodes in a spatial graph neural network The calculation formula is as follows: （11）； In the formula, Indicates the number of nodes in a spatial graph neural network, subscript Representation space; Indicates the number of feature learners; The features of each node in a spatial graph neural network are represented as follows: （12）； In the formula, This represents the set of initial features for each node in a spatial graph neural network, indicated by the superscript. Represents a node; This represents the number of nodes in a spatial graph neural network. Middle and upper bids Representation space; This represents the node features of a spatial graph neural network, with dimension . , This represents the number of channels in each node of a spatial graph neural network; The formula for calculating the pairwise relationships between nodes in a spatial graph neural network is as follows: （13）； （14）； （15）； In the formula, and They represent the first The and the first There are nodes in a spatial graph neural network, with dimensions of . , This represents the number of channels in a spatial graph neural network node. and superscript Representation space; Represents a node and The edge relationship between them; and Indicates feature space transformation; and This represents the weight matrix, with dimensions of . ; This represents the matrix transpose operation; By introducing a normalization operation, the weights of each row of the edge relationships between nodes in the spatial graph neural network are restricted to a certain value. Within the specified range, the calculation formula is as follows: （16）； In the formula, Indicates temperature parameter; subscript and Indicates the first The and the first A spatial graph neural network node Use the weights of each row in a spatial graph neural network to represent the node-edge relationships. The features of each node in the spatial graph neural network obtained by updating formula (12) The update method is as follows: （17）； In the formula, Indicates the first One spatial graph neural network node; Indicates the original number Features of nodes in a spatial graph neural network; This represents the updated features of the l-th spatial graph neural network node; This represents the learnable weight matrix, with dimension 1. , Number of channels, subscript Representation space; Indicates the original number Features of nodes in a spatial graph neural network; Using the multi-granularity spatial attention module and spatial graph neural network module from step 2, frame-level features are output based on the graph relationship model guided by spatial attention. The calculation formula is as follows: （18）； In the formula, This represents the max pooling operation; The updated set of spatial graph neural network node features obtained by formula (17); The frame-level features output by the spatial attention-guided graph relation model have the following dimensions: ; Step 6: Use the frame-level features output from Step 5 With the temporal self-attention module, construct multi-granularity attention temporal graph network nodes; Extract the frame-level features output from step 5. Construct pedestrian video sequence features from the time dimension set: （19）； In the formula, This represents the frame-level features obtained after step 5. The set, The dimension is ; Indicates the number of image frames contained in the video sequence; The frame-level feature set obtained by formula (19) is transformed into a feature space, and a temporal attention mask is constructed. The calculation formula is as follows: （20）； （21）； （22）； In the formula, Temporal attention mask for pedestrian video sequences; The pedestrian video sequence features are given by formula (19); and The feature space is represented by the dimension . , The number of image frames contained in the video sequence. Number of channels, subscript For attention; and They are respectively and The weight parameter matrix; This represents the matrix transpose operation; The output of formula (20) is extracted and normalized along the time dimension to obtain a regularized temporal attention mask. Then, an averaging operation is performed along the first dimension to output the final temporal attention mask of the pedestrian video sequence. Its dimensions are , Indicates the number of image frames contained in a pedestrian video sequence; based on The multi-granularity attention temporal graph network nodes are constructed using the following calculation formula: （23）； In the formula, This indicates that the pedestrian video sequence image frames are sorted in descending order according to the temporal attention weight; Indicates before fusion The frame-level features of the pedestrian video sequence output by formula (19); This indicates that the characteristics of nodes in a time-plotted network are established using an averaging operation. Indicates the first Each time node feature, superscript This indicates that the current variable belongs to the time plot parameters; Step 7: Extract the multi-granularity attention temporal graph network node features from the output of Step 6. The same process as in step 5 yields the time-map regularized relation matrix. Update the temporal graph network node features; perform node relationship fusion; finally obtain pedestrian video sequence-level features. Its dimensions are ; Step 8: Train the network using cross-entropy loss and hard sample triplet loss, and optimize the multi-granularity spatial attention module using the overlap penalty loss from Step 2. The total loss function is calculated as follows: （24）； In the formula, the subscript Represents cross-entropy loss; Indicates the loss for hard sample triples; superscript Indicates the output feature of the global branch; superscript Indicates the output features of the graph branch; superscript Indicates frame-level features; This indicates overlapping penalty losses.

Images