Video person re-identification method based on spatiotemporal feature extraction and adaptive aggregation

Abstract

The present invention discloses a video pedestrian re-identification method based on spatial-temporal feature extraction and adaptive aggregation. The method steps are as follows: constructing a video pedestrian re-identification model; extracting the initial frame-level feature map through the backbone network; obtaining local features and global features through the spatial feature aggregation module, and extracting frame-level features that fuse local features and global features through the feature aggregation module; the temporal feature aggregation module is used to aggregate frame-level features based on the attention mechanism to extract video-level features for video re-identification. Through this process, an end-to-end video pedestrian re-identification process is realized. Compared with the existing technology, the present invention designs two core modules, the spatial feature aggregation module and the temporal feature aggregation module, which can generate video-level features with discriminative and robustness, can effectively deal with challenges such as frame occlusion and small differences between classes, and can significantly improve the discriminativeness and robustness of video-level representation.

Description

Technical Field

[0001] The present invention relates to the field of video pedestrian re-identification in computer vision, and in particular to a video pedestrian re-identification method based on spatiotemporal feature extraction and adaptive aggregation. Background Art

[0002] Video-based person re-identification (PRI) is a crucial and challenging task in computer vision. It aims to associate similar pedestrians from a set of non-overlapping camera views. Because PRI is not constrained by identification distance and environment, it can be widely used in social security fields such as video surveillance, intelligent security, and crime investigation, and has attracted widespread attention from researchers. Currently, image-based PRI methods mainly focus on extracting appearance features from single-frame images. Due to the limited feature information a single frame can provide, it is difficult to capture discriminative pedestrian appearance features when relying solely on a single frame. Therefore, a set of continuous and temporally complementary multi-frame video clips is particularly important. They can not only provide rich information about pedestrian appearance, but also reveal the connection between motion, viewpoint changes, and identity appearance changes, thereby alleviating the limitations of image-based PRI and providing more representative and discriminative video-level features.

[0003] Currently, research on video-based person re-identification (PRI) primarily focuses on extracting discriminative features of pedestrians. For example, temporal modeling methods capture temporal consistency by modeling inter-frame correlations, while feature aggregation methods extract effective spatiotemporal features to generate robust video-level representations. However, extracting discriminative appearance features remains a challenging task. Temporal feature modeling methods are limited in their ability to capture long-range dependencies and struggle to effectively model cross-frame dependencies. For example, 3D convolutions expand the temporal dimension to capture dynamic information, but their computational complexity limits their ability to model cross-frame dependencies. Optical flow methods, on the other hand, rely on accurate motion estimation and are susceptible to frame occlusion, resulting in suboptimal dynamic feature extraction. Feature aggregation methods, on the other hand, focus on extracting spatially detailed features, which can improve the expressive power of video-level features to a certain extent. However, due to their lack of ability to model inter-frame variations and interactions, these methods struggle to capture static local details, limiting the discriminative and robust nature of the video-level representations. Therefore, existing methods remain challenged in extracting discriminative and robust video-level representations. Summary of the Invention

[0004] The purpose of the present invention is to provide a video pedestrian re-identification method based on spatial-temporal feature extraction and adaptive aggregation to solve the above problems.

[0005] To achieve the above objectives, the present invention adopts a technical solution: a video pedestrian re-identification method based on spatiotemporal feature extraction and adaptive aggregation, the method steps are as follows:

[0006] Step 1: Build a video pedestrian re-identification model, which consists of three main modules: backbone network, spatial feature aggregation module SFA and temporal feature aggregation module TFA;

[0007] Step 2: Given an input video clip, the backbone network is used to extract the initial frame-level feature map, including the shallow feature map S t and deep feature map D t The backbone network classifies and extracts feature images based on the ResNeSt50 neural network model to obtain a shallow feature map S t and deep feature map D t ;

[0008] Step 3: transform the shallow feature map S t and deep feature map D t They are input into the spatial feature aggregation module SFA respectively, and the local features are obtained by the local and global feature extraction modules in SFA respectively. and global feature g t , and extract the frame-level feature Z that fuses local features and global features through the feature aggregation module based on the attention mechanism SA ;

[0009] Step 4: Using frame-level features Z SA As input, the temporal feature aggregation module is used to aggregate frame-level features based on the attention mechanism to extract video-level features z for video re-identification. TA ,Through this process, an end-to-end video pedestrian re-identification process is realized.

[0010] Preferably, in step 1, the spatial feature aggregation module SFA includes a local feature extraction module LFE, a global feature extraction module GFE and a feature fusion module AFA based on an attention mechanism.

[0011] The local feature extraction module LFE detects local salient areas through the attention mechanism and extracts local feature descriptions with geometric characteristics from the shallow features extracted by the backbone network;

[0012] The global feature extraction module GFE extracts global feature descriptions with geometric characteristics from the deep features extracted by the backbone network through generalized mean pooling;

[0013] The feature fusion module AFA is based on the attention mechanism and uses the self-attention module to fuse local features and global features, and obtains frame-level feature descriptions with local and global semantics.

[0014] Preferably, the method for extracting local feature descriptions with geometric characteristics by the local feature extraction module LFE is as follows:

[0015] The shallow features of the t-th frame image extracted by the backbone network are Local feature description based on salient region detection It can be expressed as:

[0016]

[0017] Where, Represents the attention matrix for local salient region detection, * represents the element-wise product operation by channel, Indicates the maximum pooling operation with a kernel of m×m;

[0018] The above attention matrix A s It can be formulated as:

[0019] A s =Softplus(W2(ReLU(W1S t )))#(2)

[0020] Where W1 and W2 represent the linear mapping weights, ReLU(·) and Softplus(·) represent the ReLU and Softplus nonlinear activation functions, respectively.

[0021] Preferably, the method for extracting the global feature description with geometric characteristics by the global feature extraction module GFE is as follows:

[0022] The deep features of the t-th frame image extracted by the backbone network are The global feature description based on generalized mean pooling is where g k It can be expressed as:

[0023]

[0024] Where, Represents the feature map D t The i-th element of the k-th channel in , N = H2 × W2; p>0 is a learnable hyperparameter, initialized to 3.0.

[0025] Preferably, the feature fusion module AFA fuses local features and global features and obtains a frame-level feature description with local and global semantics as follows:

[0026] First, the global feature g t and local features Convert to feature vector of the same dimension;

[0027] Then, the fusion of local features and global features is achieved based on the multi-head self-attention mechanism;

[0028] Finally, based on the multi-head self-attention mechanism, the frame-level feature description z0 is output after fusing local features and global features.

[0029] As an advantage, the feature fusion module AFA combines the global feature g t and local features The method of converting to feature vectors of the same dimension is as follows:

[0030] For the global feature g t , its transformed feature vector x g =W g g t ,in Represents a learnable weight matrix;

[0031] Similarly, for local features Its transformed feature description for:

[0032]

[0033] Where, represents the learnable weight matrix, Indicates that along the spatial dimension Expand into a matrix of dimension C1×N1, where

[0034] Preferably, the feature fusion module AFA realizes the fusion of local features and global features based on the multi-head self-attention mechanism as follows:

[0035] remember where x fuse The token vector is fused to increase the learnable features; with X as input, the output of the self-attention mechanism is The calculation process can be expressed as:

[0036]

[0037] Where, (·) T represents the matrix transpose operation, as well as The input X is the learnable linear matrix W q , W k and W v The projection result after

[0038] Preferably, the feature fusion module AFA outputs a frame-level feature description z0 after fusing local features and global features based on a multi-head attention mechanism as follows:

[0039] Remember Z i is the i-th self-attention output, and the multi-head self-attention output for:

[0040] Z MultiHead =[Z 1 ,Z 2 ,…,Z h ]W O #(6)

[0041] Where h represents the number of multi-head attention, W O Represents the learnable weights. Based on the spatial feature fusion of the multi-head self-attention mechanism, is x fuse The output state of is the frame-level spatial fusion feature description.

[0042] Preferably, the temporal feature aggregation module TFA performs temporal feature aggregation. Temporal feature aggregation is based on an attention mechanism to aggregate the frame-level feature description of the video sequence into a video-level feature representation. The specific method is as follows:

[0043] For video sequences The frame-level feature description after spatial aggregation is recorded as in Indicates the t-th frame I t The frame-level spatial fusion features are then aggregated based on the time domain features of the attention mechanism. TA for:

[0044]

[0045] Where, is the frame-level feature description Z SA The output result of multi-head self-attention is calculated in the same way as formula (5)-(6); Indicates average pooling processing.

[0046] Preferably, in step 1, a spatial loss function and a temporal loss function are designed to supervise frame-level feature extraction and video-level feature extraction respectively, and the model is optimized by jointly minimizing the spatial loss function and the temporal loss function. The design and optimization methods of the spatial loss function and the temporal loss function are as follows:

[0047] Given training data where x n Indicates the label is y n video data, the model optimization process can be formulated as the following minimization problem:

[0048]

[0049] Where W k and b k represents the weights and biases of the kth layer, Represents x n The predicted label of

[0050] Among them, the overall loss function It can be expressed as:

[0051]

[0052] Where, and Represent the spatial domain loss function and the temporal domain loss function respectively, λ space is a hyperparameter;

[0053] Among them, the spatial loss function It is composed of local loss function and global loss function, which is used to supervise the fusion of local features and global features. The formula is as follows:

[0054]

[0055] Where, and Represent the local loss function and the global loss function respectively, and The form is consistent, both are derived from the cross entropy loss and triplet loss constitute;

[0056] Among them, the time domain loss function For supervised video-level feature extraction, the formula is as follows,

[0057]

[0058] Where λ cont is a hyperparameter, represents the contrast loss, which can be expressed as:

[0059]

[0060] Where z a ,z p ,z n Represent the feature vectors of target samples, positive samples and negative samples respectively; P(a) and N(a) represent z a The index set of all positive samples and the index set of all negative samples, d(·) represents the cosine distance function.

[0061] Compared with existing technologies, the advantages of this invention lie in its design of two core modules: a spatial feature aggregation module and a temporal feature aggregation module. The former, based on a frame-level feature extraction strategy that integrates local and global features, provides rich frame-level features for the temporal aggregation module; the latter, based on an attention mechanism, aggregates frame-level features to generate discriminative and robust video-level features. Therefore, the spatial and temporal feature aggregation modules proposed in this model can effectively address challenges such as frame occlusion and small inter-class differences, and can significantly improve the discriminativeness and robustness of video-level representations. BRIEF DESCRIPTION OF THE DRAWINGS

[0062] Figure 1 This is a schematic diagram of the structure of the video pedestrian re-identification method of the present invention;

[0063] Figure 2 This is a structural diagram of the spatial feature aggregation module of the present invention;

[0064] Figure 3 This is a structural diagram of the local feature extraction module of the present invention;

[0065] Figure 4 This is a structural diagram of the timing feature aggregation module of the present invention. DETAILED DESCRIPTION

[0066] The present invention will be further described below. A video pedestrian re-identification method based on spatial-temporal feature extraction and adaptive aggregation is described. Figures 1 to 4 , the steps are as follows,

[0067] Step 1: Build a video pedestrian re-identification model, which consists of three main modules: a backbone network, a spatial feature aggregation module SFA (Spatial Feature Aggregation, SFA) and a temporal feature aggregation module TFA (Temporal Feature Aggregation, TFA);

[0068] Step 2: Given an input video clip, the backbone network is used to extract the initial frame-level feature map, including the shallow feature map S t and deep feature map D t The backbone network classifies and extracts feature images based on the ResNeSt50 neural network model to obtain a shallow feature map S t and deep feature map D t Compared with traditional technologies, ResNet50 as the backbone network, ResNeSt50 can perceive contextual information in a wider range, thereby improving the ability to understand complex scenes;

[0069] Step 3: transform the shallow feature map S tand deep feature map D t They are input into the spatial feature aggregation module SFA respectively, and the local features are obtained by the local and global feature extraction modules in SFA respectively. and global feature g t And extract the frame-level feature Z that fuses local features and global features through the feature aggregation module based on the attention mechanism (Attention-based Feature Aggregation, AFA) SA ;

[0070] Step 4: Using frame-level features Z SA As input, the temporal feature aggregation module is used to aggregate frame-level features based on the attention mechanism to extract video-level features z for video re-identification. TA ,Through this process, an end-to-end video pedestrian re-identification process is realized.

[0071] In order to extract rich frame-level feature descriptions, this paper proposes a spatial feature fusion strategy based on the integration of local features and global features based on the attention mechanism. Figure 2 As shown, the specific implementation of the spatial feature aggregation module SFA is as follows:

[0072] The spatial feature aggregation module SFA includes a local feature extraction module (LFE), a global feature extraction module (GFE), and an attention-based feature fusion module (AFA), which are introduced below:

[0073] The local feature extraction module (LFE) detects local salient areas through the attention mechanism and extracts local feature descriptions with geometric characteristics from the shallow features extracted by the backbone network. The module designed by the present invention can effectively capture the salient areas in the image and accurately focus on the local salient areas of pedestrians even under frame occlusion or complex background conditions, thereby significantly improving the expressiveness and robustness of features. Figure 3 The specific method is as follows:

[0074] The shallow features of the t-th frame image extracted by the backbone network are Local feature description based on salient region detection It can be expressed as:

[0075]

[0076] Where, Represents the attention matrix for local salient region detection, * represents the element-wise product operation by channel, Indicates the maximum pooling operation with a kernel of m×m;

[0077] The above attention matrix A s It can be formulated as:

[0078] A s =Softplus(W2(ReLU(W1S t )))#(2)

[0079] Where W1 and W2 represent the linear mapping weights, ReLU(·) and Softplus(·) represent the ReLU and Softplus nonlinear activation functions, respectively.

[0080] The local features extracted based on the above method have the characteristics of describing the geometric structure of spatially significant regions.

[0081] The global feature extraction module (GFE) extracts a global feature description with geometric properties from the deep features extracted by the backbone network through generalized mean pooling (GeM). This module designed by the present invention can effectively extract a comprehensive and discriminative global feature representation by weighted aggregation of each important pixel in the feature map, thereby further enriching the semantic information of the global feature description. The specific method is as follows:

[0082] The deep features of the t-th frame image extracted by the backbone network are The global feature description based on generalized mean pooling is where g k It can be expressed as:

[0083]

[0084] Where, Represents the feature map D t The i-th element of the k-th channel in , N = H2 × W2; p>0 is a learnable hyperparameter, initialized to 3.0.

[0085] The global feature description extracted based on the above method has a general description of the overall spatial information.

[0086] The attention-based feature fusion module (AFA) uses the self-attention module to fuse local and global features to obtain frame-level feature descriptions with local and global semantics. The frame-level feature descriptions generated by the AFA module through feature fusion are highly discriminative, robust, and semantically consistent, providing a rich frame-level feature representation for the temporal feature aggregation module. The specific method is as follows:

[0087] First, the global feature g tand local features Convert it into a feature vector of the same dimension as follows:

[0088] For the global feature g t , its transformed feature vector x g =W g g t ,in Represents a learnable weight matrix;

[0089] Similarly, for local features Its transformed feature description for:

[0090]

[0091] Where, represents the learnable weight matrix, Indicates that along the spatial dimension Expand into a matrix of dimension C1×N1, where

[0092] Then, the fusion of local features and global features is achieved based on the multi-head self-attention mechanism. The method is as follows:

[0093] remember where x fuse The token vector is fused to increase the learnable features; with X as input, the output of the self-attention mechanism is The calculation process can be expressed as:

[0094]

[0095] Where, (·) T represents the matrix transpose operation, as well as The input X is the learnable linear matrix W q , W k and W v The projection result after .

[0096] Finally, based on the above self-attention mechanism, the output z0 is the frame-level feature description after fusing local features and global features.

[0097] Based on the above-mentioned attention mechanism feature fusion mechanism, the method of outputting the frame-level feature description z0 after fusing local features and global features is as follows:

[0098] Remember Z i is the i-th self-attention output, and the multi-head self-attention output for:

[0099] Z MultiHead=[Z 1 ,Z 2 ,…,Z h ]W O #(6)

[0100] Where h represents the number of multi-head attention, W O Represents the learnable weights. Based on the spatial feature fusion of the multi-head self-attention mechanism, is x fuse The output state of is the frame-level spatial fusion feature description.

[0101] The temporal feature aggregation module TFA of the present invention performs temporal feature aggregation. Temporal feature aggregation is based on the attention mechanism to aggregate the frame-level feature descriptions of the video sequence into video-level feature representations. The module of the present invention comprehensively explores the global correlation between frame-level features and assigns appropriate weights to frames of different importance, thereby effectively mining more discriminative clues and significantly improving the accuracy and robustness of video feature representation. Figure 4 As shown, the specific implementation is as follows:

[0102] For video sequences The frame-level feature description after spatial aggregation is recorded as in Indicates the t-th frame I t The frame-level spatial fusion features are then aggregated based on the time domain features of the attention mechanism. TA for:

[0103]

[0104] Where, is the frame-level feature description Z SA The output result of multi-head self-attention is calculated in the same way as formula (5)-(6), which will not be repeated here. Indicates average pooling processing.

[0105] Based on the temporal feature aggregation of the above-mentioned attention mechanism, video-level feature descriptions for re-identification are extracted.

[0106] The present invention designs a spatial loss function and a temporal loss function for supervising frame-level feature extraction and video-level feature extraction respectively, and realizes model optimization by jointly minimizing the spatial loss function and the temporal loss function.

[0107] The design and optimization methods of the spatial domain loss function and the temporal domain loss function are as follows:

[0108] Given training data where x n Indicates the label is y nvideo data, the model optimization process can be formulated as the following minimization problem:

[0109]

[0110] Where W k and b k represents the weights and biases of the kth layer, Represents x n The predicted label of

[0111] Overall loss function It can be expressed as:

[0112]

[0113] Where, and Represent the spatial domain loss function and the temporal domain loss function respectively, λ space is a hyperparameter;

[0114] Spatial loss function It is composed of local loss function and global loss function, which is used to supervise the fusion of local features and global features. The formula is as follows:

[0115]

[0116] Where, and Represent the local loss function and the global loss function respectively, and The form is consistent, both are derived from the cross entropy loss and triplet loss constitute;

[0117] Time domain loss function For supervised video-level feature extraction, the formula is as follows,

[0118]

[0119] Where λ cont is a hyperparameter, represents the contrast loss, which can be expressed as:

[0120]

[0121] Where, z a ,z p ,z n Represent the feature vectors of target samples, positive samples and negative samples respectively; P(a) and N(a) represent z aThe index set of all positive samples and the index set of all negative samples, d(·) represents the cosine distance function. The model is optimized by jointly minimizing the spatial domain loss function and the temporal domain loss function.

[0122] In summary, the jointly optimized loss function can focus on important features in both spatial and temporal dimensions, effectively guiding the model to generate more discriminative and expressive video-level features during the feature extraction process, and significantly improving the model's target retrieval capability in complex scenarios.

[0123] In order to better demonstrate the effectiveness of the video pedestrian re-identification method based on spatiotemporal feature extraction and adaptive aggregation of the present invention, the present invention is also verified through comparative experiments and ablation experiments.

[0124] This example experiment uses four commonly used benchmark datasets and evaluation indicators for performance evaluation. The specific information is as follows:

[0125] The MARS dataset is the first large-scale video person re-identification dataset. It consists of 20,715 video sequences collected from 6 non-overlapping regions of 1,261 pedestrians, each of which contains an average of 59 frames; 625 pedestrian data are used for the training set and 636 pedestrian data are used for the test set.

[0126] The DukeMTMC-VideoReID (DukeV for short) dataset is collected by cameras in 8 non-overlapping areas on the Duke University campus. It contains 4832 video sequences of 1812 pedestrians, and each video sequence contains an average of 168 frames; 702 pedestrian data are used for the training set and 702 pedestrian data are used for the test set.

[0127] The LS-VID dataset is collected by 15 non-overlapping cameras and contains 3772 pedestrians in 14943 video sequences, each of which contains an average of 200 frames. The training set contains 842 pedestrians and the test set contains 2730 pedestrians.

[0128] The iLIDS-VID dataset, captured by two cameras with non-overlapping areas, consists of 600 video sequences of 300 pedestrians, each containing an average of 71 frames. Of these, 150 pedestrians are used for training, and the remaining pedestrians are used for testing. This dataset contains more occluded samples, making it more challenging.

[0129] The evaluation indicators include the cumulative matching characteristics curve (CMC) and mean average precision (mAP) of Rank 1 to Rank 5.

[0130] The present invention generates video clips from the above benchmark dataset by sparse time series sampling, and performs data enhancement on each frame in the video clip by random flipping and random erasing with a probability of 0.5. The size of each frame is set to 256×128. Adam is used for model optimization in the model training stage. The specific training parameters are set as follows: the initial learning rate and weight decay are 3×10 -4 and 5×10 -4 , the learning rate decays to one tenth of the original after every 40 rounds of training; the batch size is 32, which contains 8 pedestrians, 4 video clips for each pedestrian, and each video clip contains 8 frames.

[0131] In the method evaluation stage, in order to analyze the performance advantages of the method proposed in this invention, it is compared with the existing CNN-based methods and Transformer-based methods. The specific experimental results are shown in Table 1.

[0132] Table 1 Experimental comparison results of the method proposed by the present invention and the existing method

[0133]

[0134] The experimental results in Table 1 show that the proposed method outperforms the comparison methods on four benchmark datasets. Specifically, on the large-scale MARS and LS-VID datasets, the proposed method achieves the best mAP of 89.1% and 85.6%, respectively, significantly outperforming the comparison methods. On the DukeV and iLIDS-VID datasets, the proposed method achieves comparable performance to the comparison methods; for example, on the DukeV dataset, the proposed method achieves a mAP of 97.4%, which is comparable to the highest mAP of 97.6% achieved by the comparison methods.

[0135] On the other hand, in order to analyze the overall impact of each module in the proposed method on the model, the proposed method is divided into three parts, and ablation experiments are performed on two key modules except the backbone network.

[0136] Table 2 Ablation test results of key modules of the method proposed in this invention

[0137]

[0138] Table 2 shows ablation experiments on two key modules of the proposed method. The experimental results in Table 2 show that, compared to the baseline method in the first row of Table 2, the introduction of TFA improves mAP by 0.3%; the introduction of SFA improves mAP by 1.8%; and the simultaneous introduction of TFA and SFA improves mAP and Rank1 by 1.7% and 2.2%, respectively. It can also be seen that compared to the baseline method, the TFA and SFA modules increase the number of parameters and computational complexity, with the computational complexity increasing by 0.34 GFLOPs. These experimental results verify the effectiveness of the spatial feature aggregation of the SFA module and the temporal feature aggregation of the TFA module.

[0139] Table 3 Ablation experiment results of each submodule in the spatial feature aggregation module

[0140]

[0141] Table 3 shows the ablation experiment results of each sub-module of spatial feature aggregation. From the experimental results in Table 3, it can be seen that on the basis of the shallow feature extraction and deep feature extraction of the backbone network, the local feature and global feature aggregation based on the attention mechanism are introduced, and its mAP is improved by 1.2% compared with the baseline method. The above experiments show that effective local feature and global feature aggregation can be achieved based on the attention mechanism. In addition, when the local feature extraction module (LFE) and the global feature extraction module (GFE) are introduced, the performance of the model is further improved. This also verifies that LFE can capture the saliency of local features and GFE can extract more comprehensive global features. These performance improvements further indicate that SFA is conducive to aggregating local and global features to provide richer frame-level features.

[0142] Table 4 Ablation experiment results of time domain feature aggregation module

[0143]

[0144] Table 4 shows the impact of different temporal feature aggregation methods on the performance of the proposed method. The experimental results in Table 4 show that TFA achieves significant improvements in both mAP and Rank 1, two key metrics, compared to other methods. This strongly confirms the effectiveness of TFA in temporal feature aggregation and demonstrates its superior performance for video feature aggregation.

[0145] The above is a detailed introduction to the video pedestrian re-identification method based on spatiotemporal feature extraction and adaptive aggregation provided by the present invention. Specific examples are used in this article to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea. At the same time, for those skilled in the art, according to the ideas of the present invention, there will be changes in the specific implementation methods and application scopes. Changes and improvements to the present invention will be possible without exceeding the concept and scope specified in the appended claims. In summary, the content of this specification should not be understood as limiting the present invention.

Claims

1. A video person re-identification method based on spatiotemporal feature extraction and adaptive aggregation, characterized by: The method steps are as follows: Step 1: Build a video pedestrian re-identification model, which consists of three main modules: backbone network, spatial feature aggregation module SFA and temporal feature aggregation module TFA; The spatial feature aggregation module SFA includes a local feature extraction module LFE, a global feature extraction module GFE and a feature fusion module AFA based on the attention mechanism. The local feature extraction module LFE detects local salient areas through the attention mechanism and extracts local feature descriptions with geometric characteristics from the shallow features extracted by the backbone network; The global feature extraction module GFE extracts global feature descriptions with geometric characteristics from the deep features extracted by the backbone network through generalized mean pooling; The feature fusion module AFA is based on the attention mechanism and uses the self-attention module to fuse local features and global features, and obtains frame-level feature descriptions with local and global semantics; The method for extracting local feature descriptions with geometric characteristics by the local feature extraction module LFE is as follows: Note the first The shallow features of the frame image are , then the local feature description based on salient region detection It can be expressed as: ； Where, represents the attention matrix for local salient region detection, represents the element-wise product operation by channel, Represents the core The maximum pooling operation; The above attention matrix It can be formulated as: ； Where, and represents the linear mapping weight, and Respectively and Non-linear activation functions; The method for extracting global feature descriptions with geometric characteristics by the global feature extraction module GFE is as follows: Note the first The deep features of the frame image are , then the global feature description extracted based on generalized mean pooling is ,in It can be expressed as: ； Where, Representation feature map Middle The first channel elements, ; is a learnable hyperparameter, initialized to 3.0; Step 2: Given an input video clip, the backbone network is used to extract the initial frame-level feature map, including the shallow feature map and deep feature maps ; Step 3: transform the shallow feature map and deep feature maps They are input into the spatial feature aggregation module SFA respectively, and the local features are obtained by the local and global feature extraction modules in SFA respectively. and global features , and extract frame-level features that fuse local features and global features through a feature aggregation module based on the attention mechanism ; Step 4: Frame-level features As input, the temporal feature aggregation module is used to aggregate frame-level features based on the attention mechanism to extract video-level features for video re-identification. ,Through this process, an end-to-end video pedestrian re-identification process is realized.

2. The video person re-identification method based on spatiotemporal feature extraction and adaptive aggregation according to claim 1, characterized in that: The feature fusion module AFA fuses local features and global features and obtains a frame-level feature description with local and global semantics as follows: First, the global features and local features Convert to feature vector of the same dimension; Then, the fusion of local features and global features is achieved based on the multi-head self-attention mechanism; Finally, based on the multi-head self-attention mechanism, the frame-level feature description after fusing local features and global features is output .

3. The video person re-identification method based on spatiotemporal feature extraction and adaptive aggregation according to claim 2, characterized in that: The feature fusion module AFA combines the global features and local features The method of converting to feature vectors of the same dimension is as follows: For global features , the transformed feature vector ,in Represents a learnable weight matrix; Similarly, for local features , the transformed feature description for: ； Where, represents the learnable weight matrix, Indicates that along the spatial dimension The developed dimension is The matrix of .

4. The video person re-identification method based on spatiotemporal feature extraction and adaptive aggregation according to claim 3 is characterized by: The feature fusion module AFA implements the fusion of local features and global features based on the multi-head self-attention mechanism as follows: remember ,in To increase the learnable features, the token vector is integrated; As input, the output of the self-attention mechanism is = The calculation process can be expressed as: ； Where, represents the matrix transpose operation, , and V Input Learnable linear matrix , and The projection result after .

5. The video person re-identification method based on spatiotemporal feature extraction and adaptive aggregation according to claim 4 is characterized by: The feature fusion module AFA is based on a multi-head attention mechanism and outputs a frame-level feature description after fusing local features and global features. The method is as follows: remember For the self-attention output, multi-head self-attention output = for: ； Where, represents the number of multi-head attention, Represents the learnable weights. Based on the spatial feature fusion of the multi-head self-attention mechanism, for The output state of is the frame-level spatial fusion feature description.

6. The video person re-identification method based on spatiotemporal feature extraction and adaptive aggregation according to claim 5, characterized in that: The temporal feature aggregation module TFA performs temporal feature aggregation. Temporal feature aggregation is based on the attention mechanism to aggregate the frame-level feature description of the video sequence into a video-level feature representation. The specific method is as follows: For video sequences , and its frame-level feature description after spatial aggregation is recorded as ,in Indicates the frame The frame-level spatial fusion features are then aggregated based on the time domain features of the attention mechanism. for: ； Where, Frame-level feature description The output result of multi-head self-attention is calculated in the same way as Equations (5)-(6). Indicates average pooling processing.

7. The video person re-identification method based on spatiotemporal feature extraction and adaptive aggregation according to claim 1, characterized in that: In step 1, spatial domain loss function and temporal domain loss function are designed to supervise frame-level feature extraction and video-level feature extraction respectively, and the model is optimized by jointly minimizing the spatial domain loss function and the temporal domain loss function. The design and optimization methods of the spatial domain loss function and the temporal domain loss function are as follows: Given training data ,in Indicates that the label is video data, the model optimization process can be formulated as the following minimization problem: ； Where, and represents the weights and biases of the kth layer, express The predicted label of Among them, the overall loss function It can be expressed as: ； Where, and Represent the spatial domain loss function and the temporal domain loss function respectively, is a hyperparameter; Among them, the spatial loss function It is composed of local loss function and global loss function, which is used to supervise the fusion of local features and global features. The formula is as follows: ； Where, and Represent the local loss function and the global loss function respectively, and The form is consistent, both are derived from the cross entropy loss and triplet loss constitute; Among them, the time domain loss function For supervised video-level feature extraction, the formula is as follows, ； Where, is a hyperparameter, represents the contrast loss, which can be expressed as: ； Where, Represent the feature vectors of target samples, positive samples and negative samples respectively; and Respectively The index set of all positive samples and the index set of all negative samples, Represents the cosine distance function.

Images