A video pedestrian re-identification method based on prior knowledge
By combining local and global feature networks and utilizing self-attention mechanisms and shifting window methods, a more comprehensive pedestrian feature representation is generated, which solves the problem of insufficient global feature extraction in existing technologies and improves the accuracy and robustness of video pedestrian re-identification.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- HENAN POLYTECHNIC UNIV
- Filing Date
- 2022-06-13
- Publication Date
- 2026-05-29
AI Technical Summary
In existing technologies, video pedestrian re-identification methods struggle to effectively extract global features, resulting in incomplete pedestrian feature representation and impacting recognition accuracy.
This paper adopts a method that combines local feature networks and global feature networks. Local feature networks are used to extract local features, and global feature networks are used to mine the global structural information of local features. By combining self-attention mechanism and shift window method, a more comprehensive pedestrian feature representation is generated.
It improves the accuracy of pedestrian re-identification, enhances the robustness of the model, reduces the computational cost, can be integrated with existing CNN methods, and is applicable to multiple datasets.
Smart Images
Figure CN115050050B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of video image recognition, and more particularly to a video pedestrian re-identification method based on prior knowledge. Background Technology
[0002] In recent years, pedestrian re-identification has played a crucial role in advanced fields such as safe communities and intelligent surveillance, and has become a research hotspot. The main task of video pedestrian re-identification is to determine whether pedestrians in video sequences obtained from multiple cameras within a non-overlapping field of view are the same person. Compared with other related tasks, video pedestrian re-identification provides video as input, rather than images. Although video can provide more comprehensive appearance information, gait and motion cues in the temporal dimension, it also brings challenges such as pose changes, complex backgrounds, and more occlusion.
[0003] Video pedestrian re-identification feature extraction can be broadly divided into two steps: spatial feature extraction and temporal feature aggregation. First, a convolutional neural network (CNN) is used to extract frame-level spatial features from each image. Then, spatial features from different frames are aggregated in the temporal dimension to form a video representation. Generally, this task uses a CNN as the base network to extract appearance features of pedestrian images. Simultaneously, attention weights for local features are obtained using methods such as pose and attributes. After weighting and enhancing the spatial local features, temporal pooling is performed on the frame-level features in the temporal dimension, or other methods are used to weightedly fuse the inter-frame features to obtain the final feature vector, which is then used to calculate a similarity score.
[0004] Currently, feature extraction from pedestrian images in pedestrian re-identification tasks is dominated by convolutional neural networks, which are characterized by being deeper, broader, and having more branches. However, simply using convolutional networks to extract pedestrian features can only extract relatively detailed local features, easily ignoring global features and failing to construct a comprehensive representation of pedestrian features. Summary of the Invention
[0005] The purpose of this invention is to provide a video pedestrian re-identification method based on prior knowledge. This method can extract the internal structural information of local features using a global feature network and fuse pedestrian features using an enhancement method to obtain a more comprehensive pedestrian feature representation. The network model designed in this invention can be integrated with many current convolutional neural network processing methods and can be used as a base network for feature extraction.
[0006] The present invention adopts the following technical solution:
[0007] A video pedestrian re-identification method based on prior knowledge includes the following steps:
[0008] A: Perform data preprocessing on the training set in the video-based person re-identification dataset;
[0009] B: Construct a video pedestrian re-identification network model based on prior knowledge, including local feature networks and global feature networks;
[0010] The local feature network is used to extract local features from the input image, ultimately obtaining the local features of the pedestrian image. The local feature network consists of invariant convolutional layers, pooling layers, and first to fourth residual layers. The invariant convolutional layers are used for the initial extraction of local features from the input image. The pooling layers are used to select and reduce the dimensionality of the initially extracted features. The first to fourth residual layers are used to extract features layer by layer from the dimensionality-reduced features to obtain the intermediate local features of the image. The feature vector output by the local feature network is processed by a fixed average pooling layer to obtain the local features of the pedestrian image.
[0011] The global feature network is used to extract global structural information from the intermediate local features output by the first to fourth residual layers in the local feature network. The global feature network includes a variable convolutional layer, a normalization layer, and first to fourth encoding layers. The variable convolutional layer is used to unify the number of channels in the feature maps of the intermediate features output by different residual layers of the local feature network. The normalization layer is used to normalize the data within each sample. The first to fourth encoding layers are used to extract global structural information from the input intermediate local features and output feature vectors with the same number of channels but different feature map sizes. The feature vectors output by the global feature network are processed by adaptive average pooling to obtain the global features of the pedestrian image.
[0012] C: Perform average pooling on the local and global features of the pedestrian images obtained in step B in the time dimension to obtain the local and global features of the final pedestrian video sequence.
[0013] D: Using the training set obtained in step A after data preprocessing, train the video pedestrian re-identification network model based on prior knowledge, and finally obtain the trained video pedestrian re-identification network model based on prior knowledge.
[0014] E: Using a trained video pedestrian re-identification network model based on prior knowledge, pedestrian video sequences are re-identified to generate local and global features of the pedestrian video sequences. Then, by applying different weights to the local and global features, the features are optimized and fused to obtain the final feature representation of the fused video sequence.
[0015] Step A includes the following specific steps:
[0016] A1: Obtain pedestrian images from the training set of the video-based pedestrian re-identification dataset, and set the total number of input images m×n for the video pedestrian re-identification network model based on prior knowledge, where m is the number of video sequences input into the video pedestrian re-identification network model based on prior knowledge in each batch, and n is the number of images randomly extracted from the video frames of each video sequence.
[0017] A2: Set the height and width of all input images to 256 pixels and 128 pixels respectively;
[0018] A3: Perform a random horizontal flip on the adjusted input image from step A2 with a probability of 0.5;
[0019] A4: Normalize the input image that was randomly flipped horizontally in step A3;
[0020] A5: Perform random erasure with a probability of 0.5 on the input image after normalization in step A4 to finally obtain the preprocessed input image.
[0021] In step B: each coding layer in the global feature network contains an encoder block. The encoder block consists of a normalization layer, an MSA module, and a 2-layer MLP with a GELU nonlinear unit. The normalization layer is placed before each MSA module and each MLP, and is connected by residuals after each MSA module and each MLP. A linear embedding layer is placed before the encoder block of the first coding layer, and a patch merging layer is placed before the encoder blocks of the second to fourth coding layers.
[0022] The MSA module maps the input feature vector through a linear layer to three spaces: query vector Q, keyword vector K, and value vector V. It calculates the dot product of Q and K to obtain the weight correlation matrix of Q and K, and then uses the Softmax function to normalize the weights. Finally, it performs a weighted summation of the weights and the corresponding value vectors to obtain the final vector.
[0023] The formula for the self-attention mechanism is:
[0024]
[0025] Where Attention(Q,K,V) represents the self-attention mechanism, and Softmax is the multi-class normalization exponential function. is the scaling factor, which can alleviate the gradient vanishing problem introduced by the Softmax function, and B is the relative position deviation matrix.
[0026] In step B, the W-MSA module is used in the global feature network to reduce the computational load, and the SW-MSA module is used to realize information exchange between different windows.
[0027] The calculation process for the encoder block in each coding layer is as follows:
[0028]
[0029] Among them, the W-MSA module refers to the MSA module based on a regular window, and the SW-MSA module refers to the MSA module based on a shifted window. and These represent the outputs of the W-MSA module and the SW-MSA module to l and z, respectively. l and z l+1 represents the output of the MLP module to l, and LN represents the normalization layer.
[0030] In step C, the local and global features output by the local and global feature networks are split into [m,n] in the first dimension m×n, respectively. Then, average pooling is performed on the n pedestrian images of the m video sequences to obtain the final local and global features of the pedestrian video sequences.
[0031]
[0032] Among them, f l f represents local features g The global feature is represented by t, the image number is represented by n, and the pedestrian images in a certain video sequence are represented by n images.
[0033] In step D, the same loss function is used to train both the local feature network and the global feature network. The loss function is the hard triplet loss function and the label smooth cross-entropy loss function.
[0034] In step D:
[0035] The hard triplet loss function states that the distance to the farthest positive sample in the same batch is at least m less than the distance to the nearest negative sample.
[0036]
[0037] In this process, I pedestrians and V video clips are randomly selected from each batch, resulting in a total of I×V video clips in each batch. f represents the local or global features of the pedestrian video sequence, and D represents the Euclidean distance between two features. Then, D(f) i,a ,f i,m D(f) represents the distance between V samples with the same identity as image i. i,a ,f n,m Let (z) represent the distance between samples that are different from image i, where m = 1...V. + =max(z,0) means making the loss value non-negative;
[0038] The label smooth cross-entropy loss function is:
[0039]
[0040] Where, p i,j To reveal your real identity, q i,j Let G be the predicted score represented by the features of sample i,j, where G represents the total number of pedestrian categories and ε is the smoothing parameter.
[0041] In step E, when optimizing and fusing the generated local and global features, different weights are applied to the local and global features for optimization and combination. The recognition performance of the local and global features under different weight combinations is tested, and the final feature representation of the fused video sequence is obtained through the weight combination with the highest recognition accuracy.
[0042]
[0043] Where f is the feature vector of the final pedestrian video, and α and β are the local features f l and global features f g The weight parameters, sign This indicates feature splicing.
[0044] In step B, the local feature network can use a ResNet50 network with a stride of 2 for both convolution and max pooling. The pixel size of the input image is [256, 128]. The dimension of the input array for each batch of input to the network model for image feature extraction is [m, n, 3, 256, 128], where m is the number of video sequences, n is the number of images in each video sequence, 3 is the three RGB channels of the image, and 256 and 128 are the height and width of the image, respectively.
[0045] The prior knowledge-based video pedestrian re-identification method described in this invention utilizes a local feature network to obtain local features, and uses the output of the intermediate layer in the local feature network as the prior knowledge input to the global feature network. By continuously reducing the feature map size and expanding the receptive field in the global feature network, the relationships between local features are fully explored to generate global features of the pedestrian. Simultaneously, a shift window method is used to reduce the computational load of the model. This invention further optimizes and fuses the generated local and global features using different weight combinations to achieve the best recognition accuracy. Attached Figure Description
[0046] Figure 1 This is a schematic diagram of the process of the present invention. Detailed Implementation
[0047] The present invention will now be described in detail with reference to the accompanying drawings and embodiments:
[0048] like Figure 1 As shown, the video pedestrian re-identification method based on prior knowledge described in this invention includes the following specific steps:
[0049] A: Perform data preprocessing on the training set in the video-based person re-identification dataset;
[0050] In this invention, the video-based pedestrian re-identification datasets include PRID-2011, iLIDS-VID, and MARS, all of which are existing publicly available pedestrian re-identification datasets.
[0051] Step A includes the following specific steps:
[0052] A1: Obtain pedestrian images from the training set of the video-based pedestrian re-identification dataset, and set the total number of input images m×n for the video pedestrian re-identification network model based on prior knowledge, where m is the number of video sequences input into the video pedestrian re-identification network model based on prior knowledge in each batch, and n is the number of images randomly extracted from the video frames of each video sequence.
[0053] In this embodiment, the Python programming language can be used to read pedestrian images from the training set of the video-based pedestrian re-identification dataset into memory. The number of video sequences of pedestrian images input into the video pedestrian re-identification network model based on prior knowledge in each batch is set according to the computer performance, m. Then, n images are randomly selected from the video frames of each video sequence, so the total number of images input into the network model at one time is m×n.
[0054] A2: Set the height and width of all input images to 256 pixels and 128 pixels respectively;
[0055] In this embodiment, the image aspect ratio is 2:1, which can better correspond to the human body shape and avoid distortion; the height and width of the image are both multiples of 2, which is beneficial for making full use of computer resources. Considering that if the size of the original image is increased, the image size will increase without changing the benefit, and it will instead burden the running speed of the network model due to the excessively large image, the height and width of the input image in this embodiment are set to 256 pixels and 128 pixels, respectively.
[0056] A3: Perform a random horizontal flip with a probability of 0.5 on the adjusted input image from step A2 to enhance the model's generalization ability and alleviate overfitting.
[0057] A4: Normalize the input image that was randomly flipped horizontally in step A3 to speed up the process of finding the optimal solution during gradient descent.
[0058] Normalization is a standard technique in this field and will not be described in detail here. The mean of the normalization process is [0.485, 0.456, 0.406] and the standard deviation is [0.229, 0.224, 0.225].
[0059] A5: Perform random erasure with a probability of 0.5 on the input image after normalization in step A4 to finally obtain the preprocessed input image.
[0060] In this embodiment, random erasure involves randomly selecting a rectangular area in the original image and replacing the pixels in that area with random values, creating an effect similar to occlusion, which can enhance the robustness of the model.
[0061] B: Construct a video pedestrian re-identification network model based on prior knowledge, including local feature networks and global feature networks;
[0062] In this invention, a local feature network is used to extract local features from the input image, ultimately obtaining the local features of the pedestrian image. The local feature network sequentially includes an invariant convolutional layer, a pooling layer, and first to fourth residual layers. The invariant convolutional layer is used for the initial extraction of local features from the input image. The pooling layer is used to select and reduce the dimensionality of the initially extracted features. The first to fourth residual layers are used to extract features layer by layer from the dimensionality-reduced features, obtaining more abstract and representative intermediate local features of the image. The feature vector output by the local feature network is processed by a fixed average pooling layer to obtain the local features of the pedestrian image.
[0063] In this embodiment, the local feature network can be a ResNet50 network, with a stride of 2 for both convolution and max pooling. The pixel size of the input image is [256, 128]. The dimensions of the input array for each batch of image feature extraction in the network model are [m, n, 3, 256, 128], where m is the number of video sequences, n is the number of images in each video sequence, 3 is the three RGB channels of the image, and 256 and 128 are the height and width of the image, respectively.
[0064] To facilitate the computation of image feature extraction by the network model, the input array dimension needs to be redefined as [mn,3,256,128] before feature extraction. Therefore, the output dimension after processing with invariant convolutional layers and pooling layers becomes [mn,64,64,32]. The main body of the local feature network is divided into the first to fourth residual layers. Each residual layer consists of a residual block composed of three convolutional layers. The sizes of the three convolutional kernels of the residual block are 1×1, 3×3 and 1×1, respectively. The number of residual blocks in each layer is 3, 4, 6 and 3, respectively. The stride of the fourth residual layer is 1. Therefore, the output of the local feature network (i.e. the output of the fourth residual layer) is [mn,2048,16,8]. After processing by the average pooling layer, the local features of the pedestrian image are obtained.
[0065] In this invention, a global feature network is used to extract global structural information of intermediate local features output from the first to fourth residual layers in the local feature network. The global feature network includes a variable convolutional layer, a normalization layer (LN), and first to fourth encoding layers. Each encoding layer contains an encoder block, which consists of a normalization layer, a multi-head self-attention (MSA) module, and a two-layer perceptron (MLP) with a nonlinear unit having a GELU activation function. The normalization layer is placed before each MSA module and each MLP, and is connected after each MSA module and each MLP via residual connections. A linear embedding layer is placed before the encoder block of the first encoding layer, and a patch merging layer is placed before the encoder blocks of the second to fourth encoding layers. The feature vector output by the global feature network (i.e., the output of the fourth encoding layer block) is processed by an adaptive average pooling layer to obtain the global features of the pedestrian image.
[0066] The prior knowledge-based video pedestrian re-identification network model constructed in this invention, consisting of a local feature network and a global feature network, can directly utilize intermediate features from different residual layers in the local feature network as prior knowledge without increasing computational load. Specifically, the intermediate features (feature maps) output from the first to fourth residual layers are directly input into the global feature network, generating more robust pedestrian feature representations. Therefore, this invention can utilize the intermediate features obtained from the first to fourth residual layers in the local feature network at a deeper level to explore the impact of different intermediate features as prior features on global feature extraction.
[0067] In this embodiment, the variable convolutional layer is used to uniformly convert the number of feature map channels of intermediate features output by different residual layers of the local feature network into C. Unlike the standard layer of the local feature network, which standardizes a batch, the standardization layer in this invention is used to standardize each sample. The first to fourth encoding layers are used to extract the global structural information of the intermediate features.
[0068] Since the number of channels and the size of the feature maps output from different residual layers of the local feature network are different during the experiment, the variable convolutional layer is designed as an adaptive variable 1×1 convolutional layer according to the specific output. The number of input channels of the variable convolutional layer is equal to the number of output channels of the first to fourth residual layers, and the number of output channels C of the variable convolutional layer remains fixed.
[0069] The intermediate features output from the first to fourth residual layers are passed through variable convolutional layers and normalization layers (LN) and then input into the first to fourth encoding layers of the global feature network. The intermediate features are divided into blocks according to the feature map size of the feature vector. They are first projected to dimension C through the linear embedding layer in the first encoding layer. Then the feature vector is first input into the encoder block of the first encoding layer and then further extracted through the second to fourth encoding layers.
[0070] In this invention, the Multi-Head Self-Attention (MSA) module is a form of attention mechanism that maps the input feature vector through a linear layer to three spaces: query vector Q, key vector K, and value vector V. The weight correlation matrix of Q and K is obtained by calculating the dot product of Q and K, and then these weights are normalized using the Softmax function. Finally, the weights and their corresponding value vectors are weighted and summed to obtain the final vector.
[0071] The formula for the self-attention mechanism is:
[0072]
[0073] Where Attention(Q,K,V) represents the self-attention mechanism, and Softmax is the multi-class normalization exponential function. is the scaling factor, which can alleviate the gradient vanishing problem introduced by the Softmax function, and B is the relative position deviation matrix.
[0074] The multi-head self-attention mechanism concatenates each attention calculated above as the final output. In this invention, in the global feature network, a conventional window-based MSA module (W-MSA) is used to reduce the computational load, and a shifted window-based MSA module (SW-MSA) is used to solve the problem of information exchange between different windows.
[0075] The calculation process for the encoder block in each coding layer is as follows:
[0076]
[0077] in, and represents the output of the W-MSA module and the SW-MSA module to l, respectively; zl and zl+1 represent the output of the MLP module to l, respectively; S-MSA and SW-MSA represent the multi-head self-attention modules using conventional and shifted window partitioning methods, respectively; and LN represents the normalization layer.
[0078] To reduce the size of the feature map, this invention also sets a patch merging layer between two adjacent encoder blocks, thereby generating a hierarchical representation. After averaging the values in the small windows, the patch merging layer extracts the values at the same position in each small window, assembles them into new features, and then concatenates all the features together. Specifically, the patch merging layer merges 2×2 adjacent feature patches, at which point the feature map size becomes half of the original size, and the feature dimension is 4C. Then, it is downsampled and mapped to a 2C feature space through a linear layer in the patch merging layer.
[0079] In this embodiment, a patch merging layer is added before each of the second to fourth encoder blocks. After passing through the corresponding patch merging layers, the number of features becomes 2C, 4C, and 8C, respectively, and the size of the output feature map becomes 1 / 2, 1 / 4, and 1 / 8 of the original size, respectively. After being transformed by the first to fourth encoding layers of the global feature network, each residual layer of the local feature network outputs feature vectors with the same number of channels but different feature map sizes. The feature vectors output by the global feature network are then subjected to adaptive average pooling to finally obtain the global features of the pedestrian image.
[0080] In this embodiment, the number of encoder blocks is 2, 2, 6, 2 in sequence.
[0081] C: Perform average pooling on the local and global features of the pedestrian images obtained in step B in the time dimension to obtain the local and global features of the final pedestrian video sequence.
[0082] In step C, the local and global features output by the local and global feature networks are split into [m,n] in the first dimension m×n, respectively. Then, average pooling is performed on the n pedestrian images of the m video sequences to obtain the final local and global features of the pedestrian video sequences.
[0083]
[0084] Among them, f l f represents local features g The global feature is represented by t, the image number is represented by n, and the pedestrian images in a certain video sequence are represented by n images.
[0085] D: Using the training set obtained in step A after data preprocessing, train the video pedestrian re-identification network model based on prior knowledge, and finally obtain the trained video pedestrian re-identification network model based on prior knowledge.
[0086] In this invention, when training the video pedestrian re-identification network model based on prior knowledge, the same loss function is used for both the local feature network and the global feature network. The loss function employs both the hard triplet loss function and the label smooth cross-entropy loss function. The resulting local feature f...l The inputs are fed into the hard triplet loss function and the label smoothing cross-entropy loss function to optimize the local feature network; the resulting global features f g The inputs are fed into the same hard triplet loss function and label smooth cross-entropy loss function to optimize the global feature network.
[0087] In this embodiment, I pedestrians and V video clips are randomly selected in a batch, and each batch contains a total of I×V video clips. The hard triplet loss function refers to the condition that the distance to the farthest positive sample in the same batch is at least m less than the distance to the nearest negative sample, that is:
[0088]
[0089] Where f represents the local or global features of the pedestrian video sequence, and D represents the Euclidean distance between two features, then D(f) i,a ,f i,m D(f) represents the distance between V samples with the same identity as image i. i,a ,f n,m Let (z) represent the distance between samples that are different from image i, where m = 1...V. + =max(z,0) means making the loss value non-negative.
[0090] The label smooth cross-entropy loss function is:
[0091]
[0092] Where, p i,j To reveal your real identity, q i,j Let G be the predicted score represented by the features of sample i,j, where G represents the total number of pedestrian categories and ε is the smoothing parameter.
[0093] In this embodiment, when training the video pedestrian re-identification network model based on prior knowledge, the total number of training rounds is set to 150. The Adam optimization algorithm is used, and the learning rate gradually decreases from 0.0001 to 1 / 10 of the current rate in steps [50, 100, 130].
[0094] E: Using a trained video pedestrian re-identification network model based on prior knowledge, pedestrian video sequences are re-identified to generate local and global features of the pedestrian video sequences. Then, by applying different weights to the local and global features, the features are optimized and fused to obtain the final feature representation of the fused video sequence.
[0095] In step E, when optimizing and fusing the generated local and global features, different weights are applied to the local and global features for optimization and combination. The recognition performance of the local and global features under different weight combinations is tested, and the final feature representation of the fused video sequence is obtained by using the weight combination with the highest recognition accuracy.
[0096]
[0097] Where f is the feature vector of the final pedestrian video, and α and β are the local features f l and global features f g The weight parameters, sign This indicates feature splicing.
[0098] The prior knowledge-based video pedestrian re-identification method described in this invention utilizes a local feature network to obtain local features, and uses the output of the intermediate layer in the local feature network as the prior knowledge input to the global feature network. In the global feature network, the feature map size is continuously reduced, the receptive field is expanded, and the relationships between local features are fully explored to generate global features of the pedestrian. Simultaneously, a shift window method is used to reduce the computational load of the model. Furthermore, this invention optimizes and fuses the generated local and global features through different weight combinations to achieve the best recognition accuracy. During training, the prior knowledge-based network model in this invention simultaneously employs cross-entropy loss and triplet loss for both the local and global feature networks. On the large-scale MARS dataset, Rank-1 and mAP reached 86.8% and 80.3%, respectively, an improvement of 3.8% and 3.3% over the benchmark; good results were also achieved on two smaller datasets. This invention not only successfully applies the encoder model to the field of video pedestrian re-identification, but also demonstrates through extensive experiments on several large datasets that the proposed video pedestrian re-identification method based on prior knowledge can enhance the robustness of recognition and effectively improve the accuracy of pedestrian re-identification. At the same time, this method can be integrated with many current CNN methods and is easy to integrate with other methods.
Claims
1. A video pedestrian re-identification method based on prior knowledge, characterized in that, The steps are as follows: A: Perform data preprocessing on the training set in the video-based person re-identification dataset; B: Construct a video pedestrian re-identification network model based on prior knowledge, including local feature networks and global feature networks; The local feature network is used to extract local features from the input image, ultimately obtaining the local features of the pedestrian image. The local feature network consists of invariant convolutional layers, pooling layers, and first to fourth residual layers. The invariant convolutional layers are used for the initial extraction of local features from the input image. The pooling layers are used to select and reduce the dimensionality of the initially extracted features. The first to fourth residual layers are used to extract features layer by layer from the dimensionality-reduced features to obtain the intermediate local features of the image. The feature vector output by the local feature network is processed by a fixed average pooling layer to obtain the local features of the pedestrian image. The global feature network is used to extract global structural information from the intermediate local features output by the first to fourth residual layers in the local feature network. The global feature network includes a variable convolutional layer, a normalization layer, and first to fourth encoding layers. The variable convolutional layer is used to unify the number of channels in the feature maps of the intermediate features output by different residual layers of the local feature network. The normalization layer is used to normalize the data within each sample. The first to fourth encoding layers are used to extract global structural information from the input intermediate local features and output feature vectors with the same number of channels but different feature map sizes. The feature vectors output by the global feature network are processed by adaptive average pooling to obtain the global features of the pedestrian image. C: Perform average pooling on the local and global features of the pedestrian images obtained in step B in the time dimension to obtain the local and global features of the final pedestrian video sequence. D: Using the training set obtained in step A after data preprocessing, train the video pedestrian re-identification network model based on prior knowledge, and finally obtain the trained video pedestrian re-identification network model based on prior knowledge. E: Using the trained video pedestrian re-identification network model based on prior knowledge, pedestrian video sequences are re-identified to generate local and global features of the pedestrian video sequences; Then, by applying different weights to local and global features, the features are optimized and fused to obtain the final feature representation of the fused video sequence.
2. The video pedestrian re-identification method based on prior knowledge according to claim 1, characterized in that, Step A includes the following specific steps: A1: Obtain pedestrian images from the training set of the video-based pedestrian re-identification dataset, and set the total number of input images m×n for the video pedestrian re-identification network model based on prior knowledge, where m is the number of video sequences input into the video pedestrian re-identification network model based on prior knowledge in each batch, and n is the number of images randomly extracted from the video frames of each video sequence. A2: Set the height and width of all input images to 256 pixels and 128 pixels respectively; A3: Perform a random horizontal flip on the adjusted input image from step A2 with a probability of 0.5; A4: Normalize the input image that was randomly flipped horizontally in step A3; A5: Perform random erasure with a probability of 0.5 on the input image after normalization in step A4 to finally obtain the preprocessed input image.
3. The video pedestrian re-identification method based on prior knowledge according to claim 1, characterized in that, In step B: each coding layer in the global feature network contains an encoder block. The encoder block consists of a normalization layer, an MSA module, and a 2-layer MLP with a GELU nonlinear unit. The normalization layer is placed before each MSA module and each MLP, and is connected by residuals after each MSA module and each MLP. A linear embedding layer is placed before the encoder block of the first coding layer, and a patch merging layer is placed before the encoder blocks of the second to fourth coding layers.
4. The video pedestrian re-identification method based on prior knowledge according to claim 3, characterized in that: The MSA module maps the input feature vector through a linear layer to three spaces: query vector Q, keyword vector K, and value vector V. It calculates the dot product of Q and K to obtain the weight correlation matrix of Q and K, and then uses the Softmax function to normalize the weights. Finally, it performs a weighted summation of the weights and the corresponding value vectors to obtain the final vector. The formula for the self-attention mechanism is: Where Attention(Q,K,V) represents the self-attention mechanism, and Softmax is the multi-class normalization exponential function. is the scaling factor, which can alleviate the gradient vanishing problem introduced by the Softmax function, and B is the relative position deviation matrix.
5. The video pedestrian re-identification method based on prior knowledge according to claim 4, characterized in that: In step B, the W-MSA module is used in the global feature network to reduce the computational load, and the SW-MSA module is used to realize information exchange between different windows. The calculation process for the encoder block in each coding layer is as follows: Among them, the W-MSA module refers to the MSA module based on a regular window, and the SW-MSA module refers to the MSA module based on a shifted window. and These represent the outputs of the W-MSA module and the SW-MSA module to l, zl and z, respectively. l+1 represents the output of the MLP module to l, and LN represents the normalization layer.
6. The video pedestrian re-identification method based on prior knowledge according to claim 1, characterized in that: In step C, the local and global features output by the local and global feature networks are split into [m,n] in the first dimension m×n, respectively. Then, average pooling is performed on the n pedestrian images of the m video sequences to obtain the final local and global features of the pedestrian video sequences. Among them, f l f represents local features g The global feature is represented by t, the image number is represented by n, and the pedestrian images in a certain video sequence are represented by n images.
7. The video pedestrian re-identification method based on prior knowledge according to claim 1, characterized in that: In step D, the same loss function is used to train both the local feature network and the global feature network. The loss function is the hard triplet loss function and the label smooth cross-entropy loss function.
8. The video pedestrian re-identification method based on prior knowledge according to claim 7, characterized in that, In step D: The hard triplet loss function states that the distance to the farthest positive sample in the same batch is at least m less than the distance to the nearest negative sample. In this process, I pedestrians and V video clips are randomly selected from each batch, resulting in a total of I×V video clips in each batch. f represents the local or global features of the pedestrian video sequence, and D represents the Euclidean distance between two features. Then, D(f) i,a ,f i,m D(f) represents the distance between V samples with the same identity as image i. i,a ,f n,m Let (z) represent the distance between samples that are different from image i, where m = 1...V. + =max(z,0) means making the loss value non-negative; The label smooth cross-entropy loss function is: Where, p i,j To reveal your real identity, q i,j Let G be the predicted score represented by the features of sample i,j, where G represents the total number of pedestrian categories and ε is the smoothing parameter.
9. The video pedestrian re-identification method based on prior knowledge according to claim 1, characterized in that: In step E, when optimizing and fusing the generated local and global features, different weights are applied to the local and global features for optimization and combination. The recognition performance of the local and global features under different weight combinations is tested, and the final feature representation of the fused video sequence is obtained through the weight combination with the highest recognition accuracy. Where f is the feature vector of the final pedestrian video, and α and β are the local features f l and global features f g The weight parameters, sign This indicates feature splicing.
10. The video pedestrian re-identification method based on prior knowledge according to claim 1, characterized in that: In step B, the local feature network can use a ResNet50 network with a stride of 2 for both convolution and max pooling. The pixel size of the input image is [256, 128]. The dimension of the input array for each batch of input to the network model for image feature extraction is [m, n, 3, 256, 128], where m is the number of video sequences, n is the number of images in each video sequence, 3 is the three RGB channels of the image, and 256 and 128 are the height and width of the image, respectively.