Video pedestrian re-identification method paying attention to long-term and short-term time relation

Abstract

The invention discloses a video pedestrian re-identification method paying attention to a long-term and short-term relationship, which is characterized by comprising the following steps of: firstly, sending training data into a Transform network, obtaining more distinctive features in a video frame by utilizing the strong modeling capability of the Transform network, and then mining time-space information in a video by mining a time relationship and a space clue in the video frame, so that the pedestrian re-identification is realized. Updating the to-be-trained video pedestrian re-identification model through a gradient back propagation algorithm to finally obtain a trained video pedestrian re-identification model, finally obtaining a retrieval feature closest to the cosine distance of the query feature, and taking retrieval data corresponding to the retrieval feature as an identification result of the query data; the pedestrian re-identification method has the advantages that the temporal and spatial information in the video can be better mined by fully mining the temporal relation and spatial clues in the video frame, and the robustness and accuracy of pedestrian re-identification are improved.

Description

Technical Field

[0001] The present invention relates to a video pedestrian re-identification method, in particular to a video pedestrian re-identification method focusing on long-term and short-term time relations. Background Art

[0002] In recent years, with the wide application in fields such as intelligent surveillance, person re-identification has attracted more and more attention. According to the type of query, person re-identification can be divided into two categories: image-based and video-based. Compared with image-based person re-identification, video-based person re-identification has richer temporal and spatial information, which can be used to reduce visual ambiguity, thereby improving the robustness and accuracy of person re-identification; video-based person re-identification is closer to actual application scenarios and has higher practical value. Although video pedestrian re-identification technology has made significant progress, most current methods do not fully utilize the rich temporal and spatial clues in video data, which limits their robustness and accuracy.

[0003] For the extraction of spatial clues, existing methods mainly use attention mechanisms, local features and multi-scale techniques for feature enhancement; although these methods have achieved good results, their origins still have certain limitations. The accuracy of the feature information extracted by these methods depends on the range of their receptive field, and conventional square convolutions cannot adapt well to non-square feature areas. Summary of the invention

[0004] The technical problem to be solved by the present invention is to provide a video pedestrian re-identification method that can well capture spatial and temporal feature information and focus on long-term and short-term temporal relationships, and the recognition result is relatively accurate.

[0005] The technical solution adopted by the present invention to solve the above technical problems is: a video pedestrian re-identification method focusing on long-term and short-term time relationships, comprising the following steps:

[0006] Step 1): Select a training video from a publicly available video dataset, obtain multiple video frames from the training video as training set data, perform data enhancement on the training set data to obtain enhanced training data, and adjust the video frame size to 256×128;

[0007] Step 2): Construct a video person re-identification model to be trained, including a convolutional projection layer, a Transformer network for extracting intra-frame features, a temporal clue layer for mining inter-frame relationships, a multi-scale module for mining spatial relationships, and an aggregation layer for aggregating video frame information. Input the enhanced training data into the video person re-identification model to be trained to extract video pedestrian features. The specific process is as follows:

[0008] Step 2-1): Flatten the three-dimensional features in each enhanced training data in the spatial dimension through the convolutional projection layer, and reconstruct the two-dimensional feature embedding;

[0009] Step 2-2): Extract the intra-frame features of the two-dimensional feature embedding through the Transformer network to obtain an output feature map S1 with a size of 128×210×768;

[0010] Step 2-3): Use Torch function to separate the time features from the 128 dimensions of S1 to obtain a feature map S2 of 16×8×210×768;

[0011] Step 2-4): The time clue layer includes a long-term time relationship module and a short-term time relationship module. S2 is input into the long-term time relationship module and the short-term time relationship module respectively. The specific process is as follows;

[0012] Step 2-4-1): Input S2 into the long-term temporal relationship module to obtain a long-term temporal relationship feature map S3 with a global receptive field, and the size of S3 is 16×8×210×768; input S2 into the short-term temporal relationship module to obtain a feature-enhanced short-term temporal relationship feature map S4, and the size of S4 is 16×8×210×768;

[0013] Step 2-4-2): Add S3 and S4 to obtain the time relationship feature map S5, the size of S5 is 16×8×210×768;

[0014] Step 2-4-3): Perform dimension change operation on S5 through Torch function to obtain 128×210×768 feature map S6;

[0015] Step 2-5): Input S6 into the multi-scale module to learn fine-grained information to obtain feature map S7, wherein the multi-scale module includes a first attention head, a second attention head and a third attention head, the first attention head uses a first convolution window of size (1,1), the second attention head uses a second convolution window of size (1,2), and the third attention head uses a third convolution window of size (2,1);

[0016] Step 2-6): The features of the feature map S7 in other dimensions except the channel dimension c are averaged through the aggregation layer to obtain the final model output feature S8 of size 768×1, and S8 is used as the video pedestrian feature. The other dimensions include the height dimension h, the width dimension w, the data batch b and the time series dimension t;

[0017] Step 3): define the loss function of the video pedestrian re-identification model to be trained, update the video pedestrian re-identification model to be trained by the gradient back propagation algorithm, and stop the training process when the set maximum number of training times is reached to obtain the trained video pedestrian re-identification model;

[0018] Step 4): multiple video frames are obtained from the target video as query data to form a query data set. For any query data in the query data set, the trained video person re-identification model is used to extract features of the query data to obtain corresponding query features.

[0019] Step 5): Obtain multiple video frames from the target video as retrieval data and form a retrieval data set, use the trained video pedestrian re-identification model to extract features from the retrieval data in the retrieval data set to obtain corresponding retrieval features, obtain the retrieval feature with the closest cosine distance to the query feature, and use the retrieval data corresponding to the retrieval feature as the recognition result of the query data.

[0020] Compared with the prior art, the advantages of the present invention are that, firstly, by fully mining the temporal relationship and spatial clues in the video frames, the temporal and spatial information in the video can be better mined, thereby improving the robustness and accuracy of pedestrian re-identification; by feeding the training data into the Transformer network and utilizing the powerful modeling ability of the Transformer network, more discriminative features in the video frames are obtained; the two proposed temporal relationship modules can make full use of the temporal information in the video data, considering both the global receptive field features from the perspective of long-term temporal relationships and the enhanced features from the perspective of short-term temporal relationships, thereby better mining the temporal relationships in the video and further improving the performance of the model; the proposed multi-scale module improves the adaptability and robustness of the model to multi-scale features in terms of spatial feature processing by changing the conventional shape of the convolution kernel; the effectiveness of the model and each module is verified through a large number of experiments and ablation experiments. On three existing pedestrian re-identification datasets, the video pedestrian re-identification models obtained by the present method have achieved the most significant recognition performance.

[0021] Preferably, in step 1), multiple video frames are obtained from the training video as training set data, and each training set data is a video segment consisting of 8 video frames selected using a restricted random sampling strategy. The video frame data selected in this way is more representative, ensuring the difference and effectiveness of the video frames, and can better represent this section of video data.

[0022] Preferably, the data enhancement method includes random horizontal flipping, random padding, random cropping and random erasing. Through the data enhancement method, more diverse data distribution can be simulated, the size of the data set can be increased, and the overfitting of the model to specific data can be reduced, thereby improving the robustness and generalization ability of the model.

[0023] Preferably, the specific process of step 3) is as follows:

[0024] Step 3-1): Set the loss function to the cross entropy loss function L c , Among them, y j represents the label of the enhanced training data, represents the actual output of the video person re-identification model to be trained, N represents the number of samples, 1≤j≤N;

[0025] Step 3-2): The number of forward propagation and back propagation of the enhanced training data in the video pedestrian re-identification model to be trained is set to 30, and the stochastic gradient descent optimizer is used to perform optimization. The initial learning rate is set to 0.02. The stochastic gradient descent optimizer reduces the learning rate by 0.01 every 1000 times until the maximum number of iterations is reached. The iterative process is stopped to obtain the trained video pedestrian re-identification model. DETAILED DESCRIPTION

[0026] The present invention is described in further detail below.

[0027] A video pedestrian re-identification method focusing on long-term and short-term temporal relationships comprises the following steps:

[0028] Step 1): Select training videos from the publicly available video dataset, obtain multiple video frames from the training videos as training set data, perform data enhancement on the training set data to obtain enhanced training data, and adjust the video frame size to 256×128; wherein, obtain multiple video frames from the training videos as training set data, each training set data is a video clip consisting of 8 video frames selected using the restricted random sampling strategy (RRS for short), and the data obtained by this strategy is more representative and effective; data enhancement methods include random horizontal flipping, random padding, random cropping, and random erasing. Data enhancement methods can obtain larger data scales and improve the robustness and generalization ability of the model.

[0029] Step 2): Construct a video person re-identification model to be trained, including a convolutional projection layer, a Transformer network for extracting intra-frame features, a temporal clue layer for mining inter-frame relationships, a multi-scale module for mining spatial relationships, and an aggregation layer for aggregating video frame information. Input the enhanced training data into the video person re-identification model to be trained to extract video pedestrian features. The specific process is as follows:

[0030] Step 2-1): Flatten the three-dimensional features in each enhanced training data in the spatial dimension through the convolutional projection layer, and reconstruct the two-dimensional feature embedding;

[0031] Step 2-2): The intra-frame features of the two-dimensional feature embedding are extracted through the Transformer network to obtain an output feature map S1 with a size of 128×210×768; the Transformer network has a powerful feature extraction capability, which can help the model obtain powerful saliency features.

[0032] Step 2-3): Use Torch function to separate the time features from the 128 dimensions of S1 to obtain a feature map S2 of 16×8×210×768;

[0033] Step 2-4): The time clue layer includes a long-term time relationship module and a short-term time relationship module. S2 is input into the long-term time relationship module and the short-term time relationship module respectively. The specific process is as follows;

[0034] Step 2-4-1): Input S2 into the long-term temporal relationship module to obtain a long-term temporal relationship feature map S3 with a global receptive field, and the size of S3 is 16×8×210×768; input S2 into the short-term temporal relationship module to obtain a feature-enhanced short-term temporal relationship feature map S4, and the size of S4 is 16×8×210×768;

[0035] Step 2-4-2): Add S3 and S4 to obtain the time relationship feature map S5, the size of S5 is 16×8×210×768;

[0036] Step 2-4-3): Perform dimension change operation on S5 through Torch function to obtain 128×210×768 feature map S6; the long-term and short-term time relationship network has a horizontal temporal information transmission mechanism, which can capture key features and save them through the memory transmission stream. This enables the network to learn video features with a global receptive field, thereby alleviating the impact of occlusion. At the same time, through the adjacent frame interactive information transmission mechanism, the feature update flow is used to emphasize the effective subject information in the video frame, thereby learning more significant video features.

[0037] Step 2-5): Input S6 into the multi-scale module to learn fine-grained information to obtain feature map S7, wherein the multi-scale module includes a first attention head, a second attention head and a third attention head. The first attention head uses a first convolution window of size (1,1), the second attention head uses a second convolution window of size (1,2), and the third attention head uses a third convolution window of size (2,1); by using convolution kernels of various sizes, the receptive field of the network is changed, thereby enhancing the adaptability of the model to features of different scales.

[0038] Step 2-6): The features of the feature map S7 in other dimensions except the channel dimension c are averaged through the aggregation layer. The other dimensions include the height dimension h, the width dimension w, the data batch b and the time dimension t, and the final model output feature S8 with a size of 768×1 is obtained, and S8 is used as the video pedestrian feature;

[0039] Step 3): define the loss function of the video pedestrian re-identification model to be trained, update the video pedestrian re-identification model to be trained by the gradient back propagation algorithm, and stop the training process when the set maximum number of training times is reached to obtain the trained video pedestrian re-identification model; the specific process is as follows:

[0040] Step 3-1): Set the loss function to the cross entropy loss function L c , Among them, y j represents the label of the enhanced training data, represents the actual output of the video person re-identification model to be trained, N represents the number of samples, 1≤j≤N;

[0041] Step 3-2): The number of forward propagation and back propagation of the enhanced training data in the video pedestrian re-identification model to be trained is set to 30, and the stochastic gradient descent optimizer is used to perform optimization. The initial learning rate is set to 0.02. The stochastic gradient descent optimizer reduces the learning rate by 0.01 every 1000 times until the maximum number of iterations is reached. The iterative process is stopped to obtain the trained video pedestrian re-identification model;

[0042] Step 4): multiple video frames are obtained from the target video as query data to form a query data set. For any query data in the query data set, the trained video person re-identification model is used to extract features of the query data to obtain corresponding query features.

[0043] Step 5): Obtain multiple video frames from the target video as retrieval data and form a retrieval data set, use the trained video pedestrian re-identification model to extract features from the retrieval data in the retrieval data set to obtain corresponding retrieval features, obtain the retrieval feature with the closest cosine distance to the query feature, and use the retrieval data corresponding to the retrieval feature as the recognition result of the query data.

[0044] Subsequently, the algorithm automatically compares the pedestrian label information to determine whether the trained video pedestrian re-identification model correctly identifies and matches the same pedestrian; finally, the performance of the trained video pedestrian re-identification model is quantitatively analyzed by calculating indicators such as the cumulative matching characteristic curve and average accuracy.

[0045] The video pedestrian re-identification method focusing on long-term and short-term temporal relationships of this embodiment is referred to as this method. This method is compared with the most advanced related 2D algorithms in recent years on MARS and iLIDS-VID datasets. The comparison results are shown in Table 1.

[0046] Table 1

[0047]

[0048] As can be seen from Table 1, this method achieves the highest performance in terms of mAP indicator and achieves relatively excellent performance in terms of Rank-1 indicator.

[0049] This method is compared with the most advanced related 3D algorithms in recent years on the MARS and iLIDS-VID datasets. The comparison results are shown in Table 2.

[0050] Table 2

[0051]

[0052] As can be seen from Table 2, this method achieves the highest performance in terms of mAP indicator and achieves relatively excellent performance in terms of Rank-1 indicator.

Claims

1. A video pedestrian re-identification method focusing on long-term and short-term temporal relationships, characterized in that The following steps are involved: Step 1): Select a training video from a publicly available video dataset, obtain multiple video frames from the training video as training set data, perform data enhancement on the training set data to obtain enhanced training data, and adjust the video frame size to 256×128; Step 2): Construct a video person re-identification model to be trained, including a convolutional projection layer, a Transformer network for extracting intra-frame features, a temporal clue layer for mining inter-frame relationships, a multi-scale module for mining spatial relationships, and an aggregation layer for aggregating video frame information. Input the enhanced training data into the video person re-identification model to be trained to extract video pedestrian features. The specific process is as follows: as follows: Step 2-1): Flatten the three-dimensional features in each enhanced training data in the spatial dimension through the convolutional projection layer, and reconstruct the two-dimensional feature embedding; Step 2-2): Extract the intra-frame features of the two-dimensional feature embedding through the Transformer network to obtain an output feature map S1 with a size of 128×210×768; Step 2-3): Use Torch function to separate the time features from the 128 dimensions of S1 to obtain a feature map S2 of 16×8×210×768; Step 2-4): The time clue layer includes a long-term time relationship module and a short-term time relationship module. S2 is input into the long-term time relationship module and the short-term time relationship module respectively. The specific process is as follows; Step 2-4-1): Input S2 into the long-term temporal relationship module to obtain a long-term temporal relationship feature map S3 with a global receptive field, and the size of S3 is 16×8×210×768; input S2 into the short-term temporal relationship module to obtain a feature-enhanced short-term temporal relationship feature map S4, and the size of S4 is 16×8×210×768; Step 2-4-2): Add S3 and S4 to obtain the time relationship feature map S5, the size of S5 is 16×8×210×768; Step 2-4-3): Perform dimension change operation on S5 through Torch function to obtain 128×210×768 feature map S6; Step 2-5): Input S6 into the multi-scale module to learn fine-grained information to obtain feature map S7, wherein the multi-scale module includes a first attention head, a second attention head and a third attention head, the first attention head uses a first convolution window of size (1,1), the second attention head uses a second convolution window of size (1,2), and the third attention head uses a third convolution window of size (2,1); Step 2-6): The features of the feature map S7 in other dimensions except the channel dimension c are averaged through the aggregation layer. The other dimensions include the height dimension h, the width dimension w, the data batch b and the time dimension t, and the final model output feature S8 with a size of 768×1 is obtained, and S8 is used as the video pedestrian feature; Step 3): define the loss function of the video pedestrian re-identification model to be trained, update the video pedestrian re-identification model to be trained by the gradient back propagation algorithm, and stop the training process when the set maximum number of training times is reached to obtain the trained video pedestrian re-identification model; Step 4): multiple video frames are obtained from the target video as query data to form a query data set. For any query data in the query data set, the trained video person re-identification model is used to extract features of the query data to obtain corresponding query features. Step 5): Obtain multiple video frames from the target video as retrieval data and form a retrieval data set, use the trained video pedestrian re-identification model to extract features from the retrieval data in the retrieval data set to obtain corresponding retrieval features, obtain the retrieval feature with the closest cosine distance to the query feature, and use the retrieval data corresponding to the retrieval feature as the recognition result of the query data.

2. According to claim 1, a video pedestrian re-identification method focusing on long-term and short-term temporal relationships is characterized in that In the step 1), multiple video frames are obtained from the training video as training set data, and each training set data is a video segment consisting of 8 video frames selected using a restricted random sampling strategy.

3. A video pedestrian re-identification method focusing on long-term and short-term temporal relationships according to claim 1 or 2, characterized in that The data enhancement methods include random horizontal flipping, random padding, random cropping and random erasing.

4. The video pedestrian re-identification method focusing on long-term and short-term temporal relationships according to claim 1 is characterized in that The specific process of step 3) is as follows: Step 3-1): Set the loss function to the cross entropy loss function L c , Among them, y j represents the label of the enhanced training data, represents the actual output of the video person re-identification model to be trained, N represents the number of samples, 1≤j≤N; Step 3-2): The number of forward propagation and back propagation of the enhanced training data in the video pedestrian re-identification model to be trained is set to 30, and the stochastic gradient descent optimizer is used to perform optimization. The initial learning rate is set to 0.

02. The stochastic gradient descent optimizer reduces the learning rate by 0.01 every 1000 times until the maximum number of iterations is reached. The iterative process is stopped to obtain the trained video pedestrian re-identification model.

Images