Video Person Re-identification Methods

Abstract

The present invention belongs to the technical field of video pedestrian re-identification, and provides a method for video pedestrian re-identification. The present invention uses a Transformer structure to extract multi-view features to solve the task of video pedestrian re-identification. The purpose of video pedestrian re-identification is to achieve cross-camera matching of pedestrian sequence images. The present invention proposes to use a trigeminal network to respectively obtain the spatial perspective, temporal perspective and spatiotemporal perspective of pedestrian videos, so as to obtain multi-view observations in different feature domains. Transformer is used in a single perspective feature domain to mine single-view feature relationships and optimize perspective features. Transformer is used between cross-perspective feature domains to explore multi-perspective feature relationships, and multi-perspective information is integrated to obtain a more complete feature representation. The present invention can extract more robust and discriminative feature representations of pedestrian videos, and can achieve higher-precision pedestrian video instance matching.

Description

Technical Field

[0001] The present invention belongs to the technical field of video pedestrian re-identification. For a given video of a pedestrian to be identified, other videos of the same pedestrian matching the pedestrian are found in a video collection. The present invention involves relevant knowledge of image and video processing, and particularly involves a pedestrian feature learning method. Background Art

[0002] The pedestrian re-identification task is a human-centered classification task. Its purpose is to find the correct match for a given pedestrian image or video in other image sets or video sets taken across cameras. Since pedestrian re-identification algorithms play an important role in practical applications such as suspect search and scene monitoring, more and more researchers are engaged in the study of pedestrian re-identification. In recent years, with the unprecedented development of deep learning and the effective use of massive pedestrian data, the field of pedestrian re-identification has made rapid and sufficient progress. Despite this, the re-identification task still faces many challenges, such as the frequent appearance of complex background interference in pedestrian images, as well as changes in information such as perspective, posture, and lighting. These challenges have brought great difficulties to the representation learning of pedestrian features and are the main factors affecting the recognition accuracy. The pedestrian re-identification task can be divided into two different subtasks according to the type of samples: image pedestrian re-identification and video pedestrian re-identification. Compared with image pedestrian re-identification, which uses a single pedestrian image for matching, video pedestrian re-identification uses a multi-frame image sequence for matching. Compared with a single static image, video data brings about temporal changes in pedestrians, such as gait and multi-view information, which helps to better match and identify pedestrians. However, these temporal changes also introduce more noise and background changes. Based on the above challenges, how to make full use of temporal and spatial information is the key to solving the video pedestrian re-identification task.

[0003] In recent years, researchers have mainly considered the spatial feature extraction and temporal feature learning for the video person re-identification task. In terms of spatial feature extraction, given the high efficiency of convolutional neural networks (CNN) in the field of image processing, it has been widely used in previous algorithms to extract spatial features of pedestrians. In terms of temporal feature learning, recurrent networks are designed to process serialized features. Classic recurrent networks include gated recurrent networks (GRU) and long short-term memory networks (LSTM). Among them, LSTM networks use the long-term and short-term dependencies of serialized features in time to learn temporal feature representations. In general, the input original video is successively extracted through spatial features and temporal feature learning to obtain the final video representation, so as to further perform feature matching between samples. Compared with the traditional method of manually designing features, this classic two-stage video re-identification processing flow has excellent performance. At the same time, this two-stage video person re-identification algorithm still has a lot of room for improvement. On the one hand, some algorithms focus on how to better extract more discriminative spatial features. For example, Li et al. published 'Diversity regularized spatiotemporal attention for video based person re-identification' in 2018 CVPR and designed multiple spatial attention modules to capture different spatial local features. Fu et al. published 'STA: Spatial-temporal attention for large-scale video based person re-identification' in 2019 AAAI and learned an attention matrix without parameters to adaptively weight the attention of spatially segmented features. Zhao et al. published 'Attribute-driven feature disentangling and temporal aggregation for video person re-identification' in 2019 and used attribute prediction networks to extract features corresponding to different attributes in space. On the other hand, some algorithms try to strengthen the representation learning of temporal features. For example, Zhang et al. published 'Multi-shot pedestrian re-identification via sequential decision making' in 2019 CVPR and designed a sequence reinforcement model to classify, predict and terminate decisions frame by frame.Liu et al. published the paper 'Spatial and temporal mutual promotion for video-based person re-identification' at AAAI in 2019. They used an optimized recurrent unit in the temporal processing stage to perform sequential optimization learning of spatial features. Their experimental results show that by optimizing the extraction of temporal and spatial features, the current algorithms can effectively improve the matching accuracy of video person re-identification.

[0004] Despite this, these algorithms still have the following problems: (1) Current video feature extraction methods often only focus on feature optimization learning in a single perspective, such as spatial attention mining and temporal length dependency capture, but these lack representation learning of different types of features, such as multi-perspective observations from different feature spaces, such as space, time, and spatiotemporal. (2) Current video feature extraction methods lack effective means to integrate temporal and spatial features. For example, direct feature addition and feature concatenation can achieve multi-feature fusion to a certain extent, but this method is too simple and cannot fully utilize the advantages of multi-perspective features. Therefore, we propose to use the Transformer network architecture under the guidance of spatiotemporal features to further integrate temporal features and spatial features, so as to obtain a comprehensive video feature representation. Summary of the invention

[0005] The technical problem to be solved by the present invention is: for a given pedestrian video, the representation features of the pedestrian video are extracted without using additional attribute information for subsequent instance matching. Not only that, the invention must also be able to handle the representation misalignment that occurs in pedestrian video sequences, such as the background changes of pedestrians in different locations in the video sequence, and the lighting changes at different times. The pedestrian representation misalignment caused by these spatiotemporal changes is a major challenge for the video pedestrian re-identification task. In addition, the invention also proposes a novel multi-view feature fusion module, which effectively realizes the fusion of multiple representation features after mining the temporal perspective features, spatial perspective features and spatiotemporal perspective features of the pedestrian video.

[0006] The technical solution of the present invention:

[0007] A method for video pedestrian re-identification, the steps are as follows:

[0008] Step 1: Data sampling;

[0009] The given pedestrian video is sampled to obtain a sequence of images. Each pedestrian video is divided into 8 segments, and a picture is extracted from each segment. Thus, a video segment is constructed into a sequence of images.

[0010] Step 2: Build the batch;

[0011] Batches are constructed according to pedestrian numbers. Each batch contains 8 different pedestrians. Each pedestrian has 4 different sequence image sets, for a total of 32 sequence image sets.

[0012] Step 3: Normalization processing;

[0013] Since the specifications and sizes of images in different sequence image sets may be different, the data of each image in each sequence set in the batch is normalized;

[0014] Step 4: Data augmentation;

[0015] Use data augmentation methods to change the sequence images in a batch according to various rules, including random cropping of images, random horizontal flipping of images, and random occlusion of images;

[0016] Step 5: Build the network;

[0017] A Transformer-based three-view neural network is constructed, including a three-view feature extractor, a self-attention pooling module, a Transformer-based view feature optimizer, and a Transformer-based multi-view feature fusion module. The three-view feature extractor uses ResNet-50 as the skeleton, and then uses the self-attention pooling module to map the original spatiotemporal features to spatial and temporal perspectives. Next, the view feature optimizer is used to mine the internal view dependency and optimize the three view features. Finally, the multi-view feature fusion module fuses the temporal and spatial features based on the spatiotemporal features to obtain the final pedestrian video encoding features.

[0018] (1) Trident View Feature Extractor

[0019] ResNet-50 is used to extract the spatial features of each frame in the video sequence; the ResNet-50 neural network is divided into 5 network modules, each of which contains several residual neural networks; the first 4 network modules are used as shared layers of the time, space, and spatiotemporal feature extraction networks; the 5th network module, which is not shared, is used to separate the features of the three perspectives; thus, three different video feature representations are extracted, representing the time, space, and spatiotemporal perspectives respectively;

[0020] (2) Self-Attention Pooling Module

[0021] A self-attention pooling module is introduced to transform the original spatiotemporal features into the spatial and temporal perspective domains;

[0022] For transforming spatiotemporal features to the spatial view domain, given the original three-dimensional spatiotemporal features X of a video s ∈R T ×HW×C, where T represents the number of image frames in a sequence, H, W represent the height and width of the feature respectively, and C represents the number of feature channels; first, a linear mapping layer is used to map each spatial feature X in the spatiotemporal feature. s i ∈R T×C , i∈[1,H×W] is encoded to obtain F i ∈R T×C ,

[0023]

[0024] Among them, W represents the network parameters of the linear transformation; then, the self-attention matrix M is generated by matrix calculation i ∈R T ×T ,

[0025] M i =F i F i T (2)

[0026] here,(·) T represents the transpose operation; then, M i Sum in the time dimension and use the softmax activation function to get the attention value of each spatial position feature at different time sequences.

[0027]

[0028]

[0029]

[0030] After that, the spatial feature is multiplied by its corresponding temporal attention value to obtain the self-attention pooled feature. Therefore, the spatiotemporal feature is transformed into the spatial perspective domain through self-attention feature pooling to obtain the spatial feature F s = {g 1 , …, g i , …, g H×W}, i∈[1,H×W]; Similarly, the same mechanism is used to transform the spatiotemporal features into the temporal perspective domain to obtain the temporal features F t = {g 1 , …, g i , …, g T}, i∈[1,T];

[0031] (3) Transformer-based view feature optimizer

[0032] Use Transformer to mine spatial dependencies, temporal dependencies, and spatiotemporal dependencies in each perspective domain;

[0033] In the temporal view domain, the temporal feature representation F extracted from the trifurcated view feature extractor is t ∈R T×C The Transformer neural network is passed into the Transformer neural network; the Transformer structure includes a multi-head attention layer, a forward encoding network, layer normalization, and a residual connection; first, a learnable position encoding feature and a spatiotemporal feature are added and passed into the multi-head attention layer; in each head of the multi-head attention layer, the temporal feature is generated through three different linear transformation layers to generate three codes Q, K, and V, where Q, K, V∈R T×d , d = C / N h ; N h is the number of heads in the multi-head attention layer, which is set to 12; then, the self-attention matrix in each head is calculated and multiplied by V to obtain the single-head feature,

[0034]

[0035] Feature A in multiple heads 1 , …, A Nh They are concatenated together as the output of the multi-head self-attention layer; then, the forward network, normalization layer and residual connection are used for further encoding to obtain the optimized temporal feature representation;

[0036] F t =LayerNorm(F t +MultiHead(F t )) (7)

[0037] F t =W 2 σ(W 1 F t ) (8)

[0038] Where LayerNorm represents the normalization layer; MultiHead represents the multi-head attention layer mentioned above; W 1 , W 2 Represents two fully connected neural networks, and σ represents the ReLU activation function. Similarly, spatial features and spatiotemporal features also use such a Transformer structure to mine dependencies within the visual domain, and optimize feature learning through the mined dependencies.

[0039] (4) Transformer-based multi-view feature fusion module

[0040] Based on the spatiotemporal features, the temporal features and spatial features are fused to obtain the final pedestrian video representation; the multi-view feature fusion module is based on the Transformer structure and adds a cross-view attention layer to achieve information fusion of multi-view features;

[0041] The multi-view feature fusion module includes a multi-head cross-view attention layer, a forward network, a normalization layer, and a residual connection; for the obtained temporal feature F t , spatial feature F s and the spatiotemporal characteristics F st First, 6 linear transformation layers are used to generate 6 features Q st s , Q st t , K s , K t , V s , V t , where Q st s , Q st t ∈R THW×d , K s , V s ∈R HW×d , K t , V t ∈R T×d , d = C / N h ; N h is the number of heads in the multi-head attention layer; and cross-view feature fusion is expressed as:

[0042]

[0043]

[0044]

[0045] F st =W 4 σ(W 3 F st ) (12)

[0046] Here W 3 , W 4 represents two fully connected neural networks, σ represents the ReLU activation function; finally, based on the spatiotemporal features, the temporal features and spatial features are fused to obtain the final pedestrian video representation F st ; Afterwards, a simple spatiotemporal pooling operation is used to obtain the final feature vector, which is then used to achieve subsequent pedestrian instance matching;

[0047] Step 6: Network training;

[0048] The network training includes forward feature feedback and backward gradient propagation. The batched pedestrian sequence set is passed into the constructed trifurcated view-based Transformer in batches to extract the pedestrian video encoding features. Then, the online instance matching loss at the image level and video level is calculated according to formulas (13) and (14). After that, the parameters of the neural network are updated batch by batch by using the stochastic gradient descent method.

[0049]

[0050]

[0051] Where W i Represents the parameters of the classifier;

[0052] Step 7: Instance matching;

[0053] The parameters of the trained Transformer neural network based on the trifurcated view are fixed; given a pedestrian video to be identified and a search set consisting of multiple pedestrian videos; they are respectively passed into the neural network to extract the encoding features of different pedestrian videos; the cosine similarity between the features to be identified and the features of each pedestrian in the search set is calculated.

[0054]

[0055] And sort by cosine similarity, find the sample with the greatest feature similarity between the set and the sample to be identified, which is the target matching sample of the pedestrian to be identified.

[0056] The effects and benefits of the present invention are as follows: Compared with previous methods, our transformer-based three-pronged perspective person re-identification algorithm captures multi-perspective representations of pedestrian videos from three perspectives: temporal, spatial, and spatiotemporal, and fuses them through a fusion module to obtain multi-perspective representations of pedestrians. These multi-perspective representations of pedestrians can improve the matching accuracy of pedestrian videos. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] Figure 1 It is a flowchart of the algorithm.

[0058] Figure 2 It is a schematic diagram of temporal self-attention pooling.

[0059] Figure 3 It is a schematic diagram of spatial self-attention pooling.

[0060] Figure 4 This is a schematic diagram of the Transformer-based view feature optimizer.

[0061] Figure 5 This is a schematic diagram of multi-view feature fusion based on Transformer. DETAILED DESCRIPTION

[0062] The specific implementation of the present invention is further described below in conjunction with the accompanying drawings and technical solutions.

[0063] Step 1: Given a video clip of a pedestrian, sample the frames to obtain a set of pedestrian video images, divide the image set into 8 equal segments, extract a picture from each segment, and thus construct a sequence set containing 8 frames of images as the input of the pedestrian video re-identification algorithm;

[0064] Step 2: We normalize the 8 pedestrian images obtained in step 1;

[0065] Step 3: Pass the normalized image sequence into the trained Transformer-based three-view neural network to extract the feature representation of the video. The specific steps are as follows:

[0066] (I) The image sequence is passed through the constructed three-pronged view feature extractor to extract three different features. Then, two of the features are passed to Figure 2 The temporal self-attention pooling and Figure 3 The spatial self-attention pooling shown in the figure obtains spatial perspective features and temporal perspective features, realizing the separation of three-perspective features.

[0067] (ii) The three view features are respectively passed into three Transformer-based view feature optimizers, such as Figure 4 As shown in Figure 2, the feature dependencies within the mining perspective are used to optimize the features.

[0068] (III) The optimized three-view features are simultaneously transmitted to Figure 5 In the multi-view feature fusion module shown, three view features are fused to obtain the final video feature representation.

[0069] Step 4: Calculate the cosine similarity between the pedestrian video feature and other pedestrian video features to obtain a matching score, and sort them accordingly. The pedestrian video with the highest similarity is identified as the target matching pedestrian video.

Claims

1. A method for video pedestrian re-identification, characterized in that: Here are the steps: Step 1: Data sampling; The given pedestrian video is sampled to obtain a sequence of images. Each pedestrian video is divided into 8 segments, and a picture is extracted from each segment. Thus, a video segment is constructed into a sequence of images. Step 2: Build the batch; Batches are constructed according to pedestrian numbers. Each batch contains 8 different pedestrians. Each pedestrian has 4 different sequence image sets, for a total of 32 sequence image sets. Step 3: Normalization processing; Since the specifications and sizes of images in different sequence image sets may be different, the data of each image in each sequence set in the batch is normalized; Step 4: Data augmentation; Use data augmentation methods to change the sequence images in a batch according to various rules, including random cropping of images, random horizontal flipping of images, and random occlusion of images; Step 5: Build the network; A Transformer-based three-view neural network is constructed, including a three-view feature extractor, a self-attention pooling module, a Transformer-based view feature optimizer, and a Transformer-based multi-view feature fusion module. The three-view feature extractor uses ResNet-50 as the skeleton, and then uses the self-attention pooling module to map the original spatiotemporal features to spatial and temporal perspectives. Next, the view feature optimizer is used to mine the internal view dependency and optimize the three view features. Finally, the multi-view feature fusion module fuses the temporal and spatial features based on the spatiotemporal features to obtain the final pedestrian video encoding features. (1) Trident View Feature Extractor ResNet-50 is used to extract the spatial features of each frame in the video sequence; the ResNet-50 neural network is divided into 5 network modules, each of which contains several residual neural networks; the first 4 network modules are used as shared layers of the time, space, and spatiotemporal feature extraction networks; the 5th network module, which is not shared, is used to separate the features of the three perspectives; thus, three different video feature representations are extracted, representing the time, space, and spatiotemporal perspectives respectively; (2) Self-Attention Pooling Module A self-attention pooling module is introduced to transform the original spatiotemporal features into the spatial and temporal perspective domains; For transforming spatiotemporal features to the spatial view domain, given the original three-dimensional spatiotemporal features X of a video s ∈R T×HW×C , where T represents the number of image frames in a sequence, H, W represent the height and width of the feature respectively, and C represents the number of feature channels; first, a linear mapping layer is used to map each spatial feature X in the spatiotemporal feature. s i ∈R T×C ,i∈[1,H×W] is encoded to obtain F i ∈R T×C , Among them, W represents the network parameters of the linear transformation; then, the self-attention matrix M is generated by matrix calculation i ∈R T×T , M i =F i F i T (2) here,(·) T represents the transpose operation; then, M i Sum in the time dimension and use the softmax activation function to get the attention value of each spatial position feature at different time sequences. After that, the spatial feature is multiplied by its corresponding temporal attention value to obtain the self-attention pooled feature. Therefore, the spatiotemporal feature is transformed into the spatial perspective domain through self-attention feature pooling to obtain the spatial feature F s ={g1,…,g i ,…,g H×W }, i∈[1,H×W]; Similarly, the same mechanism is used to transform the spatiotemporal features into the temporal perspective domain to obtain the temporal features F t ={g1,…,g i ,…,g T }, i∈[1,T]; (3) Transformer-based view feature optimizer Use Transformer to mine spatial dependencies, temporal dependencies, and spatiotemporal dependencies in each perspective domain; In the temporal view domain, the temporal feature representation F extracted from the trifurcated view feature extractor is t ∈R T×C The Transformer neural network is passed into the Transformer neural network; the Transformer structure includes a multi-head attention layer, a forward encoding network, layer normalization, and a residual connection; first, a learnable position encoding feature and a spatiotemporal feature are added and passed into the multi-head attention layer; in each head of the multi-head attention layer, the temporal feature is passed through three different linear transformation layers to generate three codes Q, K, and V, where Q, K, V∈R T×d , d = C / N h ; N h is the number of heads in the multi-head attention layer, which is set to 12; then, the self-attention matrix in each head is calculated and multiplied by V to obtain the single-head feature, Features A1,…,A in multiple heads Nh They are concatenated together as the output of the multi-head self-attention layer; then, the forward network, normalization layer and residual connection are used for further encoding to obtain the optimized temporal feature representation; F t =LayerNorm(F t +MultiHead(F t )) (7) F t =W2σ(W1F t ) (8) In the formula, LayerNorm represents the normalization layer; MultiHead represents the multi-head attention layer mentioned above; W1, W2 represent two fully connected neural networks, and σ represents the ReLU activation function; similarly, spatial features and spatiotemporal features also use such a Transformer structure to realize dependency mining in the visual domain, and realize feature optimization learning through the mined dependency relationships; (4) Transformer-based multi-view feature fusion module Based on the spatiotemporal features, the temporal features and spatial features are fused to obtain the final pedestrian video representation; the multi-view feature fusion module is based on the Transformer structure and adds a cross-view attention layer to achieve information fusion of multi-view features; The multi-view feature fusion module includes a multi-head cross-view attention layer, a forward network, a normalization layer, and a residual connection; for the obtained temporal feature F t , spatial feature F s and the spatiotemporal characteristics F st First, 6 linear transformation layers are used to generate 6 features Q st s ,Q st t ,K s ,K t ,V s ,V t , where Q st s , Q st t ∈R THW×d , K s ,V s ∈R HW×d , K t ,V t ∈R T×d , d = C / N h ; N h is the number of heads in the multi-head attention layer; and cross-view feature fusion is expressed as: F st =W4σ(W3F st ) (12) Here W3 and W4 represent two fully connected neural networks, and σ represents the ReLU activation function. Finally, based on the spatiotemporal features, the temporal features and spatial features are fused to obtain the final pedestrian video representation F. st ; Afterwards, a simple spatiotemporal pooling operation is used to obtain the final feature vector, which is then used to achieve subsequent pedestrian instance matching; Step 6: Network training; The network training includes forward feature feedback and backward gradient propagation. The batched pedestrian sequence set is passed into the constructed trifurcated view-based Transformer in batches to extract the pedestrian video encoding features. Then, the online instance matching loss at the image level and video level is calculated according to formulas (13) and (14). After that, the parameters of the neural network are updated batch by batch by using the stochastic gradient descent method. Where W i Represents the parameters of the classifier; Step 7: Instance matching; The parameters of the trained Transformer neural network based on the trifurcated view are fixed; given a pedestrian video to be identified and a search set consisting of multiple pedestrian videos; they are respectively passed into the neural network to extract the encoding features of different pedestrian videos; the cosine similarity between the features to be identified and the features of each pedestrian in the search set is calculated. And sort by cosine similarity, find the sample with the greatest feature similarity between the set and the sample to be identified, which is the target matching sample of the pedestrian to be identified.

Images