A Multi-Feature Fusion Gait Recognition Method Based on Spatiotemporal Dimension Screening

Abstract

The present invention discloses a multi-feature fusion gait recognition method based on spatio-temporal dimension screening, which includes the following steps: preprocessing the input image; extracting overall spatio-temporal features using a backbone network; finding local spatial features by adopting multi-scale local temporal relationships, introducing an attention scoring mechanism, and reorganizing to obtain potential spatial features; screening out significant temporal features using a spatial dimension filter; aggregating potential spatial features and significant temporal features respectively using a potential spatial feature aggregation layer and a significant temporal aggregation layer; using temporal pooling to map all feature sequences into a feature representing the entire gait cycle; using spatial pooling to map the potential spatial features into a more discriminative spatial domain; after the output of multi-feature fusion, using multiple independent fully connected layers to map it into a metric space, and the result is used for identity discrimination. The present invention obtains complementary spatio-temporal features, temporal features, and spatial features, making the expression of gait features richer and more complete.

Description

Technical Field

[0001] The present invention relates to the field of gait recognition, and specifically, to a multi-feature fusion gait recognition method based on spatio-temporal dimension screening. Background Art

[0002] As one of the new biometric technologies, gait recognition has played an important role in promoting the development of human identity recognition technology. The existing gait video feature extraction methods are mainly divided into two categories: one is to directly extract spatio-temporal features in gait by using ordinary three-dimensional convolution modules, and the other is to construct representative temporal and spatial features after obtaining high-semantic space features in the deep layer of the network. The former extracts general spatio-temporal features and does not have strong representativeness. Although the latter extracts representative features, it ignores the role of temporal and spatial features in the shallow layer of the network in gait recognition.

[0003] Gait recognition is a biometric technology that relies on the unique walking posture of humans. Compared with other biometric modes, such as face, fingerprint, and iris, it can not only capture the walking posture of people at a distance, but also does not require the active cooperation of the subjects during the recognition process. Therefore, gait recognition has valuable application prospects in the fields of identity recognition, medical diagnosis, etc. However, changes in external interferences such as clothing, carrying conditions, and viewing angles will cause huge changes in the external contour of people, usually resulting in greater changes within the subject than between subjects, posing great challenges to gait. Finding discriminative and representative gait features from the walking rules of people is an extremely important task. Currently, gait recognition methods can be roughly divided into two categories, one is the generation-based method, and the other is the discrimination-based method.

[0004] (1) The method based on generation aims to convert gait recognition under different conditions into gait recognition under the same conditions, reducing the differences of the same object under different walking conditions. It uses a gait template generated after preprocessing the gait sequence to establish a unique relationship for images from different perspectives. The literature [W. Kusakunniran, Q. Wu, H. Li and J. Zhang, "Multiple views gait recognition using View Transformation Model based on optimized Gait Energy Image," 2009 IEEE 12th International Conference on Computer Vision Workshops, ICCVW, 2009, pp. 1058 - 1064, doi: 10.1109 / ICCVW.2009.5457587.] optimized the gait energy map using Linear Discriminant Analysis (LDA) and constructed a view transformation model (VTM) to convert the perspective of the current GEI image into any other perspective; the literature [Yu, Shiqi and Chen, Haifeng and Garcia Reyes, Edel B. and Poh, Norman, "GaitGAN: Invariant Gait Feature Extraction Using Generative Adversarial Networks," Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, 2017, pp. 30 - 37.] used the generative adversarial network (GAN) to not only generate realistic side views of normal clothing for gait images from any perspective, but also retain individual recognition information at the same time.The literature [Y. He, J. Zhang, H. Shan and L. Wang, "Multi-Task GANs for View-Specific Feature Learning in Gait Recognition," in IEEE Transactions on Information Forensics and Security, vol. 14, no. 1, pp. 102-113, Jan. 2019, doi: 10.1109 / TIFS.2018.2844819.] proposed that Multi-Task GANs convert the perspectives of probe templates into the same perspectives as gallery templates, achieving view-specific feature representation.

[0005] (2) Discriminative-based methods mainly aim to directly learn condition-invariant features from gait data. In the early days, gait templates were commonly used as the input to the network. J. Han and B. Bhanu proposed generating a gait template by summing and averaging the gait cycle sequences, that is, the gait energy image [J. Han and B. Bhanu, "Individual recognition using gait energy image," IEEE Trans. Pattern Anal. Mach. Intell., vol. 28, no. 2, pp. 316–322, Feb. 2006.]. The literature [X. Hongye and H. Zhuoya, "Gait recognition based on gait energy image and linear discriminant analysis," 2015 IEEE International Conference on Signal Processing, Communications and Computing (ICSPCC), 2015, pp. 1-4, doi: 10.1109 / ICSPCC.2015.7338934.] uses principal component analysis (PCA) and linear discriminant analysis (LDA) to extract the main vectors of GEI features; the literature [N. Takemura, Y. Makihara, D. Muramatsu, T. Echigo, and Y. Yagi, “Multi-view large population gait dataset and its performance evaluation for cross-view gait recognition,” IPSJ Transactions on Computer Vision and Applications, vol. 10, no. 4, pp. 1–14, 2018.] learns multi-scale local binary representations for gait recognition.The literature [Z. Wu, Y. Huang, L. Wang, X. Wang and T. Tan, "A Comprehensive Study on Cross-View Gait Based Human Identification with Deep CNNs," in IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 39, no. 2, pp. 209-226, 1 Feb. 2017, doi: 10.1109 / TPAMI.2016.2545669.] first introduced the deep CNNs network into the field of gait recognition. The two GEI images were respectively input into the two-stream convolutional networks, and then the cross-loss function was used to train the network. By judging the similarity of the two images, it was decided whether they were the same individual. Considering that the gait template integrates time information and becomes a simple spatial expression form, and the overlap of partial contours also masks the discriminative spatial information. Therefore, later people directly used the contour sequence in the original video as the input. The literature [Chao H, He Y, Zhang J, et al. Gaitset: Regarding gait as a set for cross-view gait recognition [C] / / Proceedings of the AAAI conference on artificial intelligence. 2019, 33(01): 8126-8133.] regarded the gait sequence as an unordered set of frames, making the input of the network more flexible. In addition to the rich appearance information of gait, the available gait temporal relationship also plays an important role in gait recognition. The literature [Fan C, Peng Y, Cao C, et al. Gaitpart: Temporal part-based model for gait recognition [C] / / Proceedings of the IEEE / CVF conference on computer vision and pattern recognition. 2020: 14225-14233.] observed that different hierarchical body structures of people have significantly different movement patterns. Therefore, the micro-movement module was used to extract the most discriminative temporal features in a short range, further improving the recognition accuracy of gait.The literature [X. Huang et al., "Context-Sensitive Temporal Feature Learning for Gait Recognition," 2021 IEEE / CVF International Conference on Computer Vision (ICCV), 2021, pp. 12889-12898, doi: 10.1109 / ICCV48922.2021.01267.] constructs multi-scale temporal relationships in the deep layers of the network and uses this temporal relationship to screen the spatial features in the gait sequence. The above methods are all implemented using 2D cnns. Some studies have shown that, such as the literature [Tran D, Bourdev L, Fergus R, et al. Learning spatiotemporal features with 3d convolutional networks [C] / / Proceedings of the IEEE international conference on computer vision. 2015:4489-4497.], [S. Ji, W. Xu, M. Yang and K. Yu, "3D Convolutional Neural Networks for Human Action Recognition," in IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 35, no. 1, pp. 221-231, Jan. 2013, doi: 10.1109 / TPAMI.2012.59.], performing 3D convolution is a beneficial way to capture the spatial and temporal dimensions in videos. So, some people propose to combine 2D cnns with 3D cnns or directly use 3D cnns to extract the spatio-temporal features of gait. The literature [Lin B, Zhang S, Liu Y, et al. Multi-Scale Temporal Information Extractor For Gait Recognition [C] / / 2021 IEEE International Conference on Image Processing (ICIP). IEEE, 2021:2998-3002.] first uses 2Dcnns-3Dcnns to extract robust spatio-temporal features, and then uses dilated convolutions with different dilation rates to obtain multi-scale temporal features.The literature [Lin B, Zhang S, Yu X. Gait Recognition via Effective Global-Local Feature Representation and Local Temporal Aggregation[C] / / Proceedings of the IEEE / CVF International Conference on Computer Vision. 2021:14648-14656.] directly uses a 3DCNNs branch to extract global spatio-temporal features and another 3Dcnns branch to obtain local spatio-temporal features, achieving good recognition accuracy.

[0006] The above existing methods have the following problems:

[0007] (1) The problem that the extracted features have weak discriminability. General two-dimensional and three-dimensional convolutions can only improve the generalization performance of the features extracted by the network by stacking, and can only perform non-linear encoding on the spatio-temporal range of the kernel size area for the input features, without the function of screening features.

[0008] (2) Most of the existing research methods mine representative spatio-temporal features from the deep layers of the network, ignoring that there are also discriminative features in the shallow layers of the network. As Figure 1 shown, there are differences in the feature expressions output by different network layers. For example, the features of the deep network depict overall abstract information, while the features of the shallow network depict local detail information. Obviously, the two have different meanings and can both be used for gait information extraction. Summary of the Invention

[0009] In order to obtain more discriminative features, the present invention provides a multi-feature fusion network based on feature dimension screening, which can make full use of the gait information in the shallow layer of the network. Since the features extracted from the shallow layer of the network are also representative, fusing the features of different layers can also solve the problem of weak feature expression in the general network layer fusion method. Specifically, the present invention designs filters in the time dimension and space dimension to help two-dimensional or three-dimensional convolutions perform feature screening. The features after such screening are more discriminative. Secondly, in order to explore the discriminative temporal features and spatial features in the shallow layer of the network, time dimension filters and space dimension filters are respectively introduced behind multiple layer structures of the backbone network, and the obtained temporal features and spatial features are respectively aggregated using two layer fusion branch structures. Finally, complementary spatio-temporal features, temporal features, and spatial features are obtained, making the expression of gait features richer and more complete.

[0010] In view of this, the technical solution adopted by the present invention is a multi-feature fusion gait recognition method based on spatio-temporal dimension screening, including the following steps:

[0011] 1) Calculate the center point of each frame of the input gait image, then search for the edge critical points of the contour around the center point, determine a rectangular frame according to the four boundary points of up, down, left, and right, intercept the pedestrian contour within the rectangular frame, and scale the intercepted contour to obtain a gait contour map of a fixed size.

[0012] 2) Use the backbone network to extract the overall spatio-temporal features.

[0013] 3) Adopt multi-scale local temporal relationships to find the optimal local spatial features, introduce an attention scoring mechanism into the temporal features, and reorganize the obtained potential spatial features.

[0014] 4) Use a spatial dimension filter to screen out the significant temporal features in different horizontal spaces.

[0015] 5) Use a potential spatial feature aggregation layer and a significant temporal aggregation layer to aggregate potential spatial features and significant temporal features respectively.

[0016] 6) Use temporal pooling to map all significant temporal feature sequences of different lengths to a feature representing the entire gait cycle.

[0017] 7) Use spatial pooling to map the potential spatial features located at different horizontal positions to a more discriminative spatial domain.

[0018] 8) After the output of multi-feature fusion, use multiple independent fully connected layers to map it to a metric space, and the obtained features are used for identity discrimination.

[0019] 9) Input the labeled test data set into the gait recognition network described in steps 2) to 7), set an appropriate number of training iterations, and after the network is trained iteratively n times, a predictable gait recognition network is obtained.

[0020] 10) Preprocess the gait contour images in the registration data set and the gait contour images of the identity to be queried through step 1), and then input them into the network in step 8) respectively. Compare the similarity between the output features of the two to obtain the final identity recognition result.

[0021] The present invention adopts the above scheme and has the following characteristics and beneficial technical effects:

[0022] 1. Among the spatio-temporal features extracted from the shallow layer of the backbone network, use multi-scale temporal relationships to screen potential spatial features in the time dimension. The layer aggregation module of the spatial features contains a 1×3×3 convolution to adapt to the non-linear coding of spatial features, and the convolution kernel shares parameters with the convolution (3×3×3) of the backbone network.

[0023] 2. In the spatio-temporal features extracted from the shallow layer of the backbone network, global max pooling is used to screen significant temporal features in the spatial dimension. The layer aggregation module of the temporal features contains a 3×3×1 convolution to adapt to the non-linear encoding of the temporal features, and the convolution kernel shares parameters with the convolution (3×3×3) of the backbone network.

[0024] 3. The output features of the backbone network are spatially mapped using GeM, and the output features of the spatial aggregation layer are spatially mapped using a combination of global max pooling and global average pooling.

[0025] 4. The multi-feature fusion network based on spatio-temporal dimension screening can screen features from both the spatial dimension and the time dimension compared with the feature extraction network that can only rely on stacking three-dimensional convolutional layers to encode features. Since gait information is reflected by two aspects: gait silhouette (spatial domain) and gait sequence (time domain), the spatial dimension filter and the time dimension filter can search for useful information in gait from the spatial range and the time range respectively.

[0026] 5. The multi-feature fusion network based on spatio-temporal dimension screening is different from constructing discriminative features from the deep layer of the network. It attempts to explore the discriminative spatial features and temporal features in the shallow layer of the network. Experimental results show that mining gait information from the shallow layer of the network can effectively improve the recognition accuracy of gait. In addition, according to actual needs, the feature screening method in the layer fusion branch can also be flexibly adjusted. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 are the feature expressions of different network layers;

[0028] Figure 2 is the multi-feature fusion gait recognition network based on spatio-temporal dimension screening;

[0029] Figure 3 is the schematic diagram of the latent space feature aggregation layer. DETAILED DESCRIPTION OF THE INVENTION

[0030] The following will introduce in detail the multi-feature fusion gait recognition network based on spatio-temporal dimension screening as Figure 2 shown.

[0031] 1. Backbone Network

[0032] A feature extraction backbone network composed of multiple three-dimensional convolutions, one temporal pooling and one spatial pooling is used to extract the overall spatio-temporal feature F iHere, the present invention divides the backbone network into three layers. The first layer consists of a convolution and a temporal pooling with a temporal compression rate of r; the second layer consists of a convolution and a spatial pooling; the third layer consists of two convolutions. Then, the local mapping function can be denoted as where the superscript represents the size of the convolution kernel used in the mapping function. The overall mapping function can be denoted as F. Suppose the individual in the gait set input to the network is P j , individual P j 's gait representation is V = {v i |i = 1, 2,..., T}, T is the number of frames in the gait sequence, then the gait representations of all people are the set χ = {V j |j = 1, 2,..., N}, N represents the total number of individuals. For the input gait feature X ∈ R N×T×H×W , R represents that the dimensions N, T, H, W of the feature take values in the real number set R. H and W represent the height and width of the contour map. Then the feature output M bl output by the backbone network can be expressed as:

[0033]

[0034] 2. Layer aggregation of latent space features

[0035] The purpose of the spatial dimension filter is to find an optimal combined frame of local spatial expressions from a group of gait sequences. The present invention uses a local search method to generate a set of features. Generally, complete and robust spatial features are more conducive to constructing good temporal relationships, and compared with spatial features that perform well in a certain range of temporal relationships, spatial features that perform well in multiple different ranges of temporal relationships are more representative. The spatial contours of gaits may not be aligned in temporal modeling. To correct this problem, the present invention here uses multi-scale local temporal relationships to find the optimal local spatial features. First, compress the last dimension of the feature mapped by the function. Then convert the dimension of the compressed feature to obtain the output feature P T with a dimension of N × C'1 × T. Among them, C1 represents the number of channels of the output feature after the first layer of the network, and C'1 = C1 × H. This process is expressed by the formula as:

[0036]

[0037]

[0038]

[0039]

[0040] Among them, Trans represents the dimensionality transformation operation. P T represents the output feature after dimensionality transformation. MaxPool represents global max pooling (GMP) with a kernel size of (1, 1, W). Conv1d is a one-dimensional convolution with a kernel size of 3 for constructing short temporal relationships. As Figure 3 shown, one conv1d can be regarded as a temporal relationship builder with a window length of 3. The cascading of two conv1ds is equivalent to a temporal relationship builder with a window length of 5. T1 and T2 represent two local temporal features. MLP (multi-layer perceptron) is a module composed of two one-dimensional convolutions with a kernel size of 1, and the Sigmoid(·) function generates attention scores for each frame feature in the sequence. P T can be regarded as a set of features arranged in chronological order, P i represents a certain feature in the set, and i represents the sorting number of this feature in the set. The weighted sum of all frame features is calculated to obtain the temporal relationship of the entire gait sequence. repeat(·) is used to generate T identical features. Here, T3 represents a global temporal feature.

[0041] By introducing the attention scoring mechanism into the temporal features, the present invention emphasizes the importance of multi-scale temporal features at different spatial positions. The implementation of this formula is as follows:

[0042]

[0043] Logits = MLP(T c ) ∈ R N×H×T

[0044] Score = Sigmoid(Logits) ∈ R N×H×T

[0045] Among them, T c is a combination of multi-scale temporal features after element-wise addition and channel dimension connection. An MLP composed of two one-dimensional convolutional layers and an activation function is used to generate the weights of the temporal relationships at different horizontal positions. The perceptual features are represented by Logits. The Sigmoid(·) function calculates the importance score Score of the corresponding temporal features.

[0046] The significant temporal features are inseparable from the stable and robust local spatial features that construct them. Therefore, the potential local spatial features are reorganized to form a complete frame feature to represent the gait information of a set. This process can be expressed by the formula as:

[0047] T Indexs= Located(Max(Score, T)) ∈ R N×H

[0048]

[0049] Wherein, the Max function takes the maximum value on dimension T. Located represents the location operator. T Indexs is the index of the frame with the most significant temporal relationship formed by local spatial features. The Seek operation filters out the corresponding potential spatial features according to the frame index

[0050] In order to make the obtained temporal features contribute to assisting the gait recognition task, an identity recognition loss function is introduced to distribute the network's attention to the temporal relationships of different body parts. It can be calculated by the following formula:

[0051]

[0052]

[0053]

[0054] Wherein, a fully connected layer (FC) is used to compress the feature T c channels. The compressed feature is denoted as f. Score t and f t are the feature blocks obtained by splitting Score and f in the time dimension respectively, where t = 1, 2, 3,..., T. Prob is the predicted value of the gait feature belonging to the i-th individual. N is the number of individuals in the batch, and y i can be 1 or 0, which is the true label of the i-th individual. For the convenience of expressing the layer aggregation process, the numerical superscripts of all symbols represent the number of network layers. represents the calculated value of the recognition loss of the output features of the first network layer.

[0055] After the first and second network layers of the network output features, discriminative combined gait features are filtered out from the set of output gait features by a time dimension filter, and then aggregated using a network layer with the same structure as the backbone network. Slightly different from this, the size of the convolutional kernel is 1×3×3. This process can be described as follows:

[0056]

[0057] represents the potential spatial features output from the second network layer. M s represents the output features of the spatial aggregation layer.

[0058] 3. Layer Aggregation of Significant Temporal Features

[0059] The spatio-temporal features output by the backbone network layer can reflect the complete gait information of an individual. Different spatial regions on the features of each frame represent the temporal relationship of body parts during walking. However, in the case of changes in clothing and perspective, the complete gait features force the network to learn some local information that is expected to reduce attention or filter out. The appearance changes caused by clothing and angles rather than individual walking differences seriously affect the performance of gait recognition. Therefore, the present invention proposes a spatial search pattern to eliminate the interference of these non-ideal appearances. It can be known that the local temporal features beneficial to gait recognition are not interfered by other features within the global spatial range, and the significant local features are more representative than other general features. The purpose of the spatial dimension filter is to screen out the significant temporal features in different levels of space. First, the output features in the backbone network are horizontally divided into H parts (the number of horizontal divisions here is the same as the height of the contour map). Then, the global maximum pooling (GMP) is used to filter out the local features with weak expression ability, and the significant temporal features are obtained. It is expressed by the formula:

[0060]

[0061] Similar to the layer aggregation method of significant spatial features. Two network layer structures are used to integrate the significant temporal features output by the spatial dimension filter. The kernel size of the three-dimensional convolution in the network layer is 3×3×1. It can be expressed as follows:

[0062]

[0063] represents the significant temporal features output from the second network layer. M t represents the output features of the temporal aggregation layer.

[0064] 4. Feature Mapping

[0065] In real life, the input is usually a variable-length gait sequence. If the sequence length is not normalized, the output will be a non-fixed-length feature sequence, which will have an adverse effect on network training. Therefore, it is necessary to use time mapping to make the output length consistent. Here, time pooling is used to map all feature sequences of different lengths into a feature representing the entire gait cycle. If M b ∈R N×C×T×H / 2×W / 2 is the output of the backbone network, and the output of the significant temporal feature aggregation layer is M t ∈R N×C×T×H / 2×W / 2 . The time mapping can be expressed as:

[0066] M' b= Max(M b , T) ∈ R N×C×H / 2×W / 2

[0067] Y t = Max(M t , T) ∈ R N×c×H / 2

[0068] Among them, the Max function is used to take the maximum value of the feature M b on the dimension T. M' b represents the feature after the output feature of the backbone network is mapped in time. Y t represents the feature after the output feature of the significant time aggregation layer is mapped in time. When the feature from one spatial domain is mapped to another spatial domain, the spatial field of view that the network focuses on may change accordingly. Here, M' b and M t are constrained within a more discriminative spatial domain, which can be expressed by the formula as:

[0069] Y b = GeM(M' b ) ∈ R N×C×H / 2

[0070] Y s = GMP(M s ) + GAP(M s ) ∈ R N×C×H / 2

[0071] Among them, GeM is generalized mean pooling, and GMP and GAP are global maximum pooling and global average pooling respectively. The kernel size of these three pooling modules is (1, 1, W / 2). Y b and Y s are the features of M' b and M s after spatial mapping respectively. Multiple independent fully connected layers are used to map the features to the metric space for final feature discrimination.

[0072] 5. Loss function

[0073] In order to extract the most representative features from the network layer, a cross-entropy loss function is introduced after the attention module to supervise this process and ensure that the obtained spatial features play an active role in the final gait recognition. The trainable features finally output by the network adopt a combined loss training mode. The present invention calculates the loss value of each horizontal feature bar, and the definition of the cross-entropy loss function is as follows:

[0074]

[0075] In the formula, N is the number of samples in the batch, and P is the number of horizontal feature bars. y i,j$y_{ij}$ is the true label of the $j$-th horizontal feature bar of the $i$-th individual, with a value of 1 or 0. $p$ i,j is the predicted value of the $j$-th horizontal feature bar of the $i$-th individual.

[0076] Add a triplet loss function as an auxiliary constraint for network training. Given a triplet containing an anchor feature a positive sample feature and a negative sample feature , the triplet loss can be expressed as:

[0077]

[0078] where $D$ represents the Euclidean distance between sample features. $\alpha$ is the margin value of the triplet loss function.

[0079] The total loss function $L$ composed of the above three loss functions is expressed as follows:

[0080]

[0081] The total loss function consists of four parts, the supervision function after the first-layer attention module the supervision function after the second-layer attention module the combined training loss function $L$ of the last layer of the network ce and $L$ t . $\rho$ and $\gamma$ are the proportion coefficients of the two-layer supervision functions.

[0082] The following gait classification experiments were respectively implemented on two large public gait datasets, CASIA-B and OU-MVLP. The advanced methods for comparison include GaitSet, GaitPart, MT3D, and GLN.

[0083] The CASIA-B dataset contains 124 objects. Each object has 10 clothing sequences, 6 sequences are samples of normal walking, 2 sequences are for walking with a backpack, and the last 2 sequences are for walking while wearing a coat. Each clothing sequence contains 11 gait sequences from different perspectives (0 to 180 degrees, at 18-degree intervals), for a total of 13,640 gait sequences.

[0084] In the CASIA-B dataset, the image size is normalized to 64×44. The data batch size $(p, k)$ for network training is set to $(8, 8)$, where $p$ represents the number of objects and $k$ represents the number of samples of each object in the training batch. The Adam optimizer is used to train the entire network. The initial learning rate of the optimizer is 1e-4. When the number of training times $n$ reaches 70k, the learning rate is adjusted to 1e-5. The weight decay is set to 5e-4. The total number of training times is set to 80k.

[0085] Table 1 provides the comparison results of the method of the present invention and the four listed advanced methods under three walking conditions (including normal clothing, carrying a backpack, and wearing a coat) in the CASIA-B dataset. Among them, the statistics of the average Rank-1 recognition accuracy under each perspective do not include the recognition accuracy under the same perspective.

[0086] Table 1 Rank-1 recognition accuracy under three walking conditions on the CASIA-B dataset (excluding the same perspective)

[0087]

[0088] As can be seen from Table 1, the method of the present invention has achieved the best recognition effect under the three walking conditions of NM, BG, and CL. Among them, under the NM and BG conditions, the recognition accuracies reached 97.1% and 94.3% respectively. In addition, under the CL condition, the recognition accuracy of the method of the present invention has been improved by 6% compared with the GLN recognition method.

[0089] In order to verify the generalization performance of the method of the present invention, experiments were evaluated on the OU-MVLP dataset. It contains 10,307 objects. Each object contains two sequences (00 and 01). Both of these sequences contain gait sequences under 14 perspectives.

[0090] In the OU-MVLP dataset, the image size is normalized to 64×44. The data batch size (p, k) of network training is set to (32, 8). The Adam optimizer is also used to train the entire network. The initial learning rate of the optimizer is 1e-4. When the number of training times n reaches 150k, the learning rate is adjusted to 1e-5. When the number of training times reaches 200k, the learning rate is set to 5e-6. The weight decay is initially set to 0, and when the number of training times reaches 200k, it is set to 5e-4.

[0091] Table 2 Rank-1 recognition accuracy under 14 probe perspectives on the OU-MVLP dataset (excluding the same perspective)

[0092]

[0093]

[0094] As can be seen from Table 2, the recognition accuracy of the method of the present invention under 14 perspectives is better than that of GEINet and GaitSet. Among them, the average recognition rate under all perspectives is 0.5% higher than that of the GaitPart recognition method.

Claims

1. A multi-feature fusion gait recognition method based on spatio-temporal dimension screening, characterized in that It includes the following steps: 1) Calculate the center point of each frame of the input gait image, then search for the edge critical points of the contour around the center point, determine a rectangular frame according to the four boundary points of up, down, left, and right, intercept the pedestrian contour within the rectangular frame, and scale the intercepted contour to obtain a gait contour map of a fixed size; 2) Extract the overall spatio-temporal features using the backbone network. The backbone network consists of three layers. The first layer consists of a convolution and a temporal pooling with a temporal compression rate of r. The second layer consists of a convolution and a spatial pooling. The third layer consists of two convolutions. Then the local mapping function is denoted as where the superscript represents the size of the convolution kernel used in the mapping function; 3) Use multi-scale local temporal relationships to find the optimal local spatial features, introduce an attention scoring mechanism into the temporal features, and reorganize to obtain potential spatial features; 4) Use a spatial dimension filter to screen out significant temporal features in different horizontal spaces; 5) Use a potential spatial feature aggregation layer and a significant temporal aggregation layer to aggregate potential spatial features and significant temporal features respectively; 6) Use temporal pooling to map all significant temporal feature sequences of different lengths to a feature representing the entire gait cycle; 7) Use spatial pooling to map potential spatial features located at different horizontal positions to a more discriminative spatial domain; 8) After the output of multi-feature fusion, use multiple independent fully connected layers to map it to a metric space, and the obtained features are used for identity discrimination.

2. The multi-feature fusion gait recognition method based on spatio-temporal dimension screening according to claim 1, wherein: Step 3) includes first compressing the last dimension of the features after function mapping , then performing dimensionality conversion on the compressed features to obtain the output feature P T has a dimension of N×C′1×T, where H and W represent the height and width of the contour map, T is the number of frames of the gait sequence, N represents the total number of individuals, C1 represents the number of channels of the output features after the first layer of the network, and C′1 = C1×H. Here, this process is expressed by the formula: Among them, Trans represents a dimension transformation operation, MaxPool represents global max pooling with a kernel size of (1, 1, W), Conv1d is a one-dimensional convolution with a kernel size of 3, used to construct short temporal relationships, T1 and T2 represent two local temporal features, MLP is a module composed of two one-dimensional convolutions with a kernel size of 1, and the Sigmoid(·) function generates an attention score for each frame feature in the sequence, repeat(·) is used to generate the same features for T frames, and T3 represents a global temporal feature.

3. The multi-feature fusion gait recognition method based on spatio-temporal dimension screening according to claim 2, characterized in that: The attention scoring mechanism is Logits = MLP(T c ) ∈ R N×H×T Score=Sigmoid(Logits)∈R N×H×T Among them, T c is a combination of multi-scale temporal features after element-wise addition and channel dimension concatenation. Logits represent the perceived features, and the Sigmoid(·) function calculates the importance score Score of the corresponding temporal features.

4. The multi-feature fusion gait recognition method based on spatio-temporal dimension screening according to claim 3, characterized in that: The reorganization in step 3) is expressed by the formula: T Indexs = Located(Max(Score, T)) ∈ R N×H Among them, the Max function takes the maximum value on dimension T, Located represents the positioning operator, and T Index is the index of the frame with the most significant temporal relationship formed by local spatial features, and the Seek operation filters out the corresponding spatial features according to the frame index 5. The multi-feature fusion gait recognition method based on spatio-temporal dimension screening according to claim 4, characterized in that: An identity recognition loss function is introduced, which distributes the network's attention to the temporal relationships of different body parts. The calculation formula is: Among them, a fully connected layer FC is used to compress the features T c channels, f represents the compressed features, Score t and f t are the feature blocks obtained by splitting Score and f in the time dimension respectively, where t = 1, 2, 3,..., T, Prob is the predicted value of the gait feature belonging to the i-th individual, N is the number of individuals in the batch, and y i is 1 or 0, indicating the true label of the i-th individual, represents the calculated value of the recognition loss of the output features of the first network layer.

6. The multi-feature fusion gait recognition method based on spatio-temporal dimension screening according to claim 5, wherein: Step 4) Screening significant temporal features in different levels of space, including first dividing the output feature levels in the backbone network into H parts; then, using global maximum pooling (GMP) to filter out local features with weak expression ability; finally, the significant temporal features are represented by the formula: are expressed by the formula:

7. The multi-feature fusion gait recognition method based on spatio-temporal dimension screening according to claim 6, wherein: In step 5), the significant temporal feature aggregation layer is used to integrate the significant temporal features output by the spatial dimension filter. The kernel size of the three-dimensional convolution in the network layer is 3×3×1, and it is expressed as follows: Represents the significant temporal features output from the second network layer, Represents the significant temporal features output from the first network layer, M t Represents the output features of the temporal aggregation layer; The potential spatial feature aggregation layer is used to integrate the potential spatial features output by the temporal dimension filter. The kernel size of the three-dimensional convolution in the network layer is 1×3×3, and it is expressed as follows: M s represents the output features of the spatial feature aggregation layer, represents the latent spatial features output by the first network layer, represents the latent spatial features output from the second network layer.

8. The multi-feature fusion gait recognition method based on spatio-temporal dimension screening according to claim 7, wherein: Step 6) includes that if M b ∈R N×C×T×H / 2×W / 2 is the output of the backbone network, and the output of the significant temporal feature aggregation layer is M t ∈R N×C×T×H / 2×W / 2 , the time mapping is expressed as: M' b = Max(M b , T) ∈ R N×C×H / 2×W / 2 Y t = Max(M t , T) ∈ R N×C×H / 2 Among them, the Max function is used to separately obtain the maximum values of the features M b and M t on the dimension T to obtain the time-mapped feature M' b and Y t , where M' b represents the feature after time mapping of the output feature of the backbone network, and Y t represents the feature after time mapping of the output feature of the significant time aggregation layer.

9. The multi-feature fusion gait recognition method based on spatio-temporal dimension screening according to claim 8, characterized in that: Step 7) specifically includes: In the spatial domain, the formula is expressed as: Y b = GeM(M' b ) ∈ R N×C×H / 2 Y s = GMP(M s ) + GAP(M s ) ∈ R N×C×H / 2 Among them, GeM is Generalized Mean Pooling, GMP and GAP are Global Max Pooling and Global Average Pooling respectively, Y b and Y s are the features of M′ b and M s after spatial mapping respectively.

10. A multi-feature fusion gait recognition method based on spatio-temporal dimension screening according to any one of claims 1-9, characterized in that: The total loss function L is expressed as follows: The total loss function consists of four parts, the supervision function after the first-layer attention module The supervision function after the second-layer attention module L ce and L t represent the cross-entropy loss and the triplet loss respectively, and ρ and γ are the proportion coefficients of the two-layer supervision functions.

Images