Gait recognition method based on multi-granularity feature learning based on keyframes

Abstract

The present invention discloses a multi-granularity feature learning gait recognition method based on key frames. The method extracts key frames from gait profiles in a gait sequence, selects gait profiles that have a great influence on the recognition results, and composes them into a key frame sequence. Gait features are extracted from the key frame sequence and the original gait sequence respectively, and finally fused into recognition features, so that the extracted recognition features are more discriminative; when extracting features, attention is paid to temporal information, and global features are fused with local features. The invention uses cross entropy loss and triplet loss to optimize recognition features to improve recognition accuracy. The present invention solves the problems of past sequence-based gait recognition technologies in terms of undifferentiated feature extraction, insufficient local feature extraction, and insufficient temporal feature extraction during recognition. The present invention is suitable for pedestrian gait recognition tasks, with high recognition accuracy and good algorithm robustness. The present invention has a wide range of applications in the field of gait recognition.

Description

Technical Field

[0001] The present invention belongs to the field of biometric recognition and aims to perform identity recognition based on the changes in posture of people during normal walking. Specifically, it is a gait recognition method based on multi-granularity feature learning based on key frames. Background Art

[0002] Gait recognition is an emerging human biometric task that aims to identify people by detecting changes in their posture while walking. Compared to other biometric recognition technologies such as facial, fingerprint, and iris recognition, gait recognition offers significant advantages: it can be detected from a distance, requires no contact, and is difficult to conceal or disguise. With the widespread use of video surveillance, gait recognition has even greater potential and can play a vital role in various fields.

[0003] Currently, most existing gait recognition technologies are based on deep learning. There are two main strategies for deep learning-based gait recognition: template-based gait recognition and sequence-based gait recognition. Template-based gait recognition compresses the gait profile image of a pedestrian into a single image or a gait template, and then performs recognition on the resulting image or gait template. This recognition method is fast and requires few network parameters, but the compression process causes some information loss, resulting in low recognition accuracy.

[0004] Therefore, sequence-based gait recognition methods are often used in gait recognition. Sequence-based gait recognition methods treat multiple existing gait profile images of the same pedestrian as a gait sequence, inputting these into a recognition model to extract features and perform recognition. Although existing sequence-based gait recognition methods have achieved significant progress in both accuracy and speed, these methods still need improvement in terms of undifferentiated feature extraction, local feature extraction, and temporal feature extraction during recognition. Therefore, a new gait recognition method is currently needed to address these issues. Summary of the Invention

[0005] The problem to be solved by the present invention is: in sequence-based gait recognition technology, although existing gait recognition methods have made great progress in both accuracy and running speed, these methods still need to be improved in terms of non-differentiated feature extraction, local feature extraction and temporal feature extraction during recognition.

[0006] To address the above-mentioned issues, the present invention provides a gait recognition method based on multi-granularity feature learning based on keyframes. This method extracts keyframes from gait profiles within a gait sequence, selects gait profiles that have a significant impact on the recognition results, and assembles them into a keyframe sequence. Gait features are extracted from the keyframe sequence and the original gait sequence respectively, and the gait features are finally fused into gait recognition features, making the extracted gait recognition features more discriminative. During feature extraction, attention is paid to temporal information, and global features are fused with local features to make the recognition features more discriminative. The method comprises the following steps:

[0007] 1) The gait profiles that have the greatest impact on gait recognition results are selected from the gait sequence to form a key frame sequence. The processing branch that takes the key frame sequence as input is called the key frame branch, and the processing branch that takes the original gait sequence as input is called the main branch. This ensures that more attention can be paid to the gait profiles that have the greatest impact on the recognition results.

[0008] 2) Feature extraction operations are performed on the key frame branch and the main branch respectively, and the feature extraction operations of the two branches are the same. In the feature extraction stage, spatial feature extraction operations are performed on the gait profile of each frame in the sequence, and then temporal features are extracted based on the spatial features extracted from adjacent frames; then multi-granularity feature learning is performed, firstly, feature extraction operations are performed on the entire feature map to obtain global features, and at the same time, the entire feature map is divided into blocks, and feature extraction operations are performed on the block-based feature maps to obtain local features, and then global features and local features are fused to obtain multi-granularity features; then, temporal feature pooling operations are performed on the multi-granularity features, and the features after pooling of the two branches are fused to obtain gait recognition features;

[0009] 3) Gait recognition features are used for gait feature matching. During gait feature matching, a generalized average pooling operation is used to adaptively extract spatial features. A fully connected layer is then used to adjust the features. Finally, the similarity of the adjusted gait recognition features of different samples is calculated, and the gait recognition task is completed based on the feature similarity.

[0010] Furthermore, the step 1) of extracting key frames in the gait sequence is as follows:

[0011] 1.1) All gait profiles in the pedestrian gait sequence of the input model are regarded as a set of matrices, and the probability value k of each gait profile frame in the pedestrian gait sequence is calculated as a key frame. The calculation formula of the key frame probability value k is:

[0012]

[0013] in, is the jth gait profile in the gait sequence of the pedestrian with label i, for The row vector of the hth row in , h∈[1,H], where H is the height of the gait profile. is a row vector The wth element in . for Part of The first The matrix composed of row vectors from row 1 to row H corresponds to all parts below the head of the pedestrian in the pedestrian gait profile image.

[0014] 1.2) Set the threshold δ, which is related to the angle between the pedestrian’s walking direction and the sampling lens. δ can be expressed as:

[0015] δ=(De / 18)%6×β+α (3)

[0016] Among them, De is the angle between the pedestrian's walking direction and the sampling lens when the current gait profile is sampled, and α and β are both hyperparameters;

[0017] 1.3) If the keyframe probability k for a gait profile in a gait sequence is greater than δ, the gait profile is identified as a keyframe. For each gait sequence, each gait profile within the sequence is determined to be a keyframe, and those that meet the criteria are selected to form a keyframe sequence. The processing branch that takes the keyframe sequence as input is called the keyframe branch, and the processing branch that takes the original gait sequence as input is called the main branch. Both branches perform the same subsequent feature extraction operations.

[0018] Furthermore, the feature extraction process in step 2) is specifically as follows:

[0019] The key frame branch and the main branch have the same operation in the feature extraction stage. The main branch is taken as an example to illustrate its feature extraction process.

[0020] 2.1) Extract spatiotemporal features of the gait sequence input into this branch, X in Represents the gait sequence of the current branch input, and the spatiotemporal feature extraction process can be expressed as:

[0021] X ST =Te(Sp(X in )) (4)

[0022] Among them, Sp is the spatial feature extraction operation, represents a 3D convolution with a convolution kernel of 3×3×3 and a stride of 3×3×3. Te is a temporal feature extraction operation. Represents a 3D convolution with a kernel of 3×1×1 and a stride of 3×1×1;

[0023] 2.2) For the feature map X obtained in 2.1) ST Perform multi-granularity feature learning operation, and the multi-granularity feature Y obtained after learning MF It can be expressed as:

[0024] Y MF =Y G +Y L (5)

[0026] Among them, Y G is the global feature, Y L It is a local feature.

[0027] Global feature Y G Expressed as:

[0028] Y G =f 3×3×3 (X ST ) (6)

[0029] Among them, f 3×3×3 (·) represents a 3D convolution layer with a kernel size of 3×3×3, which yields a feature map Y containing global information. G .

[0030] Local feature Y L Expressed as:

[0031]

[0032] in, For the feature map X ST The local features of different granularities are obtained by using three partitioning methods. The three partitioning methods are respectively represented by dividing the feature map horizontally into 3 blocks, 4 blocks, and 6 blocks.

[0033] First Take the following example to illustrate the extraction process.

[0034] First, the feature map X ST The horizontal layer is divided into 3 blocks; then the features of each block are extracted separately; finally, the features extracted from each block are cat-operated to obtain the features The processing process is shown in formula (8):

[0035]

[0036] Among them, f 3×3×3 (·) represents a 3D convolutional layer with a convolution kernel of 3×3×3. Is the input feature map X STThe i-th feature map is divided into 3 blocks horizontally and evenly. cat represents the connection operation in the matrix. The obtained feature maps are horizontally spliced ​​to form a feature map containing local information.

[0037] and Extraction process and The extraction process is similar, the only difference is the number of blocks used.

[0038] 2.3) The feature map Y obtained in 2.2) MF Perform time series feature pooling. The pooling process can be expressed as:

[0039] Y T =αF Max (Y MF )+βF Avg (Y MF )+γF Mean (Y MF ) (9)

[0040] Among them, F Max (·) represents the Max-pooling layer, F Avg (·) is the Median-pooling layer, F Mean (·) is the mean-pooling layer, α, β, γ are parameters, and α+β+γ=1.

[0041] 2.4) Based on step 2.3), the features of the key frame branches are obtained respectively Features from the master branch The two features are fused to obtain the gait recognition feature Y M , the processing process is shown in formula (10):

[0042]

[0043] in, It is a feature fusion operation, and the specific operation is to perform matrix cat operation on the channel dimension.

[0044] The step 3) performs gait feature matching on the gait recognition features, specifically:

[0045] 3.1) First, generalized average pooling is used to improve the self-learning ability of the model, enabling it to adaptively integrate spatial information. The feature Y after generalized average pooling is GeM The processing process is shown in formula (11):

[0046]

[0047] Among them, F Avg(·) is the Median-pooling layer, p is an adaptive parameter that can be learned by the network. When p = 1, the generalized average pooling layer is equivalent to the average pooling;

[0048] 3.2) After generalized average pooling, the obtained feature Y GeM Put it into the 2D fully connected layer for adjustment to get Y out , the process is shown in formula (12):

[0049] Y out =f 1×1 (Y GeM ) (12)

[0050] Among them, f 1×1 (·) represents a 2D convolutional layer with a 1×1 convolution kernel.

[0051] 3.3) Calculate the similarity of the gait recognition features of different samples. This model uses Euclidean similarity to calculate the similarity between the gait recognition features of different samples. The similarity S calculation process is shown in formula (13):

[0052]

[0053] Among them, f i and f j are the feature vectors obtained after inputting different samples, is the eigenvector f i With f j Normalized Euclidean distance in feature space.

[0054] The features with the closest feature similarity are matched as gait recognition features belonging to the same pedestrian.

[0055] 3.4) Use cross entropy loss and triplet loss to train the model simultaneously, and use the sum of the two as the final loss of the model. The final loss L com It can be expressed as:

[0056] L com =L cse +L tri (14)

[0057] Among them, L cse and L tri They are cross entropy loss and triple loss respectively. This model uses L com Train the model.

[0058] Cross entropy loss L cse Expressed as:

[0059] L cse =-Σ xp(x)log q(x) (15)

[0060] Where x is the gait recognition feature output by the model, p(·) is the probability value that the current feature belongs to the target label, and q(·) is the probability value that the current feature does not belong to the target label, that is, q(x) = 1-p(x).

[0061] Triplet loss L tri It can be expressed as:

[0062] L tri =[D(F(i),F(k))-D(F(i),F(j))+m] + (16)

[0063] where i and j are samples from the same pedestrian label, k is a sample from a different label than i and j, F(·) is the feature extraction operation corresponding to the model, D(d1,d2) is the Euclidean distance between d1 and d2, m is the boundary of the triplet loss, and the operation [γ] + Equal to max(γ,0).

[0064] The present invention provides a multi-granularity feature learning gait recognition method based on key frames. The method extracts key frames from gait profiles within a gait sequence, selects gait profiles that have a great influence on the recognition result, and composes them into a key frame sequence. Gait features are extracted from the key frame sequence and the original gait sequence respectively, and finally fused into recognition features, so that the extracted recognition features are more discriminative; when extracting features, attention is paid to temporal information, and global features are fused with local features. The method uses cross entropy loss and triplet loss to optimize recognition features to improve recognition accuracy. The application of the present invention solves the problems of past sequence-based gait recognition technologies in performing feature extraction without differentiation, insufficient local feature extraction, and insufficient temporal feature extraction during recognition. The present invention is suitable for pedestrian gait recognition tasks, with high recognition accuracy and good algorithm robustness.

[0065] The advantages of the present invention are: first, the present invention proposes a key frame extraction method to solve the problem of undifferentiated feature extraction; second, the present invention focuses on the temporal information in the gait sequence during feature extraction, and uses temporal feature pooling to process multi-frame information, making full use of the temporal information contained in the sequence; finally, a multi-granularity feature learning method is proposed to fully learn global and local information, and triple loss and cross entropy loss are used to optimize the features, thereby improving the recognition accuracy of the gait recognition method. BRIEF DESCRIPTION OF THE DRAWINGS

[0066] Figure 1 This is a flowchart of the gait recognition method based on multi-granularity feature learning of key frames of the present invention.

[0067] Figure 2 This is a structural diagram of the gait recognition system based on multi-granularity feature learning of key frames of the present invention. DETAILED DESCRIPTION

[0068] The present invention provides a gait recognition method based on multi-granularity feature learning of keyframes. The method takes a gait sequence containing several gait profiles as input, first extracts keyframes from the gait sequence to form a keyframe sequence. The processing branch that takes the keyframe sequence as input is called the keyframe branch, and the processing branch that takes the original gait sequence as input is called the main branch. The keyframe branch and the main branch perform the same feature extraction operation. In the feature extraction stage, a spatial feature extraction operation is performed on the gait profile of each frame in the sequence, and then temporal features are extracted based on the spatial features extracted from adjacent frames. Multi-granularity feature learning is then performed, and the entire feature map is extracted to obtain global features. At the same time, the entire feature map is divided into blocks to obtain local features. The global features and local features are then fused to obtain gait recognition features. The gait recognition features are then matched with gait features. During gait feature matching, a generalized average pooling operation is used to adaptively extract spatial features, and then a fully connected layer is used to adjust the features. Finally, the adjusted gait recognition features of different samples are similarity calculated. The features with the closest feature similarity are matched as gait features belonging to the same pedestrian. This model uses cross entropy loss and triplet loss to optimize the recognition features, such as Figure 1 As shown, the present invention includes the following steps:

[0069] 1) Obtain a gait sequence, which contains gait profiles of several pedestrians walking continuously. The gait sequence is used as a set of matrices as input.

[0070] 2) Gait profiles within a gait sequence have varying impacts on recognition results. Because gait profiles encompass the pedestrian's continuous motion, some body parts may overlap. Therefore, an adaptive keyframe extraction module was designed to identify gait profiles within a gait sequence that contain more gait information and construct keyframe sequences to obtain more discriminative pedestrian gait recognition features.

[0071] 2.1) All gait profiles in the pedestrian gait sequence of the input model are regarded as a set of matrices, and the probability value k of each gait profile frame in the pedestrian gait sequence is calculated as a key frame. The calculation formula of the key frame probability value k is:

[0072]

[0073] in, is the jth gait profile in the gait sequence of the pedestrian with label i, for The row vector of the hth row in , h∈[1,H], where H is the height of the gait profile. is a row vector The wth element in . for Part of The first The matrix composed of row vectors from row 1 to row H corresponds to all parts below the head of the pedestrian in the pedestrian gait profile image.

[0074] 2.2) Set the threshold δ, which is related to the angle between the pedestrian’s walking direction and the sampling lens. δ can be expressed as:

[0075] δ=(De / 18)%6×β+α (19)

[0076] Where De is the angle between the pedestrian's walking direction and the sampling lens when the current gait profile is sampled. α and β are both hyperparameters. The values ​​of the hyperparameters are set to α = 0.5 and β = 0.1.

[0077] 2.3) If the keyframe probability k for a gait profile in a gait sequence is greater than δ, the gait profile is identified as a keyframe. For each gait sequence, each gait profile within the sequence is determined to be a keyframe. Gait profiles that meet the criteria are selected to form a keyframe sequence. The processing branch that takes the keyframe sequence as input is called the keyframe branch, and the processing branch that takes the original gait sequence as input is called the main branch. These two branches perform subsequent feature extraction operations separately.

[0078] 3) For recognition tasks, attention must be paid to both global and local image information. Global information enables discriminative features, while local information allows for greater focus on image detail. Therefore, both global and local information must be considered during feature extraction. For gait recognition tasks, pedestrians move continuously, so extracting gait recognition features also requires attention to the temporal information inherent in the gait sequence. After selecting keyframes, the two branches perform subsequent feature extraction operations.

[0079] The key frame branch and the main branch have the same operation in the feature extraction stage. The main branch is taken as an example to illustrate the operation in the feature extraction process.

[0080] 3.1) Extract spatiotemporal features of the gait sequence input into this branch, X in Represents the gait sequence of the current branch input, and the spatiotemporal feature extraction process can be expressed as:

[0081] X ST =Te(Sp(X in )) (20)

[0082] Among them, Sp is the spatial feature extraction operation, represents a 3D convolution with a convolution kernel of 3×3×3 and a stride of 3×3×3. Te is a temporal feature extraction operation. Represents a 3D convolution with a kernel of 3×1×1 and a stride of 3×1×1;

[0083] 3.2) For the feature map X obtained in 3.1) ST Perform multi-granularity feature learning operation, and the multi-granularity feature Y obtained after learning MF for:

[0084] Y MF =Y G +Y L (twenty one)

[0085] Among them, Y G is the global feature, Y L It is a local feature.

[0086] Global feature Y G Expressed as:

[0087] Y G =f 3×3×3 (X ST ) (twenty two)

[0088] Among them, f 3×3×3 (·) represents a 3D convolution layer with a kernel size of 3×3×3, which yields a feature map Y containing global information. G .

[0089] Local feature Y L Expressed as:

[0090]

[0091] in, For the feature map X ST The local features of different granularities are obtained by using three partitioning methods. The three partitioning methods are respectively represented by dividing the feature map horizontally into 3 blocks, 4 blocks, and 6 blocks.

[0092] First Take the following example to illustrate the extraction process.

[0093] First, the feature map X ST The horizontal layer is divided into 3 blocks; then the features of each block are extracted respectively; finally, the features extracted from each block are cat-operated to obtain the features The processing process is shown in formula (24):

[0094]

[0095] Among them, f 3×3×3 (·) represents a 3D convolutional layer with a convolution kernel of 3×3×3. Is the input feature map X ST The i-th feature map is divided into 3 blocks horizontally and evenly. cat represents the connection operation in the matrix. The obtained feature maps are horizontally spliced ​​to form a feature map containing local information.

[0096] and The extraction process and The extraction process is similar, the only difference is the number of blocks when dividing;

[0097] 3.3) The feature map Y obtained in 3.2) MF Perform time series feature pooling. The pooling process can be expressed as:

[0098] Y T =αF Max (Y MF )+βF Avg (Y MF )+γF Mean (Y MF ) (25)

[0100] Among them, F Max (·) represents the Max-pooling layer, F Avg (·) is the Median-pooling layer, F Mean (·) is the mean-pooling layer, setting α = 0.5, β = 0.25, γ = 0.25.

[0101] 3.4) Based on step 3.3), the features of the key frame branches are obtained respectively Features from the master branch The two features are fused to obtain the gait recognition feature Y M , the processing process is shown in formula (26):

[0102]

[0103] in, It is a feature fusion operation, and the specific operation is to perform matrix cat operation on the channel dimension.

[0104] 4) Gait recognition features are used for gait feature matching. During gait feature matching, generalized average pooling is used to adaptively extract spatial features. Fully connected layers are then used to adjust the features. Finally, similarity is calculated for the adjusted gait recognition features of different samples. Features with the closest feature similarity are matched as belonging to the same pedestrian.

[0105] 4.1) Use generalized average pooling to improve the self-learning ability of the model, so that it can adaptively integrate spatial information. The feature Y after generalized average pooling GeM The processing process is shown in formula (27):

[0106]

[0107] Among them, F Avg (·) is the Median-pooling layer, p is an adaptive parameter that can be learned by the network. When p = 1, the generalized average pooling layer is equivalent to the average pooling;

[0108] 4.2) After generalized average pooling, the obtained feature Y GeM Put it into the 2D fully connected layer for adjustment to get Y out , the process is shown in formula (12):

[0109] Y out =f 1×1 (Y GeM ) (28)

[0110] Among them, f 1×1 (·) represents a 2D convolutional layer with a 1×1 convolution kernel.

[0111] 4.3) The similarity of the gait recognition features obtained from different samples is calculated. This model uses Euclidean similarity to calculate the gait recognition features obtained from different samples. The similarity S calculation process is shown in formula (29):

[0112]

[0113] Among them, f i and f j are the feature vectors obtained after inputting different samples, is the eigenvector f i With f j Normalized Euclidean distance in feature space.

[0114] The features with the closest feature similarity are matched as gait features belonging to the same pedestrian.

[0115] 4.4) Use cross entropy loss and triplet loss to train the model simultaneously, and use the sum of the two as the final loss of the model. The final loss Lcom It can be expressed as:

[0116] L com =L cse +L tri (30)

[0117] Among them, L cse and L tri are cross entropy loss and triplet loss respectively.

[0118] Cross entropy loss L cse Expressed as:

[0119] L cse =-Σ x p(x)log q(x) (31)

[0120] Among them, x is the identification feature output by the model, p(·) is the probability value of the current feature belonging to the target label, and q(·) is the probability value of the current feature not belonging to the target label, that is, q(x) = 1-p(x).

[0121] Triplet loss L tri It can be expressed as:

[0122] L tri =|D(F(i),F(k))-D(F(i),F(j))+m] + (32)

[0123] where i and j are samples from the same pedestrian label, k is a sample from a different label than i and j, F(·) is the feature extraction operation corresponding to the model, D(d1,d2) is the Euclidean distance between d1 and d2, m is the boundary of the triplet loss, and the operation [γ] + Equal to max(γ,0).

[0124] The present invention has a wide range of applications in the field of gait recognition technology, such as pedestrian recognition in public places, long-distance pedestrian recognition, public security, etc. The present invention is described in detail below with reference to the accompanying drawings.

[0125] (1) In an embodiment of the present invention, a gait sequence is obtained and taken as a set of matrices as input. First, a key frame probability value k is calculated for each gait profile in the pedestrian gait sequence. The hyperparameter values ​​α = 0.5 and β = 0.1 are set, and a threshold δ is calculated based on the input gait sequence information. k is compared with δ. If k > δ, the current frame is determined to be a key frame, and all key frames are combined into a key frame sequence.

[0126] (2) The key frame sequence is used as the input of the key frame branch, and the original gait sequence is used as the input of the main branch. The same feature extraction operation is performed on the gait sequences in the two branches.

[0127] (2.1) Perform spatiotemporal feature extraction on the gait sequence. First, use 3D convolution with a kernel size of 3×3×3 to extract spatial features, and then use 3D convolution with a kernel size of 3×1×1 to extract temporal features.

[0128] (2.2) The feature map obtained after spatiotemporal feature extraction is subjected to multi-granularity feature learning, and a 3D convolution layer with a convolution kernel size of 3×3×3 is used to extract global information. At the same time, the input feature map is horizontally divided into blocks, with the number of blocks being 3, 4, and 6. Each block of the divided features is convolved using a separate convolution layer to extract the local information contained in each feature map. The local features obtained after the block division are added together to obtain the total local features. The feature map containing local information is then fused with the feature map containing global information and spliced ​​in the spatial dimension.

[0129] (2.3) To compress the feature information in the temporal dimension of the feature map, temporal feature pooling is performed on the feature map obtained through multi-granularity feature learning. Temporal feature pooling uses maximum pooling, average pooling, and median pooling. Parameters α = 0.5, β = 0.25, and γ = 0.25 are set.

[0130] (2.4) The features of the key frame branch obtained after temporal feature pooling and the features of the main branch are spliced ​​in the channel dimension to obtain the gait recognition features.

[0131] (3) Gait recognition features are used for gait feature matching. When matching gait features, generalized average pooling is used to adaptively extract spatial features. The fully connected layer is used to adjust the features. The similarity of the adjusted gait recognition features of different samples is calculated. The loss function is used to optimize the model.

[0132] (3.1) Improve the self-learning ability of the model and use generalized average pooling on the gait recognition features to enable them to adaptively integrate spatial information, making the recognition features more discriminative;

[0133] (3.2) After generalized average pooling, a 2D fully connected layer is used to adjust the gait recognition features to change the dimension of the gait recognition features;

[0134] (3.3) Use Euclidean similarity to calculate the similarity of gait recognition features of different samples;

[0135] (3.4) The model is trained simultaneously using cross entropy loss and triplet loss, and the sum of the two is used as the final loss of the model to make the final recognition features more discriminative.

[0136] This method was implemented using Python 3.9 programming on an Intel Core i9-10980 and Ubuntu 20.04 operating system.

[0137] This paper provides a gait recognition method based on multi-granularity feature learning based on keyframes. It is suitable for gait recognition tasks and has high recognition accuracy and good algorithm robustness. Experiments show that this method can quickly and effectively perform gait recognition.

Claims

1. A gait recognition method based on multi-granularity feature learning based on keyframes, characterized by: For a given gait sequence, perform the following operations: 1) The gait profiles that have the greatest impact on gait recognition results in the gait sequence are selected to form a key frame sequence. The processing branch that takes the key frame sequence as input is called the key frame branch, and the processing branch that takes the original gait sequence as input is called the main branch, so as to ensure that the gait profiles that have the greatest impact on the recognition results are focused on; 2) The same feature extraction operation is performed on the keyframe branch and the main branch respectively. In the feature extraction stage, spatial feature extraction is performed on each frame of the gait profile in the sequence, and then temporal features are extracted based on the spatial features extracted from adjacent frames. Multi-granularity feature learning is then performed. The entire feature map is first extracted to obtain global features. The entire feature map is then divided into blocks, and local features are extracted from each block. Finally, the global features and local features are fused to obtain multi-granularity features. The multi-granularity features are subjected to temporal feature pooling, and the pooled features of the keyframe branch and the main branch are fused to obtain gait recognition features. 3) Gait feature matching is performed based on gait recognition features. When matching gait features, a generalized average pooling operation is used to adaptively extract spatial features, and then a fully connected layer is used to adjust the spatial features. Finally, the similarity of the adjusted gait recognition features of different samples is calculated. The features with the closest feature similarity are matched as gait features belonging to the same pedestrian.

2. The gait recognition method based on keyframe multi-granularity feature learning according to claim 1 is characterized in that: The step 1) extracts the key frames in the gait sequence, specifically: 1.1) All gait profiles in the pedestrian gait sequence of the input model are regarded as a set of matrices, and the probability value k of each gait profile frame in the pedestrian gait sequence is calculated as a key frame. The calculation formula of the key frame probability value k is: in, is the jth gait profile in the gait sequence of the pedestrian with label i, for The row vector of the hth row in , h∈[1,H], H is the height of the gait profile; is a row vector The w-th element in ; for Part of The first The matrix composed of row vectors from row 1 to row H corresponds to all parts below the head of the pedestrian in the pedestrian gait profile diagram; 1.2) Set the threshold δ, which is related to the angle between the pedestrian’s walking direction and the sampling lens. δ is expressed as: δ=(De / 18)%6×β+α (3) Among them, De is the angle between the pedestrian's walking direction and the sampling lens when the current gait profile is sampled, and α and β are both hyperparameters; 1.3) If the key frame probability value k>δ for the gait profile in the gait sequence, then the gait profile is identified as a key frame. For each gait sequence, each gait profile in the sequence is determined to be a key frame, and the gait profiles that meet the conditions are selected to form a key frame sequence. The processing branch that takes the key frame sequence as input is called the key frame branch, and the processing branch that takes the original gait sequence as input is called the main branch. The two branches perform subsequent feature extraction operations separately.

3. The gait recognition method based on keyframe multi-granularity feature learning according to claim 1 is characterized in that: The feature extraction process in step 2) is specifically as follows: 2.1) Extract spatiotemporal features of the gait sequence input into this branch, X in Represents the gait sequence of the current branch input, and the spatiotemporal feature extraction process is expressed as: X ST =Te(Sp(X in )) (4) Among them, Sp is the spatial feature extraction operation, represents a 3D convolution with a convolution kernel of 3×3×3 and a stride of 3×3×3. Te is a temporal feature extraction operation. Represents a 3D convolution with a kernel of 3×1×1 and a stride of 3×1×1; 2.2) For the feature map X obtained in 2.1) ST Perform multi-granularity feature learning operation, and the multi-granularity feature Y obtained after learning MF for: AND MF =And G +Y L (5) Among them, Y G is the global feature, Y L It is a local feature; Global feature Y G Expressed as: Y G =f 3×3×3 (X ST ) (6) Among them, f 3×3×3 (·) represents a 3D convolution layer with a kernel size of 3×3×3, which yields a feature map Y containing global information. G ; Local feature Y L Expressed as: in, For the feature map X ST The local features of different granularities are obtained by using three partitioning methods. The three partitioning methods are respectively represented by dividing the feature map horizontally into 3 blocks, 4 blocks, and 6 blocks; The extraction process is as follows: First, the feature map X ST The horizontal layer is divided into 3 blocks; then the features of each block are extracted respectively; finally, the features extracted from each block are cat-operated to obtain the features The processing process is shown in formula (8): Among them, f 3×3×3 (·) represents a 3D convolutional layer with a convolution kernel of 3×3×3. Is the input feature map X ST The feature map of the i-th block after being horizontally divided into 3 blocks, cat represents the connection operation in the matrix, and the obtained feature maps are horizontally spliced ​​to form a feature map containing local information; and The extraction process and The extraction process is similar, the only difference is the number of blocks used; 2.3) The feature map Y obtained in 2.2) MF Perform time series feature pooling. The pooling process is as follows: AND T =αF Max (AND MF )+βF Avg (AND MF )+γF Mean (AND MF ) (9) Among them, F Max (·) represents the Max-pooling layer, F Avg (·) is the Median-pooling layer, F Mean (·) is the mean-pooling layer, α, β, γ are parameters, and α+β+γ=1; 2.4) According to step 2.3), the features of the key frame branches are obtained respectively. Features from the master branch The two features are fused to obtain the gait recognition feature Y M , the processing process is shown in formula (10): in, It is a feature fusion operation, and the specific operation is to perform matrix cat operation on the channel dimension.

4. The gait recognition method based on keyframe multi-granularity feature learning according to claim 1 is characterized in that: The gait feature matching process in step 3); 3.1) Using generalized average pooling operation, the feature Y after generalized average pooling GeM The processing process is shown in formula (11): Among them, F Avg (·) is the Median-pooling layer, p is the adaptive parameter learned by the network. When p = 1, the generalized average pooling layer is equivalent to the average pooling; 3.2) After generalized average pooling, the obtained feature Y GeM Put it into the 2D fully connected layer for adjustment to get Y out , the process is shown in formula (12): Y out =f 1×1 (Y GeM ) (12) Among them, f 1×1 (·) represents a 2D convolutional layer with a convolution kernel of 1×1; 3.3) The gait recognition features obtained from different samples are similarly calculated. The Euclidean similarity is used to calculate the gait recognition features obtained from different samples. The similarity S calculation process is shown in formula (13): Among them, f i and f j are the feature vectors obtained after inputting different samples, is the eigenvector f i With f j Normalized Euclidean distance in feature space; Match the features with the closest feature similarity as gait features belonging to the same pedestrian; 3.4) Use cross entropy loss and triplet loss to train the model simultaneously, and use the sum of the two as the final loss of the model; the final loss L com Expressed as: L com =L cse +L tri (14) Among them L cse and L tri They are cross entropy loss and triple loss respectively; this model uses L com To train the model; Cross entropy loss L cse Expressed as: L cse =-S x p(x)logq(x) (15) Where x is the identification feature output by the model, p(·) is the probability value of the current feature belonging to the target label, and q(·) is the probability value of the current feature not belonging to the target label, that is, q(x) = 1-p(x); Triplet loss L tri Expressed as: L tri =[D(F(i),F(k))-D(F(i),F(j))+m] + (16) where i and j are samples from the same pedestrian label, k is a sample from a different label than i and j, F(·) is the feature extraction operation corresponding to the model, D(d1,d2) is the Euclidean distance between d1 and d2, m is the margin of the triplet loss, and the operation [γ] + Equal to max(γ,0).

5. The gait recognition method based on keyframe multi-granularity feature learning according to claim 2, 3 or 4, characterized in that: The given gait sequence is obtained by processing videos shot by cameras at multiple angles. The gait sequence contains gait profiles of the same pedestrian at different angles, and there is no overlap between the pedestrian labels contained in the training set and the test set.

Images