Gait recognition model training method, gait recognition method and device

Abstract

This application provides a gait recognition model training method, a gait recognition method, and an apparatus. The recognition method includes: acquiring a gait image sequence to be recognized; inputting it into a trained Visual Geometric Group (VGG) network to obtain a spatial feature vector set of the gait image sequence; inputting the spatial feature vector set of the gait image sequence into a trained Long Short-Term Memory (LSTM) network to obtain a periodic motion feature vector set of the gait image sequence; and inputting the periodic motion feature vector set into a feature fusion unit to obtain a fused feature vector of the gait image sequence used for gait recognition. Because it fuses significant spatial and temporal features of the gait sequence, it improves the accuracy of gait recognition to a certain extent.

Description

Technical Field

[0001] This application relates to the field of computer vision technology, and in particular to a gait recognition model training method, gait recognition method and apparatus. Background Technology

[0002] Pedestrian gait is a crucial biometric feature, possessing the unique ability to remotely identify pedestrians. Compared to other biometrics such as face, iris, and fingerprint, gait has the following significant advantages: 1) it can be acquired remotely and non-contactly; 2) it is difficult to imitate or disguise; and 3) even in low-quality video, pedestrian gait can still be effectively acquired. In recent years, gait recognition in video surveillance systems has attracted widespread attention from researchers. Research findings can be applied to border security management, smart camps, warehouse security management, smart security, intelligent criminal investigation, and identity recognition, among other fields. These challenging applications largely depend on the accuracy of gait recognition.

[0003] Currently, research on gait recognition methods mainly falls into two categories. The first category is model-based gait recognition methods: these methods model the movements of the human limbs and extract human structures from gait images to achieve gait recognition. Some researchers have achieved gait recognition by calculating optical flow fields to statistically analyze two-point data points in gait sequences, encoding pedestrian limb movements to form two-point gait, and applying human proportion measurement methods to model lower limb postures. However, in low-quality surveillance videos, it is difficult to accurately estimate the motion model parameters of pedestrian gait, resulting in low accuracy for these methods. The second category is appearance-based gait recognition methods: these methods extract gait features from acquired gait image sequences, enabling gait recognition even in low-resolution videos. Researchers have proposed recognition methods based on deep learning, arbitrary viewpoint transformation models (AVTM), and gait temporal templates, mapping multi-view, multi-channel gait sequences to the same state space for gait matching. These methods typically use gait energy maps (GEI) as the processing object. However, GEI loses a lot of dynamic information about gait during the calculation process. GEI can only represent a small amount of dynamic information about gait, which means that the periodic motion features of gait cannot be effectively utilized, and the recognition accuracy is not high, so it cannot meet the application requirements.

[0004] In summary, existing gait recognition methods suffer from low accuracy. Summary of the Invention

[0005] This application provides a gait recognition model training method, a gait recognition method, and an apparatus to improve the accuracy of gait recognition.

[0006] A gait recognition model training method includes:

[0007] Obtain gait image sequence training pairs and labels, wherein the labels are used to characterize whether the gait image sequence training pairs belong to the same identity;

[0008] The two gait image sequences that constitute the training pair are respectively input into two parallel Visual Geometry Group (VGG) networks to obtain the spatial feature vector sets of the two gait image sequences;

[0009] The two spatial feature vector sets are respectively input into two parallel Long Short Term Memory (LSTM) networks to obtain the periodic motion feature vector sets of the two gait image sequences.

[0010] The periodic motion feature vector sets of two gait image sequences are respectively input into two parallel feature fusion units to obtain the feature vectors of the two fused gait image sequences. The VGG network + LSTM network + feature fusion unit constitutes a sub-network in the twin LSTM architecture network, and this sub-network constitutes the gait recognition model.

[0011] The two fused feature vectors are input into the contrast loss objective function to obtain the loss value. This loss value is used to characterize the difference between the two fused feature vectors and the label. In the second and subsequent iterations, it is determined whether the loss value has decreased. If so, the training parameters of the Siamese LSTM architecture network are adjusted according to the loss value, and the gait image sequence training pair is input into the adjusted Siamese LSTM architecture network again until the loss value no longer decreases, and the training ends.

[0012] A gait recognition method, comprising:

[0013] Obtain the gait image sequence to be identified;

[0014] Inputting it into the trained VGG network yields a set of spatial feature vectors for the gait image sequence;

[0015] The spatial feature vector set of the gait image sequence is input into the trained LSTM network to obtain the periodic motion feature vector set of the gait image sequence;

[0016] The periodic motion feature vector set is input into the feature fusion unit to obtain the feature vector of the fused gait image sequence used to realize gait recognition;

[0017] The trained VGG network and the trained LSTM network were trained using the methods described above.

[0018] A gait recognition model training device, comprising:

[0019] The sample acquisition unit is used to acquire gait image sequence training pairs and labels, wherein the labels are used to characterize whether the gait image sequence training pairs belong to the same identity;

[0020] Two parallel VGG network training units are used to receive two gait image sequences that constitute a training pair and obtain a set of spatial feature vectors for the two gait image sequences.

[0021] Two parallel LSTM network training units are used to receive two spatial feature vector sets to obtain two periodic motion feature vector sets of gait image sequences. Among them, the VGG network + LSTM network + feature fusion unit constitute a sub-network in the twin LSTM architecture network, and this sub-network constitutes the gait recognition model.

[0022] Two parallel feature fusion units are used to fuse the periodic motion feature vector sets of two gait image sequences respectively to obtain the feature vectors of the two fused gait image sequences.

[0023] The loss calculation unit is used to input the two fused feature vectors into the contrast loss objective function to obtain the loss value, which is used to characterize the difference between the two fused feature vectors and the label.

[0024] The judgment unit is used to determine whether the loss value has decreased in the second and subsequent iterations. If so, the parameter adjustment unit and the sample acquisition unit are triggered until the loss value no longer decreases.

[0025] The parameter tuning unit is used to adjust the training parameters of the Siamese LSTM architecture network based on the loss value.

[0026] A gait recognition device, comprising:

[0027] The image acquisition unit is used to acquire the gait image sequence to be identified;

[0028] The trained VGG network is used to extract the set of spatial feature vectors from the gait image sequence to be identified;

[0029] The trained LSTM network is used to extract the set of periodic motion feature vectors of the gait image sequence from the set of spatial feature vectors of the gait image sequence;

[0030] The feature fusion unit is used to perform temporal fusion of the periodic motion feature vector set to obtain the fused feature vector for gait recognition.

[0031] The trained VGG network and the trained LSTM network were obtained through the gait recognition model training device described above.

[0032] In this embodiment, a twin LSTM architecture network is first used to train the VGG network and the LSTM network. After training, one sub-network of the twin LSTM architecture network is used as the gait recognition training model for gait recognition. The VGG network learns the appearance features of the gait, and the LSTM network learns the periodic motion features of the gait. The learned periodic motion features are then fused. Because the significant spatial and temporal features of the gait sequence are fused, the accuracy of gait recognition is improved to a certain extent. Attached Figure Description

[0033] Figure 1 This is a framework diagram of the gait recognition training model provided in the embodiments of this application;

[0034] Figure 2 A flowchart of the gait recognition training method provided in the embodiments of this application;

[0035] Figure 3 A flowchart of the gait recognition method provided in the embodiments of this application;

[0036] Figure 4 This is a schematic diagram of the gait recognition training device provided in the embodiments of this application;

[0037] Figure 5 This is a schematic diagram of a gait recognition device provided in an embodiment of this application. Detailed Implementation

[0038] This application provides a gait recognition model training method, a gait recognition method, and an apparatus to improve gait recognition accuracy. In the embodiments of this application, firstly, a gait recognition model training method is provided. In this method, to obtain discriminative appearance features of gait, the VGG network is fine-tuned using gait images, and the fine-tuned model is used to extract spatial features from pedestrian gait images to represent gait appearance information. Secondly, to accurately capture the significant periodic motion features of pedestrian gait, a temporal summarization Siamese LSTM neural network is designed to automatically learn the significant periodic motion features of gait sequences. The LSTM unit is used to model the periodic motion of the gait sequence, the feature fusion unit captures significant temporal information, and the Siamese network architecture addresses the problem of a large number of categories and a small number of samples per category in gait recognition. Then, a gait recognition method is provided. This method uses a branch of the Siamese LSTM network as a feature extractor to extract and fuse the appearance and periodic motion features of the gait image sequence to be recognized, thereby achieving gait recognition functionality. Because it incorporates significant spatial and temporal features from gait image sequences, it improves the accuracy of gait recognition to some extent.

[0039] The preferred embodiments of this application are described below with reference to the accompanying drawings. It should be understood that the preferred embodiments described herein are for illustration and explanation only and are not intended to limit this application. Furthermore, the embodiments and features described herein can be combined with each other unless otherwise specified.

[0040] Example 1

[0041] like Figure 1 As shown, this is a framework diagram of the gait recognition training model provided in the application embodiment. The training model framework consists of two identical sub-networks and a contrastive loss objective function connecting the two sub-networks. One sub-network consists of a VGG network + an LSTM network + a feature fusion unit. The two identical sub-networks combined can also be called a twin LSTM architecture network. Furthermore, this sub-network constitutes the gait recognition model of this application embodiment.

[0042] Figure 2 In this model, the VGG network is used to extract spatial features of gait images, the LSTM network is used to extract periodic features of gait images, and the feature fusion unit is used to fuse the periodic features of gait images.

[0043] Below is a comparison Figure 1 The training model framework of this application is described in detail, and the gait recognition model training method provided in the embodiments of this application is described in detail.

[0044] like Figure 2 As shown, the specific steps include:

[0045] Step 201: Obtain gait image sequence training pairs (S) i ,S j ) and label Y, the label being used to characterize whether the gait image sequence training pairs belong to the same identity.

[0046] In this embodiment of the application, gait recognition is regarded as a binary classification problem, and the training data includes gait image sequence pairs and labels.

[0047] In this embodiment, gait image sequence pairs are used as input, each gait image sequence is mapped to a feature vector, and then a similarity measurement is performed. For all gait sequences in the training set, this embodiment randomly selects two gait sequences belonging to the same category to form a positive training pair and randomly selects two sequences belonging to different categories to form a negative training pair. Given a gait sequence training pair (S) i ,S j ), where Y∈[0,1] are the labels of the training pairs, and Y=1 indicates that S i and S j They belong to the same identity, while Y=0 indicates that they belong to different identities.

[0048] Here, the gait image sequences used as training samples are divided into many pairs. Each time this step is performed, a new pair of gait image sequences is obtained.

[0049] Step 202: Combine the two gait image sequences S that constitute the training pair i and S j The inputs are fed into two parallel VGG networks to obtain spatial feature vector sets X for two gait image sequences. i and X j .

[0050] Here, the VGG network model can select the appropriate number of layers based on actual needs, such as VGG16, VGG19, and VGG20.

[0051] This application's embodiments preferentially use a VGG16 network model to learn the spatial features of gait images. Since the pre-trained VGG16 model uses ImageNet as training data, it cannot be directly applied to gait recognition. Therefore, the pre-trained VGG16 model is fine-tuned on a gait dataset to transfer it to the gait image domain. The fine-tuning solution involves setting the output parameters of the fully connected fc8 layer to the number of gait image categories, training the VGG16 model several times using gait image sequences, and thus adjusting the parameters of all layers in the VGG16 model. After fine-tuning, the VGG16 network model can capture richer spatial features of gait images. Given a gait image sequence processing unit S = {s1,…,s…} t ,…s T}, where s t Here, is the gait image at time step t, and T is the length of the gait sequence processing unit S. The spatial feature vector set X = {x1, ..., x...} of the gait image sequence processing unit S is extracted using a fine-tuned VGG16 network model. t ,…x T}

[0052] Step 203: Input the two spatial feature vector sets into two parallel LSTM networks respectively to obtain the periodic motion feature vector sets H of the two gait image sequences. i and H j .

[0053] LSTM networks are a special type of recurrent neural network that uses memory unit modules to regulate the interaction between specific relevant information and its surrounding environment, purposefully retaining specific relevant information. This allows them to capture long-term temporal dynamics in video sequences and enhance the ability to discriminate temporal information.

[0054] The gait periodicity of pedestrians is an important feature for distinguishing different pedestrians. Therefore, this application uses an LSTM neural network to model the periodic motion of pedestrian gait and extract periodic motion features with more discriminative characteristics to represent the gait periodicity, thereby improving the performance of pedestrian gait recognition.

[0055] The LSTM network memory unit consists of four main parts: an input gate, a neuron with a self-recurrent connection, a forget gate, and an output gate. The input gate updates the memory unit's state based on the input data or blocks updates; the self-recurrent connection ensures that the memory unit's state has feedback with a one-time-step delay; the forget gate allows the memory unit to remember or forget its previous state by adjusting its self-recurrent connection; and the output gate allows the memory unit's state to influence or prevent influence on other neurons. In summary, the LSTM network memory unit allows the neural network to learn when to forget previous hidden states and when to update the hidden state to retain new relevant information.

[0056] Next, the spatial feature vector set X of the gait sequence processing unit S is input into the LSTM units constituting the LSTM network to learn the periodic temporal information of the gait image sequence. For time step t, x t and h t The input vectors are W and W respectively. x For the input weight matrix, W h Let b be the cyclic weight matrix and b be the bias vector. Sigmoid function. It is an element-wise nonlinear activation function that maps real number inputs to the range (0,1). Hyperbolic tangent function. It is also an element-wise non-linear activation function that maps the input to the range (-1, 1). The operator ⊙ represents element-wise multiplication.

[0057] Given input x t h t-1 and c t-1 The update equation for LSTM cell time step t can be expressed as:

[0058] i t =σ(W xi x t +W hi h t-1 +b i (1)

[0059] f t =σ(W xf x t +W hf h t-1 +b f (2)

[0060] o t=σ(W xo x t +W ho h t-1 +b o (3)

[0061]

[0062] c t =f t ⊙c t-1 +i t ⊙g t (5)

[0063]

[0064] As can be seen from formulas (1) to (6), a hidden unit is obtained at each time step. An LSTM unit consists of input gates Forgotten Gate Output gate Input regulating gate and memory unit Composition. Memory unit c t It is C t-1 Memory part and regulated i t The sum of . f in the LSTM unit t and g t The knob is considered to selectively forget its previous memory and consider the current input. t This is used to learn which information in the memory unit needs to be transferred to the hidden unit. Finally, the hidden state set H = {h1,…,h} of the gait sequence processing unit S is obtained. t ,…h T} represents the long-term temporal information of the gait sequence processing unit S, namely, periodic motion information.

[0065] Since the number of positive training pairs for gait sequences is far less than the number of negative training pairs, it may lead to data imbalance and overfitting problems. To avoid these problems, it is preferable to randomly discard 40%-60% of the neuron outputs in the LSTM layer during training, while retaining the neuron outputs of the LSTM layer and passing them to subsequent layers. With multiple iterations on the training data, the model is trained to convergence, and training ends.

[0066] Step 204: Extract the periodic motion feature vector set H from the two gait image sequences. i and H j The inputs are fed into two parallel feature fusion units to obtain feature vectors V of the two fused gait image sequences. i and V j .

[0067] Specifically, feature fusion can be achieved through three methods: the first is the Gait Temporal Information Weighted Summary (WTS) fusion method; the second is Mean-pooling, which fuses the outputs of all LSTM time steps; and the third is Max-pooling, which fuses the outputs of all LSTM time steps. Specifically, the Gait Temporal Information Weighted Summary assigns unequal importance to the outputs of all LSTM time steps, the Mean-pooling method considers all LSTM time step outputs to be equally important, and the Max-pooling method uses the maximum activation value of the LSTM outputs across all time steps as the feature vector.

[0068] Preferably, the periodic motion feature vector sets of the two gait image sequences are summed by temporal weighting, wherein the weight of the feature value that is earlier in the temporal sequence is less than the weight of the feature value that is later in the temporal sequence.

[0069] Although LSTM can capture the temporal information of gait sequences, it places more emphasis on the output information of the last time step, which reduces gait recognition performance because the outputs of earlier time steps also contain some useful temporal information. To address this issue, this application proposes a Weighted Summary of Temporal Information (WTS) module, providing a weighted solution to aggregate and summarize the information from all time steps, thereby capturing temporal features with strong discriminative power. Specifically, the gait feature vector V generated by the WTS module can be formalized as:

[0070]

[0071] Where T represents the number of time steps (length) of the gait sequence, h t It is the output of the LSTM at time step t.

[0072] Ideally, through experimental verification, the LSTM network has 1 layer and the number of hidden units is greater than or equal to 3096.

[0073] Furthermore, in the field of gait recognition, sample data has the following characteristics: a very large number of categories that cannot be determined during model training, and a very small number of samples for each category. Siamese architecture networks can learn a similarity measurement model from training data using discriminative learning techniques, without requiring specific sample category information. It consists of two structurally identical subnetworks with shared weights, mapping the input object to a low-dimensional target space for similarity measurement. Therefore, Siamese architecture networks are suitable for solving gait recognition problems. The embodiments of this application are designed with... Figure 1 The Siamese LSTM network shown is used to learn the periodic motion features of gait sequences. Due to the low dimensionality of the Siamese LSTM network's target space, it is able to learn highly discriminative features between different gait sequences from a small number of samples in each class.

[0074] Step 205: Input the two fused feature vectors into the contrast loss objective function to obtain the loss value, which is used to characterize the difference between the two fused feature vectors and the label.

[0075] For gait recognition tasks, it is necessary to learn a nonlinear function to map gait sequences to points in a low-dimensional space, such that positive training pairs are as close as possible, while negative training pairs are as far apart as possible, but at least greater than a minimum distance interval. Therefore, embodiments of this application employ a contrastive loss function to connect two LSTM networks.

[0076] For the feature vector V of the gait sequence i and V j The distance metric method can be expressed as:

[0077] D(V i V j )=||V i -V j ||2 (8)

[0078] The contrastive loss objective function can be formally expressed as:

[0079]

[0080] Where the positive number m is the minimum distance interval between different types of gait sequences.

[0081] Step 206: Determine if the number of iterations is greater than or equal to 2. If yes, proceed to step 207; otherwise, proceed to step 201.

[0082] Here, performing steps 201 to 205 once constitutes one iteration.

[0083] Step 207: Determine whether the loss value obtained in this iteration is less than the loss value obtained in the previous iteration. If yes, proceed to step 208; otherwise, proceed to step 209.

[0084] Step 208: Adjust the network parameters of the Siamese LSTM architecture based on the loss value obtained in this iteration, and then jump to step 201.

[0085] Here, adjusting the parameters of the twin LSTM architecture specifically involves adjusting the parameters of the VGG network model and the LSTM network.

[0086] Step 209: Training ends.

[0087] Steps 205 to 209 above refer to determining whether the loss value decreases in each subsequent iteration. If so, the parameters of the Siamese LSTM architecture network are adjusted according to the loss value, and the gait image sequence training pair is input into the adjusted Siamese LSTM architecture network again until the loss value no longer decreases, at which point the training ends.

[0088] In the scheme of this application embodiment, a Siamese LSTM network is used in the gait recognition training stage. This is because in the field of gait recognition, the sample data has the characteristics of a very large number of categories and the model training stage cannot determine the number of samples, while the number of samples in each category is very small. The Siamese architecture network can learn a similarity measurement model by applying discriminative learning techniques from the training data without requiring specific sample category information. It contains two subnetworks with identical structures and shared weights, mapping the input object to a low-dimensional target space for similarity measurement. Therefore, the Siamese architecture network is suitable for solving the gait recognition problem. This application embodiment designs a Siamese LSTM network to learn the periodic motion features of gait sequences. Furthermore, because the target space of the Siamese LSTM network has a low dimension, it can learn strong discriminative features between different gait sequences from a small number of samples in each category.

[0089] During the training phase described above, a weight-sharing mechanism is used to simultaneously optimize two LSTM networks. Paired gait sequences are input into the two LSTM networks respectively, and the contrastive loss is calculated using the contrastive loss function and backpropagated to train the twin LSTM network.

[0090] Since the number of positive training pairs for gait sequences is far less than the number of negative training pairs, it may lead to data imbalance and overfitting problems. To avoid these problems, it is preferable to randomly discard 40%-60% of the neuron outputs of the LSTM layer during training, for example, randomly discarding 50% of the neuron outputs. The LSTM layer retains the neuron outputs and passes them to subsequent layers. With multiple iterations on the training data, the model is trained to convergence and training ends.

[0091] Experiments have shown that the optimal LSTM network has 1 layer and 3096 or more hidden units.

[0092] Experimental verification shows that the optimal minimum distance interval for training pairs of gait image sequences that do not belong to the same identity, based on the contrast loss objective function, is greater than or equal to 0.5 and less than or equal to 1.5.

[0093] Example 2

[0094] Based on the gait recognition model trained using the model training method in Embodiment 1, Embodiment 2 of this application provides a gait recognition method, such as... Figure 3 As shown, it includes the following steps:

[0095] Step 301: Obtain the gait image sequence to be identified.

[0096] Step 302: Input it into the trained VGG network to obtain the spatial feature vector set of the gait image sequence.

[0097] Step 303: Input the spatial feature vector set of the gait image sequence into the trained LSTM network to obtain the periodic motion feature vector set of the gait image sequence.

[0098] Step 304: Input the periodic motion feature vector set into the feature fusion unit to obtain the feature vector of the fused gait image sequence used to realize gait recognition.

[0099] The solution in Embodiment 2 of this application can be applied to fields such as border security management, smart barracks, warehouse security management, smart security, intelligent criminal investigation, and identity recognition. Specifically, it can be used to obtain feature vectors of a reference gait image sequence and feature vectors of gait image sequences to be determined using the gait recognition method of this application embodiment. The similarity (e.g., Euclidean distance) between the feature vectors of the reference gait image sequence and the feature vectors of each gait image sequence to be determined is then calculated, and the similarity is used to determine whether it is the same person / identity.

[0100] Example 3

[0101] Based on the same inventive concept as Embodiment 1 of this application, Embodiment 3 of this application also provides a gait recognition model training device, such as... Figure 4 As shown, it includes:

[0102] The sample acquisition unit 401 is used to acquire gait image sequence training pairs and labels, wherein the labels are used to characterize whether the gait image sequence training pairs belong to the same identity;

[0103] Two parallel VGG network training units 402 are used to receive two gait image sequences that constitute a training pair and obtain a set of spatial feature vectors for the two gait image sequences.

[0104] Two parallel LSTM network training units 403 are used to receive two spatial feature vector sets to obtain two periodic motion feature vector sets of gait image sequences. Among them, the VGG network + LSTM network + feature fusion unit constitute a sub-network in the twin LSTM architecture network, and this sub-network constitutes the gait recognition model.

[0105] Two parallel feature fusion units 404 are used to perform feature temporal summarization on the periodic motion feature vector sets of the two gait image sequences respectively, so as to obtain the feature vectors of the two fused gait image sequences.

[0106] The loss calculation unit 405 is used to input the two fused feature vectors into the contrast loss objective function to obtain the loss value, which is used to characterize the difference between the two fused feature vectors and the label.

[0107] The judgment unit 406 is used to determine whether the loss value has decreased in the second and subsequent iterations. If so, the parameter adjustment unit and the sample acquisition unit are triggered until the loss value no longer decreases.

[0108] The parameter adjustment unit 407 is used to adjust the training parameters of the Siamese LSTM architecture network according to the loss value.

[0109] Preferably, the feature fusion unit 404 is specifically used to input the periodic motion feature vector sets of the two gait image sequences into two parallel feature fusion units respectively, and to perform temporal weighted summation on the periodic motion feature vector sets, wherein the weight of the feature values ​​that are earlier in the time sequence is less than the weight of the feature values ​​that are later in the time sequence.

[0110] Example 4

[0111] like Figure 5 As shown in the figure, this application embodiment also provides a gait recognition device, including:

[0112] The image acquisition unit 501 is used to acquire the gait image sequence to be identified;

[0113] The trained VGG network 502 is used to extract the set of spatial feature vectors from the gait image sequence to be identified;

[0114] The trained LSTM network 503 is used to extract the set of periodic motion feature vectors of the gait image sequence from the set of spatial feature vectors of the gait image sequence;

[0115] The feature fusion unit 504 is used to perform temporal fusion of the periodic motion feature vector set to obtain the fused feature vector for gait recognition.

[0116] The trained VGG network and the trained LSTM network are trained using any of the step recognition model training devices shown in Example 3.

[0117] Example 5

[0118] Embodiment 5 of this application describes the testing and related experimental conditions.

[0119] During testing, the Siamese architecture and contrastive loss function were discarded. Instead, a branch of the Siamese LSTM network (i.e., a fine-tuned VGG16 model and the LSTM network portion) was used as a feature extractor to extract appearance and periodic motion features of gait sequences from the reference and query sets. When jointly extracting appearance and periodic motion features of gait sequences, normalization was performed before feature fusion. This application's dataset and evaluation criteria are described below.

[0120] 1) Dataset: The performance of our proposed method was evaluated using the OULP-C1V1-A gait dataset, a large-scale and comprehensive benchmark dataset for gait recognition, containing 3836 pedestrians aged 1 to 94 years. Each pedestrian included two types of gait sequences: a reference sequence and a query sequence. The gait sequences consisted of pedestrian silhouette images, which were normalized to 128×88 pixels. Pedestrians in the reference and query sets were divided into five subsets based on viewing angle: 55°, 65°, 75°, 85°, and all angles. In the experiments, the reference set was used for training, while the query set was not used.

[0121] 2) Evaluation Criteria: Since each query sequence in the OULP-C1V1-A gait dataset has multiple corresponding cross-view reference sequences, MAP is used to evaluate the performance of the proposed method. MAP is used to calculate the recognition accuracy of all query gait sequences, and is defined as follows:

[0122]

[0123] Where Q represents the query set, m j Precision(R) represents the number of reference gait sequences corresponding to query sequence j. jk ) represents the average accuracy (AP) of the reference sequence k corresponding to the query sequence j.

[0124] In addition, this paper uses cumulative matching curves (CMC) to calculate Rank-1 and Rank-5, which represent the percentage of all correctly matched reference sequences ranked first and top five, respectively. These are also widely used methods for evaluating gait recognition performance.

[0125] Experimental setup

[0126] To objectively evaluate the proposed method, the minimum distance interval, the number of hidden units in the LSTM, and the number of LSTM layers were adjusted based on negative training of the loss function using training data. First, the minimum distance interval *m* of the negative samples in the Siamese architecture network was adjusted. The validation performance was best when *m* = 1; there was no significant change in validation performance when *m* = 0.5 or *m* = 1.5; however, the validation performance significantly decreased when *m* < 0.5 or *m* > 1.5. Therefore, *m* was set to 1 in subsequent experimental evaluations.

[0127] Secondly, experiments were conducted to explore the impact of the number of hidden units in the LSTM on its learning ability. Due to the large number of pedestrian categories in the dataset (3836 pedestrians) and the small differences in pedestrian gait images, experiments were conducted to test the accuracy using matching numbers of hidden units in the LSTM, specifically 1024, 2048, 3096, and 5120 hidden units. As the number of hidden units in the LSTM increased, the gait recognition accuracy continuously improved. However, when the hidden unit size reached 5120, the accuracy could not be further improved. Therefore, in subsequent evaluation experiments, the number of hidden units in the LSTM was set to 5120.

[0128] Finally, experiments were conducted to verify the number of LSTM layers. One, two, and four-layer LSTM network structures were tried. The results showed that a one-layer LSTM achieved the best recognition performance on the OULP-C1V1-A gait dataset. Therefore, the number of LSTM layers was set to 1 in subsequent evaluation experiments.

[0129] Time-weighted summary performance verification and analysis

[0130] In the experiment, the outputs of all time steps of the LSTM in the gait sequence need to be fused into a single feature vector to complete gait recognition. Besides using the proposed Gait Temporal Information Weighted Summary (WTS) fusion method, mean pooling and max pooling were also used to fuse the outputs of all time steps of the LSTM. Specifically, the mean pooling method considers all time step outputs of the LSTM to be equally important, while the max pooling method uses the maximum activation value of all time step outputs of the LSTM as the feature vector. On the OULP-C1V1-A gait dataset, the Siamese LSTM network structure designed in this paper was trained and tested using temporal weighted summary, mean pooling, and max pooling, respectively, and their respective gait recognition performance was compared and analyzed. As shown in Tables 1 and 2, WTS achieved better gait recognition performance than mean pooling and max pooling methods. The validation results show that the WTS method can better capture the periodic motion cues of the gait sequence.

[0131] Table 1 Comparison of recognition performance of different fusion methods based on MAP evaluation criteria

[0132]

[0133]

[0134] Table 2 Comparison of recognition performance of different fusion methods based on the Rank-1 recognition rate evaluation criterion

[0135]

[0136] (1) Gait recognition performance evaluation analysis from the same perspective

[0137] First, the proposed method is evaluated under the same viewpoints, comparing its recognition performance with that of GEI, FDF, woVTM, CNN, and SiaNet. The performance comparison results are shown in Table 3. The proposed method outperforms other comparative methods in recognition performance at 55°, 65°, 75°, and 85° viewpoints, achieving the best gait recognition accuracy. Compared to existing traditional methods (GEI, FDF, and woVTM), the proposed method captures discriminative appearance features and richer pedestrian motion information from gait images. Most traditional methods use GEI to represent pedestrian gait motion information, leading to the loss of a large amount of useful motion information. The proposed method achieves a significant improvement in gait recognition performance. Compared to existing deep learning-based methods (CNN and SiaNet), the proposed method learns significant periodic motion information of gait sequences using a Siamese LSTM architecture. Existing deep learning-based methods use GEI as training data, capturing only a small amount of motion information. The proposed method achieves the best gait recognition performance.

[0138] Table 3 Comparison of Rank-1 and Rank-5 recognition rates of different time-state recognition methods under the same viewpoint.

[0139]

[0140] (2) Performance evaluation and analysis of gait recognition under cross-view

[0141] In real-world applications, gait sequences are often affected by changes in camera shooting angle. Therefore, the recognition performance of our proposed method is evaluated under cross-viewpoint conditions, and compared with methods such as woVTM, AVTM, AVTM_PdVS, and SiaNet. The performance comparison results are shown in Table 4. The first three methods construct 3D gait sequences specifically for cross-viewpoint gait matching, while the SiaNet method uses a Siamese neural network to capture the appearance features of the gait sequence. Our proposed method considers both the viewpoint differences in gait appearance and significant periodic motion information, achieving the best gait recognition performance. When the viewpoint differences are large, the recognition performance of our proposed method will slightly decrease.

[0142] Table 4. Comparison of Rank-1 recognition rates of different time-state recognition methods under cross-view perspectives.

[0143]

[0144] Through the above description of the embodiments, those skilled in the art can clearly understand that the embodiments of this application can be implemented by hardware or by using software plus necessary general-purpose hardware platforms.

[0145] Based on this understanding, the technical solutions of the embodiments of this application can be embodied in the form of a software product. The software product can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, mobile hard drive, etc.) and includes several instructions to cause a computer device (such as a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0146] Those skilled in the art will understand that the accompanying drawings are merely schematic diagrams of a preferred embodiment, and the modules or processes shown in the drawings are not necessarily essential for implementing this application.

[0147] Those skilled in the art will understand that the modules in the terminal of the embodiments can be distributed in the terminal of the embodiments as described in the embodiments, or they can be located in one or more terminals different from this embodiment with corresponding changes. The modules of the above embodiments can be combined into one module, or they can be further divided into multiple sub-modules.

[0148] The sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0149] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the spirit and scope of this application. Therefore, if such modifications and variations fall within the scope of the claims of this application and their equivalents, this application also intends to include such modifications and variations.

Claims

1. A method for training a gait recognition model, characterized in that, include: Obtain gait image sequence training pairs and labels, wherein the labels are used to characterize whether the gait image sequence training pairs belong to the same identity; The two gait image sequences that constitute the training pair are respectively input into two parallel visual geometry groups (VGG) networks to obtain the spatial feature vector sets of the two gait image sequences. The two spatial feature vector sets are respectively input into two parallel long short-term memory (LSTM) networks to obtain the periodic motion feature vector sets of the two gait image sequences. The periodic motion feature vector sets of two gait image sequences are respectively input into two parallel feature fusion units to obtain the feature vectors of the two fused gait image sequences. The VGG network + LSTM network + feature fusion unit constitutes a sub-network in the twin LSTM architecture network, and this sub-network constitutes the gait recognition model. The two fused feature vectors are input into the contrast loss objective function to obtain the loss value. This loss value is used to characterize the difference between the two fused feature vectors and the label. In the second and subsequent iterations, it is determined whether the loss value has decreased. If so, the training parameters of the Siamese LSTM architecture network are adjusted according to the loss value, and the gait image sequence training pair is input into the adjusted Siamese LSTM architecture network again until the loss value no longer decreases, and the training ends.

2. The method as described in claim 1, characterized in that, The periodic motion feature vector sets of the two gait image sequences are respectively input into two parallel feature fusion units, including: The periodic motion feature vector sets of the two gait image sequences are input into two parallel feature fusion units respectively, and the periodic motion feature vector sets are summed by temporal weighting, wherein the feature values ​​with earlier temporal order have less weight than the feature values ​​with later temporal order.

3. The method as described in claim 1, characterized in that, During training, 40% to 60% of the neuron outputs of the LSTM network are randomly discarded.

4. The method as described in claim 1, characterized in that, The LSTM network has 1 layer and 3096 or more hidden units.

5. The method as described in claim 1, characterized in that, The minimum distance interval between training pairs of gait image sequences that do not belong to the same identity and whose objective function is compared is greater than or equal to 0.5 and less than or equal to 1.

5.

6. The method as described in claim 1, characterized in that, The VGG network is a VGG16 network, and the output parameter of its fully connected fc8 layer is the number of gait image categories.

7. A gait recognition method, characterized in that, include: Obtain the gait image sequence to be identified; Inputting it into the trained Visual Geometry Group (VGG) network yields a set of spatial feature vectors for the gait image sequence; The spatial feature vector set of the gait image sequence is input into a trained Long Short-Term Memory (LSTM) network to obtain the periodic motion feature vector set of the gait image sequence. The periodic motion feature vector set is input into the feature fusion unit to obtain the feature vector of the fused gait image sequence used to realize gait recognition; The trained VGG network and the trained LSTM network are trained using the method described in any one of claims 1-6.

8. A gait recognition model training device, characterized in that, include: The sample acquisition unit is used to acquire gait image sequence training pairs and labels, wherein the labels are used to characterize whether the gait image sequence training pairs belong to the same identity; Two parallel Visual Geometry Group (VGG) network training units are used to receive two gait image sequences that constitute a training pair and obtain a set of spatial feature vectors for the two gait image sequences. Two parallel Long Short-Term Memory (LSTM) network training units are used to receive two spatial feature vector sets to obtain two periodic motion feature vector sets of gait image sequences. Among them, the VGG network + LSTM network + feature fusion unit constitute a sub-network in the twin LSTM architecture network, and this sub-network constitutes the gait recognition model. Two parallel feature fusion units are used to fuse the periodic motion feature vector sets of two gait image sequences respectively to obtain the feature vectors of the two fused gait image sequences. The loss calculation unit is used to input the two fused feature vectors into the contrast loss objective function to obtain the loss value, which is used to characterize the difference between the two fused feature vectors and the label. The judgment unit is used to determine whether the loss value has decreased in the second and subsequent iterations. If so, the parameter adjustment unit and the sample acquisition unit are triggered until the loss value no longer decreases. The parameter tuning unit is used to adjust the training parameters of the Siamese LSTM architecture network based on the loss value.

9. A gait recognition device, characterized in that, include: The image acquisition unit is used to acquire the gait image sequence to be identified; The trained Visual Geometry Group (VGG) network is used to extract the set of spatial feature vectors from the gait image sequence to be identified; A trained Long Short-Term Memory (LSTM) network is used to extract a set of periodic motion feature vectors from the spatial feature vector set of gait image sequences. The feature fusion unit is used to perform temporal fusion of the periodic motion feature vector set to obtain the fused feature vector for gait recognition. The trained VGG network and the trained LSTM network are obtained by training the gait recognition model training device described in claim 8.

Images