Cross-view gait recognition method based on metric learning and spatiotemporal dual-stream network
By constructing appearance motion two-flow sub network and feature mapper network, combining metric learning and joint loss function, the problems of time information loss and insufficient robustness in cross-view gait recognition are solved, and efficient spatio-temporal feature extraction and identification are achieved accurately Sex improvement.
Patent Information
- Application Number
- CN202210664898.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-06-13
- Publication Date
- 2024-11-26
- Estimated Expiration
- 2042-06-13
AI Technical Summary
The prior art loses time information and space-time feature learning ability in cross-view gait recognition, has large computing overhead, and is not robust under complex conditions, resulting in low recognition accuracy.
Using preprocessed gait profile sequence as input, an appearance motion bifollower network and feature mapper network are constructed. Through feature fusion and metric learning, a joint loss function is designed to extract robust features, reduce computational overhead and improve robustness. sex.
Effectively extracting rich spatiotemporal motion features of advanced semantics improves the learning ability of spatiotemporal feature and its robustness to noise, and improves the accuracy of cross-view gait recognition.
Smart Images

Figure CN114998995B_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the field of computer technology, and further relates to a cross-view gait recognition method based on metric learning and spatiotemporal dual-stream network in the field of computer vision recognition technology. The present invention can be used to extract rich spatiotemporal motion information from a human gait profile sequence to obtain a robust cross-view gait feature, and perform gait recognition based on the obtained cross-view gait feature to complete human identity authentication. Background Art
[0002] Gait recognition technology is a biometric technology that uses the unique information of a person's walking pattern and extracts the gait characteristics of the person when walking to identify the person. Gait recognition technology has a wide range of applications in crime prevention, forensic identification, and medical health due to its characteristics such as long-distance recognition, no need for subject cooperation, and difficulty in concealment. In actual scenarios, gait recognition will be affected by complex factors such as perspective changes, clothing conditions, and occlusion. Among them, the perspective change caused by cross-perspective is the most common problem, which will greatly affect the accuracy of gait recognition.
[0003] Fudan University has disclosed a cross-view gait recognition method based on a multi-task generative adversarial network in its patented technology "Cross-view gait recognition method based on a multi-task generative adversarial network" (application number: CN201710373017.5 authorization announcement number: CN107085716B). The specific implementation steps of this method are: for the original pedestrian video frame sequence, first pre-process each frame image and extract the gait template feature of the periodic energy graph; then encode it into a gait hidden representation through a neural network and perform angle transformation in the latent space; then reconstruct the gait template features of other perspectives through a multi-task generative adversarial network; finally, use the gait hidden representation for recognition. However, this method still has two shortcomings. First, since this method uses the periodic energy graph as a gait template, part of the time information in the gait profile sequence will be lost, and it is impossible to obtain accurate spatiotemporal motion features representing gait information. Secondly, this method cannot solve the problems of clothing changes and occlusion under complex conditions and resist the influence of environmental interference factors, resulting in low gait recognition performance under complex conditions, and it is difficult to meet the gait recognition scenarios with complex conditions in actual scenarios.
[0004] Beijing Jiaotong University disclosed a gait recognition method based on attention mechanism in its patent document "A neural network gait recognition method based on attention mechanism" (patent application number: CN202010778327.7 authorization announcement number: CN111967358B). This method first constructs the training set and test set of the neural network, and through the iterative optimization strategy, uses samples and sample labels to pre-train the gait feature extraction model, so that the trained gait feature extraction model can extract better features from the human gait; then, based on the previous stage, the time domain attention mechanism module and the spatial domain attention mechanism module are embedded in the network, and the network is retrained; finally, the trained network model is used for gait recognition. However, there are still two shortcomings in this method. First, this method requires a complex attention calculation process, and the calculation overhead and video memory usage are relatively large, which is not conducive to deployment and use in actual scenarios. Secondly, because the change in perspective causes the differences between samples of the same category to increase, making recognition more difficult, this method lacks robustness to perspective changes and has low cross-perspective recognition accuracy. It also does not fully exploit the gait motion information in the time dimension of the video sequence, and has insufficient spatiotemporal feature learning capabilities. Summary of the invention
[0005] The purpose of the present invention is to address the deficiencies of the above-mentioned prior art and propose a cross-view gait recognition method based on metric learning and spatiotemporal dual-stream network, aiming to solve the problems of loss of part of the time information by the gait template method and insufficient spatiotemporal feature learning ability of the existing method, as well as the low accuracy of cross-view gait recognition due to high computational overhead and insufficient robustness under complex conditions.
[0006] The idea of achieving the purpose of the present invention is that the present invention uses the preprocessed gait profile sequence as input to save the original time information, thereby solving the problem of losing part of the time information in the gait template method of the prior art. The present invention forms a deep learning network with the constructed appearance motion two-stream sub-network and feature mapping sub-network, wherein the appearance motion two-stream sub-network extracts appearance and motion features, and captures the relationship between appearance and motion features by feature fusion, thereby solving the problem of insufficient spatiotemporal feature learning ability of the prior art; the feature mapping sub-network refines and compresses the input features, learns the human body part features of different granularities, and does not require a complex attention calculation process, thereby solving the problem of large computational overhead and large video memory usage, as well as insufficient robustness to noise in the prior art. The present invention adopts the idea of metric learning in training the deep learning network, designs a joint loss function including a triple loss function and a multi-classification cross entropy loss function, and learns more robust features, thereby solving the problem of insufficient robustness of feature extraction under perspective changes in the prior art. Finally, the trained deep learning network is used for cross-perspective gait recognition.
[0007] The technical solutions for achieving the purpose of the present invention include the following:
[0008] Step 1: Generate training set:
[0009] Step 1.1, select 8140 gait profile sequences of at least 74 subjects;
[0010] Step 1.2, preprocessing each gait profile sequence to obtain an aligned gait profile sequence in which the pedestrian's head is located at the upper edge of the image and the foot is located at the lower edge of the image; scaling the aligned gait profile sequence to the same image size of 64×64;
[0011] Step 1.3, all data samples are combined into a training set;
[0012] Step 2: Construct the appearance-motion two-stream subnetwork:
[0013] Step 2.1, construct the appearance flow feature module, whose structure is: head, first appearance convolution block, second appearance convolution block, third appearance convolution block, fourth appearance convolution block. The structures of the first to fourth appearance convolution blocks are the same, and each appearance convolution block consists of a backbone branch and a skip connection branch;
[0014] The head is composed of a convolutional layer and a pooling layer in series;
[0015] The backbone branch of the appearance convolution block is composed of the first convolution layer, the first LeakyReLU activation layer, the second convolution layer, the second LeakyReLU activation layer, and the third convolution layer in series. The skip connection branch of the appearance convolution block uses a fourth convolution layer so that the number of channels of the output features of the skip connection is consistent with the backbone branch. The backbone branch and the skip connection branch of the appearance convolution block are connected through a position-by-position adder and then connected to the third LeakyReLU activation layer;
[0016] Set the convolution kernel size of the convolution layer in the head to 1×3×3, the number of convolution kernel channels to 1, and the number of convolution kernels to 32. Set the pooling layer in the head to the maximum pooling layer, and the size of the pooling kernel to 1×3×3;
[0017] The convolution kernel size of the first convolution layer in the first to fourth appearance convolution blocks is set to 1×1×1, and the convolution kernel sizes of the second to fourth convolution layers are set to 1×3×3, 1×1×1, and 1×1×1, respectively. The number of channels of the convolution kernels of the first to fourth convolutional layers are set to 32, 32, 32, 32, and the number of convolution kernels are set to 32, 32, 128, and 128, respectively; in the second appearance convolution block, the number of channels of the convolution kernels of the first to fourth convolutional layers are set to 128, 32, 32, and 128, respectively, and the number of convolution kernels are set to 32, 32, 128, and 128, respectively; in the third appearance convolution block, the number of channels of the convolution kernels of the first to fourth convolutional layers are set to 128, 64, 64, and 128, respectively, and the number of convolution kernels are set to 64, 64, 256, and 256, respectively; in the fourth appearance convolution block, the number of channels of the convolution kernels of the first to fourth convolutional layers are set to 256, 64, 64, and 256, respectively, and the number of convolution kernels are set to 64, 64, 256, and 256, respectively;
[0018] Step 2.2, construct the motion flow feature module structure in order: head, first motion convolution block, second motion convolution block, third motion convolution block, fourth motion convolution block. The structures of the first to fourth motion convolution blocks are the same, and each motion convolution block consists of a backbone branch and a skip connection branch;
[0019] The head is composed of a convolutional layer and a pooling layer in series;
[0020] The structure of the motion convolution block is consistent with the structure of the appearance convolution block in step 2.1;
[0021] Set the convolution kernel size of the convolution layer in the head to 3×3×3, the number of convolution kernel channels to 1, and the number of convolution kernels to 32. Set the pooling layer in the head to the maximum pooling layer, and the size of the pooling kernel to 1×3×3;
[0022] The convolution kernel size of the first convolution layer in the first to fourth motion convolution blocks is set to 3×1×1, and the convolution kernel sizes of the second to fourth convolution layers are set to 1×3×3, 1×1×1, and 1×1×1, respectively. In the first motion convolution block, the number of channels of the convolution kernels of the first to fourth convolution layers are 32, 32, 32, 32, and the number of convolution kernels are 32, 32, 128, and 128 respectively; in the second motion convolution block, the number of channels of the convolution kernels of the first to fourth convolution layers are 128, 32, 32, 128, and the number of convolution kernels are 32, 32, 128, and 128 respectively; in the third motion convolution block, the number of channels of the convolution kernels of the first to fourth convolution layers are 128, 64, 64, 128, and the number of convolution kernels are 64, 64, 256, and 256 respectively; in the fourth motion convolution block, the number of channels of the convolution kernels of the first to fourth convolution layers are 256, 64, 64, 256, and the number of convolution kernels are 64, 64, 256, and 256 respectively;
[0023] Step 2.3, construct the first to fourth feature fusion modules. The structures of the first to third feature fusion modules are the same, and each feature fusion module consists of a convolutional layer and a feature concatenation block;
[0024] The feature splicing block is composed of feature splicing operations, which splice the input appearance features and motion features to obtain richer gait features;
[0025] The convolution kernel size of the convolution layers in the first to third feature fusion modules is set to 1×1×1, the number of convolution kernel channels is set to 32, 128, and 256, and the number of convolution kernels is set to 32, 128, and 256, respectively;
[0026] The feature concatenation layers in the first to third feature fusion modules are respectively set to concatenate features by channel, by channel, and by height dimension;
[0027] Step 2.4, constructing an appearance flow feature module and a motion flow feature module in parallel, and using the first to third feature fusion modules to connect the appearance flow feature module and the motion flow module to obtain an appearance-motion two-flow sub-network;
[0028] The convolution layer in the first feature fusion module is connected to the pooling layer of the head in the appearance flow feature module, and the output features of the convolution layer are spliced with the output features of the pooling layer of the head in the motion flow feature module according to the channel dimension, and then connected to the first motion convolution block in the appearance flow feature module; the convolution layer in the second feature fusion module is connected to the second appearance convolution block in the appearance flow feature module, and the output features of the convolution layer are spliced with the output features of the second motion convolution block in the motion flow feature module according to the channel dimension, and then connected to the third motion convolution block in the appearance flow feature module; the convolution layer in the third feature fusion module is connected to the fourth appearance convolution block in the appearance flow feature module, and the output features of the convolution layer are spliced with the output features of the fourth motion convolution block in the motion flow feature module according to the channel dimension to obtain the final output features, whose channel, height, and width are 256, 64, and 22 respectively; the construction of the appearance motion two-stream sub-network is completed;
[0029] Step 3: Construct the feature mapping subnetwork:
[0030] Step 3.1, construct a time domain pooling module, whose structure consists of a maximum pooling layer in the time dimension;
[0031] The pooling kernel size of the maximum pooling layer in the temporal pooling module is set to T1×1×1, where T1 is the length of the input gait profile sequence;
[0032] Step 3.2, construct a horizontal segmentation pyramid module, using 6 scales, and its structure is composed of the first scale horizontal segmentation block, the second scale horizontal segmentation block, the third scale horizontal segmentation block, the fourth scale horizontal segmentation block, the fifth scale horizontal segmentation block, and the sixth scale horizontal segmentation block in parallel. The structures of the first to sixth scale horizontal segmentation blocks are the same, and each scale horizontal segmentation block is composed of a feature horizontal segmentation layer;
[0033] The feature horizontal segmentation layer consists of feature segmentation operations in the height dimension, which divides the input features into multiple strips;
[0034] The number of strips in the first to sixth horizontal segmentation layers is set to 1, 2, 4, 8, 16, and 32 respectively, the height of the strips is set to 64, 32, 16, 8, 4, and 2 respectively, and the width is set to 22, so a total of 63 strips are obtained, each strip represents a feature of each human body part;
[0035] Step 3.3, constructing a generalized average pooling module structure is as follows: the first generalized pooling block, the second generalized pooling block, the third generalized pooling block, the fourth generalized pooling block, the fifth generalized pooling block, and the sixth generalized pooling block are connected in parallel;
[0036] The generalized pooling block consists of a first power operation on each strip, an average pooling layer, and a second power operation on the feature in series;
[0037] The exponents of the first and second power operations in the first to sixth generalized pooling blocks are set to the learnable parameter p. The first power operation is to raise the input feature to the power of p, and the second power operation is to raise the input feature to the power of The power, p, is set to 6.5 by default and can be learned and changed during training using the back-propagation algorithm. The height of the pooling kernel size of the average pooling layer in the first to sixth generalized pooling blocks is set to 64, 32, 16, 8, 4, 2, and the width is set to 22;
[0038] In step 3.4, the structure of the feature projection module is constructed as follows: independent fully connected block, batch normalization layer.
[0039] The independent fully connected block is composed of 63 parallel fully connected layers with different parameters;
[0040] The parameters of the 63 fully connected layers in the independent fully connected blocks are initialized using the normal distribution N(0,0.01) by default. The batch normalization layer is set to batch normalize the one-dimensional feature vector, and the number of channels is set to 256;
[0041] Step 3.5, connect the time domain pooling module, the horizontal segmentation pyramid module, the generalized average pooling module, and the feature projection module in series to form a feature mapping subnetwork;
[0042] Step 4, connect the appearance motion two-stream sub-network and the feature mapping sub-network in series to form a deep learning network;
[0043] Step 5, generate the joint loss function as follows:
[0044] L total =αL triplet +βL ce +λL weight
[0045] Among them, L total represents the joint loss function, L triplet represents the triple loss function, L ce represents the multi-classification cross entropy loss function, L weight represents the weight decay loss function, α represents the coefficient of the triplet loss function, its value is 1, β represents the coefficient of the multi-classification loss function, its value is 0.1, and λ represents the coefficient of the weight decay loss function, its value is 0.00005;
[0046] Step 6: Train the deep learning network:
[0047] Input the training set into the deep learning network. Use the joint loss function to calculate the overall loss value. Back-propagate the overall loss value in the deep learning network, and use the gradient descent method to update the weight parameters of each network node until the loss value converges to obtain a trained deep learning network.
[0048] Step 7: Cross-view gait recognition:
[0049] Step 7.1, send the samples in the registered data set to the trained deep learning network to obtain a one-dimensional vector output as the feature vector representation of each gait profile sequence, and finally obtain the feature database of the registered data set;
[0050] Step 7.2: Send the samples in the query data set to be identified into the trained deep learning network to obtain the feature vectors of the samples in the query data set. Perform Euclidean distance calculation on the feature vectors of each sample in the query data set and the feature vectors of all samples in the registration data set, and finally identify the query sample as the identity label of the feature vector in the registration data set with the smallest Euclidean distance, output the identity label of the query sample, and complete cross-view recognition.
[0051] Compared with the prior art, the present invention has the following advantages:
[0052] First, the present invention overcomes the problem of losing time information and the insufficient learning ability of spatiotemporal features in the prior art by constructing an appearance-motion two-stream sub-network and taking the original gait contour sequence as the input of the network, so that the present invention can extract rich spatiotemporal motion features containing high-level semantics. Compared with the existing methods, the present invention has the advantages of preserving time information and improving the learning ability of spatiotemporal features.
[0053] Second, the feature mapping subnetwork constructed by the present invention overcomes the defects of the attention calculation process in the prior art, such as the large computational overhead and large memory usage, as well as the insufficient robustness to noise, making the present invention advantageous for deployment and use in actual situations and improving the robustness to noise under complex conditions.
[0054] Third, the present invention constructs a joint loss function for training deep learning networks, which overcomes the shortcomings of the features extracted by the prior art that are not robust enough under changes in viewing angles, so that the present invention can learn more general and identifiable features, making the present invention have the advantages of robustness to changes in viewing angles and improving the overall recognition accuracy. BRIEF DESCRIPTION OF THE DRAWINGS
[0055] Figure 1 It is a flow chart of the implementation of the present invention;
[0056] Figure 2 is a structural diagram of the appearance convolution block and the motion convolution block in the present invention;
[0057] Figure 3 is a structural diagram of the appearance motion two-flow subnetwork in the present invention;
[0058] Figure 4 It is a structural diagram of the feature mapping subnetwork in the present invention;
[0059] Figure 5 It is a structural diagram of the deep learning network constructed by the present invention. DETAILED DESCRIPTION
[0060] The present invention is further described below in conjunction with the accompanying drawings and embodiments.
[0061] Reference Figure 1 The specific steps of implementing the present invention are further described with reference to the accompanying drawings and embodiments.
[0062] Step 1: Generate a training set.
[0063] Step 1.1, the data set of the present invention is to apply for the CASIA-B data set on the Internet by email to the Institute of Automation of the Chinese Academy of Sciences. This data set covers a comprehensive perspective, a widely used gait data set, and contains 13,640 videos of 124 subjects. Each subject has 10 types of gait profile sequences, including 6 types collected under normal walking conditions (indexed as NM#01-06), 2 types collected under the condition of carrying a backpack (indexed as BG#01-02), and 2 types collected under the condition of wearing a coat (indexed as CL#01-02). Each type includes 11 gait profile sequences with different perspectives (0°-180° intervals of 18°).
[0064] Step 1.2, each gait profile sequence (each gait profile sequence is a gait profile video, including multiple / multi-frame gait profile images) is processed as follows:
[0065] Step 1.2.1, read each gait profile sequence, place the part including the pedestrian at the center of each frame image, and crop and align the image to remove useless black background information, so that the pedestrian's head is located at the upper edge of the image and the foot is located at the lower edge of the image, and obtain the aligned gait profile sequence;
[0066] In step 1.2.2, the aligned gait contour sequence is adjusted to the same image size of 64×64 to obtain the final processed gait contour sequence as the input of the appearance-motion two-stream sub-network.
[0067] Step 1.3, a total of 8140 gait profile sequences of the first 74 subjects were used as training sets. All gait profile sequences of the remaining 50 subjects were used as test sets, and the first four types (i.e., NM#01-04) of the six types of gait profile sequences sampled under normal conditions were used as registration data sets, and the remaining NM#05-06, BG#01-02, and CL#01-02 were used as query data sets.
[0068] Step 2: Construct the appearance-motion two-stream subnetwork.
[0069] Reference Figure 3 , the present invention is further described in detail to construct the appearance motion two-stream subnetwork.
[0070] Step 2.1, construct the appearance flow feature module, whose structure is: head, first appearance convolution block, second appearance convolution block, third appearance convolution block, fourth appearance convolution block. The structures of the first to fourth appearance convolution blocks are the same, and each appearance convolution block consists of a backbone branch and a skip connection branch.
[0071] The head is composed of a convolutional layer and a pooling layer in series.
[0072] The backbone branch of the appearance convolution block is composed of the first convolution layer, the first LeakyReLU activation layer, the second convolution layer, the second LeakyReLU activation layer, and the third convolution layer in series. The skip connection branch of the appearance convolution block uses a fourth convolution layer so that the number of channels of the output features of the skip connection is consistent with the backbone branch. The backbone branch and the skip connection branch of the appearance convolution block are connected through a position-by-position adder and then connected to the third LeakyReLU activation layer. Figure 2 (a) shown.
[0073] The convolution kernel size of the convolution layer in the head is set to 1×3×3, the number of convolution kernel channels is set to 1, and the number of convolution kernels is set to 32. The pooling layer in the head is set to the maximum pooling layer, and the size of the pooling kernel is set to 1×3×3.
[0074] The convolution kernel size of the first convolution layer in the first to fourth appearance convolution blocks is set to 1×1×1, and the convolution kernel sizes of the second to fourth convolution layers are set to 1×3×3, 1×1×1, and 1×1×1, respectively. The number of channels of the convolution kernels of the first to fourth convolutional layers are set to 32, 32, 32, 32, respectively, and the number of convolution kernels is set to 32, 32, 128, 128, respectively; in the second appearance convolution block, the number of channels of the convolution kernels of the first to fourth convolutional layers are set to 128, 32, 32, 128, respectively, and the number of convolution kernels is set to 32, 32, 128, 128, respectively; in the third appearance convolution block, the number of channels of the convolution kernels of the first to fourth convolutional layers are set to 128, 64, 64, 128, respectively, and the number of convolution kernels is set to 64, 64, 256, 256, respectively; in the fourth appearance convolution block, the number of channels of the convolution kernels of the first to fourth convolutional layers are set to 256, 64, 64, 256, respectively, and the number of convolution kernels is set to 64, 64, 256, 256, respectively.
[0075] Step 2.2, construct the motion flow feature module structure in order: head, first motion convolution block, second motion convolution block, third motion convolution block, fourth motion convolution block. The structures of the first to fourth motion convolution blocks are the same, and each motion convolution block consists of a backbone branch and a skip connection branch.
[0076] The head is composed of a convolutional layer and a pooling layer in series.
[0077] The structure of the motion convolution block is consistent with the structure of the appearance convolution block in step 2.1, as shown in Figure 2 (b) as shown.
[0078] The convolution kernel size of the convolution layer in the head is set to 3×3×3, the number of convolution kernel channels is set to 1, and the number of convolution kernels is set to 32. The pooling layer in the head is set to the maximum pooling layer, and the size of the pooling kernel is set to 1×3×3.
[0079] The convolution kernel size of the first convolution layer in the first to fourth motion convolution blocks is set to 3×1×1, and the convolution kernel sizes of the second to fourth convolution layers are set to 1×3×3, 1×1×1, and 1×1×1, respectively. In the first motion convolution block, the number of channels of the convolution kernels of the first to fourth convolution layers are 32, 32, 32, 32, and the number of convolution kernels are 32, 32, 128, and 128 respectively; in the second motion convolution block, the number of channels of the convolution kernels of the first to fourth convolution layers are 128, 32, 32, 128, and the number of convolution kernels are 32, 32, 128, and 128 respectively; in the third motion convolution block, the number of channels of the convolution kernels of the first to fourth convolution layers are 128, 64, 64, 128, and the number of convolution kernels are 64, 64, 256, and 256 respectively; in the fourth motion convolution block, the number of channels of the convolution kernels of the first to fourth convolution layers are 256, 64, 64, 256, and the number of convolution kernels are 64, 64, 256, and 256 respectively.
[0080] Step 2.3, construct the first to fourth feature fusion modules. The structures of the first to third feature fusion modules are the same, and each feature fusion module consists of a convolutional layer and a feature concatenation block.
[0081] The feature splicing block is composed of feature splicing operations, which splice the input appearance features and motion features and fuse them to obtain richer gait features.
[0082] The convolution kernel sizes of the convolution layers in the first to third feature fusion modules are all set to 1×1×1, the number of convolution kernel channels is set to 32, 128, and 256, respectively, and the number of convolution kernels is set to 32, 128, and 256, respectively.
[0083] The feature splicing layers in the first to third feature fusion modules are respectively set to splice features by channel, by channel, and by height dimension.
[0084] Step 2.4, construct an appearance flow feature module and a motion flow feature module in parallel, and use the first to third feature fusion modules to connect the appearance flow feature module and the motion flow module to obtain an appearance-motion two-flow sub-network.
[0085] The convolution layer in the first feature fusion module is connected to the pooling layer of the head in the appearance flow feature module. The output features of the convolution layer are spliced with the output features of the pooling layer of the head in the motion flow feature module according to the channel dimension, and then connected to the first motion convolution block in the appearance flow feature module; the convolution layer in the second feature fusion module is connected to the second appearance convolution block in the appearance flow feature module, the output features of the convolution layer are spliced with the output features of the second motion convolution block in the motion flow feature module according to the channel dimension, and then connected to the third motion convolution block in the appearance flow feature module; the convolution layer in the third feature fusion module is connected to the fourth appearance convolution block in the appearance flow feature module, the output features of the convolution layer are spliced with the output features of the fourth motion convolution block in the motion flow feature module according to the channel dimension to obtain the final output features, whose channel, height, and width are 256, 64, and 22 respectively. The construction of the appearance motion two-stream sub-network is completed.
[0086] Step 3: Construct the feature mapping subnetwork.
[0087] Reference Figure 3 , the feature mapping subnetwork constructed in the present invention is further described in detail.
[0088] Step 3.1, construct a time domain pooling module, whose structure consists of a maximum pooling layer in the time dimension.
[0089] The pooling kernel size of the maximum pooling layer in the temporal pooling module is set to T1×1×1, where T1 is the length of the input gait profile sequence.
[0090] Step 3.2, construct a horizontal segmentation pyramid module, using 6 scales, and its structure is composed of the first scale horizontal segmentation block, the second scale horizontal segmentation block, the third scale horizontal segmentation block, the fourth scale horizontal segmentation block, the fifth scale horizontal segmentation block, and the sixth scale horizontal segmentation block in parallel. The structures of the first to sixth scale horizontal segmentation blocks are the same, and each scale horizontal segmentation block consists of a feature horizontal segmentation layer.
[0091] The feature level segmentation layer consists of feature segmentation operations in the height dimension, which divides the input features into multiple strips.
[0092] The number of strips in the first to sixth horizontal segmentation layers is set to 1, 2, 4, 8, 16, and 32, respectively, the height of the strips is set to 64, 32, 16, 8, 4, and 2, respectively, and the width is set to 22, resulting in a total of 63 strips, each of which represents a feature of each human body part.
[0093] Step 3.3, constructing the generalized average pooling module structure is: the first generalized pooling block, the second generalized pooling block, the third generalized pooling block, the fourth generalized pooling block, the fifth generalized pooling block, and the sixth generalized pooling block are connected in parallel.
[0094] The generalized pooling block consists of a first power operation on each strip, an average pooling layer, and a second power operation on the features in series.
[0095] The exponents of the first and second power operations in the first to sixth generalized pooling blocks are set to the learnable parameter p, and the input features are raised to the power of p. The default value of p is 6.5, which can be learned and changed during training using the back propagation algorithm. The height of the pooling kernel size of the average pooling layer in the first to sixth generalized pooling blocks is set to 64, 32, 16, 8, 4, 2, and the width is set to 22.
[0096] In step 3.4, the structure of the feature projection module is constructed as follows: independent fully connected block, batch normalization layer.
[0097] The independent fully connected block is composed of 63 parallel fully connected layers with different parameters.
[0098] The parameters of the 63 fully connected layers in the independent fully connected blocks are initialized with the default normal distribution N(0,0.01). The batch normalization layer is set to batch normalize the one-dimensional feature vector, and the number of channels is set to 256.
[0099] Step 3.5, connect the time domain pooling module, the horizontal segmentation pyramid module, the generalized average pooling module, and the feature projection module in series to form a feature mapping subnetwork, such as Figure 4 shown.
[0100] Step 4: Connect the appearance-motion two-stream sub-network and the feature mapping sub-network in series to form a deep learning network, such as Figure 5 shown.
[0101] Step 5, generate the joint loss function as follows:
[0102] L total =αL triplet +βL ce +λL weight
[0103] Among them, L total represents the joint loss function, L triplet represents the triple loss function, L ce represents the multi-classification cross entropy loss function, L weight represents the weight decay loss function, α represents the coefficient of the triplet loss function, its value is 1, β represents the coefficient of the multi-classification loss function, its value is 0.1, and λ represents the coefficient of the weight decay loss function, its value is 0.00005;
[0104] The triplet loss function is as follows:
[0105]
[0106] in, represents the total number of non-zero items of triplet loss in a batch, parts represents the total number of partial features at all scales, S represents the total number of multi-scales in the horizontal segmentation pyramid module, s represents the index of the scale, t represents the index of the partial feature, anchors represents the set of anchor samples (indicates that when the sample is first selected in the triplet sample pair, it is used as the anchor point, the positive sample and the anchor sample have the same label, and the negative sample and the anchor sample have different labels), H, M represent the number of subjects in a batch and the number of gait profile sequences of each subject, h, m represent the subject index and the index of the gait profile sequence, pos represents the set of positive samples with the same label as the anchor sample, a represents the index of the positive sample, negative represents the set of negative samples with different labels from the anchor sample, b, c represent the index of the subject in the negative sample set and the index of the gait profile sequence, V2 represents the feature vector of the sample obtained after the gait profile sequence is input into the deep learning network, ω is the boundary threshold margin of the triplet loss, dist represents the distance between the positive sample pair and the negative sample pair, [x] + represents the maximum value of input x and 0, D(·) represents the Euclidean distance function, represents the t-th partial feature of the s-th scale of the m-th gait profile sequence of the h-th subject, and similar.
[0107] The number of multi-scales S in the triplet loss function is set to 6, H and M are set to 12 and 4 respectively, ω is set to 0.2, and D is set to the Euclidean distance calculation function.
[0108] The multi-classification cross entropy loss function is as follows:
[0109]
[0110] The identity labels of the subjects in the data set are used as the label information of the classification task. The feature vector V2 is processed using an independent fully connected layer and a Softmax function to obtain the category prediction probability distribution p, and then the multi-classification cross loss is calculated.
[0111] Where N is the number of all subject categories in the training set, represents the predicted probability that the m-th gait profile sequence of h subjects belongs to the n-th subject category, Indicates the true probability that the mth gait profile sequence of h subjects belongs to the nth subject category (taking the φth subject as an example). In the test phase, the classification probability results are discarded, and only the one-dimensional vector expanded by the feature vector matrix V2 is used as the feature vector representation of the final profile sequence.
[0112] The N in the multi-class cross entropy loss function is set to 74.
[0113] The weight decay loss function is as follows;
[0114]
[0115] Where Z represents the total number of parameters in the deep learning network, i represents the index of the parameter in the deep learning network, and W i represents the i-th parameter in the deep learning network.
[0116] Set Z in the weight decay loss function to 5864580.
[0117] Step 6: Train the deep learning network:
[0118] Input the training set into the deep learning network. Use the joint loss function to calculate the overall loss value. Back-propagate the overall loss value in the deep learning network, and use the gradient descent method to update the weight parameters of each network node until the loss value converges to obtain a trained deep learning network.
[0119] Step 7: cross-view gait recognition.
[0120] In step 7.1, the samples in the registered data set are sent to the trained deep learning network to obtain a one-dimensional vector output as the feature vector representation of each gait profile sequence, and finally the feature database of the registered data set is obtained.
[0121] Step 7.2: Send the samples in the query data set to be identified into the trained deep learning network to obtain the feature vectors of the samples in the query data set. Perform Euclidean distance calculation on the feature vectors of each sample in the query data set and the feature vectors of all samples in the registration data set, and finally identify the query sample as the identity label of the feature vector in the registration data set with the smallest Euclidean distance, output the identity label of the query sample, and complete cross-view recognition.
[0122] The effect of the present invention is further illustrated by the following simulation experiment.
[0123] 1. Simulation experiment conditions:
[0124] The hardware platform of the simulation experiment of the present invention is: the processor is Intel i7-8700K CPU, the main frequency is 3.70GHz, the memory is 64G, the graphics card is NVIDIA GEFORCE GTX 1080Ti, and the video memory is 11GB.
[0125] The software platform of the simulation experiment of the present invention is: Ubuntu operating system and python 3.7.
[0126] 2. Simulation experiment content and result analysis:
[0127] The simulation experiment of the present invention adopts the method of the present invention and three cross-view recognition methods of GaitSe, Gaitpart and MT3D in the prior art to perform cross-view gait recognition on each sample in the CASIA-B data set in step 1 of the specific implementation method of the present invention.
[0128] In the simulation experiment of the present invention, the GaitSet cross-view recognition method used refers to:
[0129] Chao H, He Y, Zhang J et al. proposed a gait set-based method in their paper “Gaitset: Regarding gait as a set for cross-view gait recognition” ([C] / / Proceedings of the AAAI conference on artificial intelligence. 2019, 33(01):8126-8133).
[0130] In the simulation experiment of the present invention, the Gaitpart cross-view recognition method used refers to:
[0131] Fan C, Peng Y, Cao C et al. proposed a method based on human body part features in their paper "Gaitpart: Temporal part-based model for gait recognition" ([C] / / Proceedings of the IEEE / CVF conference on computer vision and pattern recognition.2020:14225-14233).
[0132] In the simulation experiment of the present invention, the MT3D cross-view recognition method used refers to:
[0133] Lin B, Zhang S, Bao F proposed a method based on 3D convolutional neural network in their published paper "Gait recognition with multiple-temporal-scale 3D convolutional neural network" ([C] / / Proceedings of the 28thACM International conference on Multimedia.2020:3054-3062).
[0134] The steps of cross-view gait recognition in the simulation experiment of the present invention are as follows: first, gait profile preprocessing is performed on CASIA-B data, gait profile sequences of the first 74 subjects are selected to generate a training set, and the remaining 50 subjects are selected to generate a test set. Second, the training set data is input into a deep learning network for training.
[0135] The third step is to perform cross-view gait recognition on the test set after training.
[0136] In order to verify the effect of the simulation experiment of the present invention, the cross-viewing accuracy rate (%) of the deep learning network of the present invention and the three cross-viewing recognition methods of the prior art GaitSe, Gaitpart and MT3D are obtained according to the cross-viewing accuracy calculation formula.
[0137] In the simulation experiment, the steps for calculating the cross-view accuracy are as follows:
[0138] The first step is to select a query perspective from the perspective set A = {0°, 18°, 36°, 54°, 72°, 90°, 108°, 126°, 144°, 162°, 180°} and registration perspective Select the query collection Samples to be identified under viewing angle And in the registration collection Samples in perspective Where N p for Number of query samples under perspective, N g for The number of registered samples under the viewing angle.
[0139] The second step is A sample to be identified in the collection Calculate the sum of the registered sets The similarity of all samples in Where D(·) is the Euclidean distance calculation function, find and The registered sample with the smallest sample distance.
[0140] Step 3: If the two identities are the same, the number of successful identifications is N. correct Add one, otherwise the recognition fails.
[0141] Step 4: Repeat the second to third steps for the remaining samples to be identified under the viewing angle.
[0142] Step 5: Get Corresponds to Cross-view recognition accuracy of perspective
[0143] Step 6: Repeat steps 1 to 5 to obtain the cross-view recognition accuracy of 11 query views corresponding to 11 registered views.
[0144] In the seventh step, the cross-view average accuracy under each query perspective is further calculated, and the cross-view accuracy of the remaining 10 perspectives except the current perspective is summed and averaged to obtain the cross-view average accuracy of the current query perspective.
[0145] In the eighth step, steps one to seven are repeated to obtain the cross-view recognition accuracy of 11 viewing angles under normal walking conditions, carrying a backpack, and wearing a coat.
[0146] According to the above cross-view accuracy calculation steps, the cross-view recognition accuracy results (%) of the three cross-view recognition methods of the present invention and the prior art GaitSe, Gaitpart, and MT3D under 11 viewpoints under three different walking conditions in the simulation experiment of the present invention are obtained, as shown in Figure 1.
[0147] Table 1 Results of cross-view recognition accuracy of the present invention and prior art in simulation experiments
[0148]
[0149] Combined with Table 1, the higher the cross-view accuracy, the better the performance of the method. The method of the present invention has achieved the best recognition performance under all walking conditions. Under normal walking conditions, the cross-view recognition accuracy of the method of the present invention in the remaining 10 viewing angles except the 36° viewing angle is better than other advanced methods, and the average accuracy rate reaches 97.2%; under the condition of walking with a backpack, the method of the present invention and MT3D have their own advantages in the cross-view recognition accuracy under 11 viewing angles, but the average accuracy rate of the method of the present invention is better; under the condition of walking in a coat, there is a large occlusion problem under this condition, which causes the recognition accuracy rate to drop rapidly, but the method of the present invention is better than other advanced methods in the cross-view recognition accuracy rate under the remaining 10 viewing angles except the 36° viewing angle, and the average accuracy rate reaches 83.9%, which has a significant advantage. Therefore, compared with the existing methods, the recognition performance under complex conditions and cross-view recognition accuracy are significantly improved.
Claims
1. A cross-view gait recognition method based on metric learning and spatiotemporal dual-stream network, characterized in that: A deep learning network consisting of an appearance-motion two-stream subnetwork and a feature mapping subnetwork is constructed to generate a joint loss function consisting of a triplet loss function and a multi-classification cross entropy loss. The specific steps of the recognition method include the following: Step 1: Generate training set: Step 1.1, select 8140 gait profile sequences of at least 74 subjects; Step 1.2, preprocessing each gait profile sequence to obtain an aligned gait profile sequence in which the pedestrian's head is located at the upper edge of the image and the foot is located at the lower edge of the image; scaling the aligned gait profile sequence to the same image size of 64×64; The preprocessing refers to reading each gait profile sequence, placing the part including the pedestrian at the center of each frame image, and cropping and aligning the image to remove useless black background information so that the pedestrian's head is located at the upper edge of the image and the foot is located at the lower edge of the image, thereby obtaining an aligned gait profile sequence; Step 1.3, all data samples are combined into a training set; Step 2: Construct the appearance-motion two-stream subnetwork: Step 2.1, construct an appearance flow feature module, whose structure is: head, first appearance convolution block, second appearance convolution block, third appearance convolution block, fourth appearance convolution block; the structures of the first to fourth appearance convolution blocks are the same, and each appearance convolution block consists of a backbone branch and a skip connection branch; The head is composed of a convolutional layer and a pooling layer in series; The backbone branch of the appearance convolution block is composed of a first convolution layer, a first LeakyReLU activation layer, a second convolution layer, a second LeakyReLU activation layer, and a third convolution layer connected in series; the skip connection branch of the appearance convolution block adopts a fourth convolution layer, so that the number of channels of the output feature of the skip connection is consistent with the backbone branch; the backbone branch and the skip connection branch of the appearance convolution block are connected through a position-by-position adder and then connected to the third LeakyReLU activation layer; The convolution kernel size of the convolution layer in the head is set to 1×3×3, the number of convolution kernel channels is set to 1, and the number of convolution kernels is set to 32; the pooling layer in the head is set to the maximum pooling layer, and the size of the pooling kernel is set to 1×3×3; The convolution kernel size of the first convolution layer in the first to fourth appearance convolution blocks is set to 1×1×1, and the convolution kernel sizes of the second to fourth convolution layers are set to 1×3×3, 1×1×1, and 1×1×1, respectively; the number of channels of the convolution kernel of the first to fourth convolution layers is set to 32, 32, 32, and 32, respectively, and the number of convolution kernels is set to 32, 32, 128, and 128, respectively; in the second appearance convolution block, the number of channels of the convolution kernel of the first to fourth convolution layers is set to 128, 32, 3 2, 128, the number of convolution kernels is set to 32, 32, 128, 128 respectively; in the third appearance convolution block, the number of channels of the convolution kernels of the first to fourth convolution layers are set to 128, 64, 64, 128 respectively, and the number of convolution kernels is set to 64, 64, 256, 256 respectively; in the fourth appearance convolution block, the number of channels of the convolution kernels of the first to fourth convolution layers are set to 256, 64, 64, 256 respectively, and the number of convolution kernels is set to 64, 64, 256, 256 respectively; Step 2.2, construct the motion flow feature module structure in order: head, first motion convolution block, second motion convolution block, third motion convolution block, fourth motion convolution block; the structures of the first to fourth motion convolution blocks are the same, and each motion convolution block consists of a backbone branch and a skip connection branch; The head is composed of a convolutional layer and a pooling layer in series; The structure of the motion convolution block is consistent with the structure of the appearance convolution block in step 2.1; The convolution kernel size of the convolution layer in the head is set to 3×3×3, the number of convolution kernel channels is set to 1, and the number of convolution kernels is set to 32; the pooling layer in the head is set to the maximum pooling layer, and the size of the pooling kernel is set to 1×3×3; The convolution kernel size of the first convolution layer in the first to fourth motion convolution blocks is set to 3×1×1, and the convolution kernel sizes of the second to fourth convolution layers are set to 1×3×3, 1×1×1, and 1×1×1, respectively; in the first motion convolution block, the number of channels of the convolution kernels of the first to fourth convolution layers are 32, 32, 32, and 32, and the number of convolution kernels are 32, 32, 128, and 128, respectively; in the second motion convolution block, the number of channels of the convolution kernels of the first to fourth convolution layers are 12, 128, 128, and 128, respectively. 8, 32, 32, 128, and the number of convolution kernels is 32, 32, 128, and 128 respectively; in the third motion convolution block, the number of channels of the convolution kernels of the first to fourth convolution layers are 128, 64, 64, and 128 respectively, and the number of convolution kernels is 64, 64, 256, and 256 respectively; in the fourth motion convolution block, the number of channels of the convolution kernels of the first to fourth convolution layers are 256, 64, 64, and 256 respectively, and the number of convolution kernels is 64, 64, 256, and 256 respectively; Step 2.3, construct the first to fourth feature fusion modules. The structures of the first to third feature fusion modules are the same, and each feature fusion module consists of a convolutional layer and a feature concatenation block; The feature splicing block is composed of feature splicing operations, which splice the input appearance features and motion features to obtain richer gait features; The convolution kernel size of the convolution layers in the first to third feature fusion modules is set to 1×1×1, the number of convolution kernel channels is set to 32, 128, and 256, and the number of convolution kernels is set to 32, 128, and 256, respectively; The feature concatenation layers in the first to third feature fusion modules are respectively set to concatenate features by channel, by channel, and by height dimension; Step 2.4, constructing an appearance flow feature module and a motion flow feature module in parallel, and using the first to third feature fusion modules to connect the appearance flow feature module and the motion flow module to obtain an appearance-motion two-flow sub-network; The convolution layer in the first feature fusion module is connected to the pooling layer of the head in the appearance flow feature module, and the output features of the convolution layer are spliced with the output features of the pooling layer of the head in the motion flow feature module according to the channel dimension, and then connected to the first motion convolution block in the appearance flow feature module; the convolution layer in the second feature fusion module is connected to the second appearance convolution block in the appearance flow feature module, and the output features of the convolution layer are spliced with the output features of the second motion convolution block in the motion flow feature module according to the channel dimension, and then connected to the third motion convolution block in the appearance flow feature module; the convolution layer in the third feature fusion module is connected to the fourth appearance convolution block in the appearance flow feature module, and the output features of the convolution layer are spliced with the output features of the fourth motion convolution block in the motion flow feature module according to the channel dimension to obtain the final output features, whose channel, height, and width are 256, 64, and 22 respectively; the construction of the appearance motion two-stream sub-network is completed; Step 3: Construct the feature mapping subnetwork: Step 3.1, construct a time domain pooling module, whose structure consists of a maximum pooling layer in the time dimension; The pooling kernel size of the maximum pooling layer in the temporal pooling module is set to T1×1×1, where T1 is the length of the input gait profile sequence; Step 3.2, construct a horizontal segmentation pyramid module, using 6 scales, whose structure is composed of a first-scale horizontal segmentation block, a second-scale horizontal segmentation block, a third-scale horizontal segmentation block, a fourth-scale horizontal segmentation block, a fifth-scale horizontal segmentation block, and a sixth-scale horizontal segmentation block in parallel; the structures of the first to sixth-scale horizontal segmentation blocks are the same, and each horizontal segmentation block is composed of a feature horizontal segmentation layer; The feature horizontal segmentation layer consists of feature segmentation operations in the height dimension, which divides the input features into multiple strips; The number of strips in the first to sixth horizontal segmentation layers is set to 1, 2, 4, 8, 16, and 32 respectively, the height of the strips is set to 64, 32, 16, 8, 4, and 2 respectively, and the width is set to 22, so a total of 63 strips are obtained, each strip represents a feature of each human body part; Step 3.3, constructing a generalized average pooling module structure is as follows: the first generalized pooling block, the second generalized pooling block, the third generalized pooling block, the fourth generalized pooling block, the fifth generalized pooling block, and the sixth generalized pooling block are connected in parallel; The generalized pooling block consists of a first power operation on each strip, an average pooling layer, and a second power operation on the feature in series; The exponents of the first and second power operations in the first to sixth generalized pooling blocks are set to the learnable parameter p. The first power operation is to raise the input feature to the power of p, and the second power operation is to raise the input feature to the power of The power, p, is set to 6.5 by default and can be learned and changed during training using the back-propagation algorithm; the height of the pooling kernel size of the average pooling layer in the first to sixth generalized pooling blocks is set to 64, 32, 16, 8, 4, 2, and the width is set to 22; Step 3.4, construct the feature projection module structure in sequence: independent fully connected block, batch normalization layer; The independent fully connected block is composed of 63 parallel fully connected layers with different parameters; The parameters of the 63 fully connected layers in the independent fully connected blocks are initialized using the normal distribution N(0,0.01) by default; the batch normalization layer is set to batch normalize the one-dimensional feature vector, and the number of channels is set to 256; Step 3.5, connect the time domain pooling module, the horizontal segmentation pyramid module, the generalized average pooling module, and the feature projection module in series to form a feature mapping subnetwork; Step 4, connect the appearance motion two-stream sub-network and the feature mapping sub-network in series to form a deep learning network; Step 5, generate the joint loss function as follows: L total =αL triplet +βL ce +λL weight Among them, L total represents the joint loss function, L triplet represents the triple loss function, L ce represents the multi-classification cross entropy loss function, L weight represents the weight decay loss function, α represents the coefficient of the triplet loss function, its value is 1, β represents the coefficient of the multi-classification loss function, its value is 0.1, and λ represents the coefficient of the weight decay loss function, its value is 0.00005; Step 6: Train the deep learning network: Input the training set into the deep learning network; use the joint loss function to calculate the overall loss value; back-propagate the overall loss value in the deep learning network, and use the gradient descent method to update the weight parameters of each network node until the loss value converges to obtain a trained deep learning network; Step 7: Cross-view gait recognition: Step 7.1, send the samples in the registered data set to the trained deep learning network to obtain a one-dimensional vector output as the feature vector representation of each gait profile sequence, and finally obtain the feature database of the registered data set; Step 7.2, send the samples in the query data set to be identified into the trained deep learning network to obtain the feature vectors of the samples in the query data set; calculate the Euclidean distance between the feature vector of each sample in the query data set and the feature vectors of all samples in the registration data set, and finally identify the query sample as the identity label of the feature vector in the registration data set with the smallest Euclidean distance, output the identity label of the query sample, and complete cross-view recognition.
2. The cross-view gait recognition method based on metric learning and spatiotemporal dual-stream network according to claim 1 is characterized in that: The triplet loss function described in step 5 is as follows: in, represents the total number of non-zero items of triplet loss in a batch, parts represents the total number of partial features at all scales, S represents the total number of multi-scales in the horizontal segmentation pyramid module, s represents the index of the scale, t represents the index of the partial feature, anchors represents the anchor sample set, H and M represent the number of subjects in a batch and the number of gait profile sequences of each subject, h and m represent the subject index and the index of the gait profile sequence, pos. represents the positive sample set with the same label as the anchor sample, a represents the index of the positive sample, negative represents the negative sample set with different labels from the anchor sample, b and c represent the index of the subject in the negative sample set and the index of the gait profile sequence, V2 represents the feature vector of the sample obtained after the gait profile sequence is input into the deep learning network, ω is the boundary threshold margin of the triplet loss, dist represents the distance between the positive sample pair and the negative sample pair, [x] + represents the maximum value of input x and 0, D(·) represents the Euclidean distance function, Represents the t-th partial feature of the s-th scale of the m-th gait profile sequence of the h-th subject.
3. The cross-view gait recognition method based on metric learning and spatiotemporal dual-stream network according to claim 2 is characterized in that: The multi-class cross entropy loss function described in step 5 is as follows: Among them, H and M represent the number of subjects in a batch and the number of gait profile sequences of each subject, respectively. h and m represent the index of the subject and the index of the gait profile sequence, respectively. N represents the total number of all subject categories in the training set, and n represents the index of the category. represents the predicted probability that the m-th gait profile sequence of the h-th subject belongs to the n-th subject category, p represents the category prediction probability, Indicates the true label that the m-th gait profile sequence of the h-th subject belongs to the n-th subject category.
4. The cross-view gait recognition method based on metric learning and spatiotemporal dual-stream network according to claim 1 is characterized in that: The weight decay loss function described in step 5 is as follows; Where Z represents the total number of parameters in the deep learning network, i represents the index of the parameter in the deep learning network, and W i Represents the i-th parameter in the deep learning network.
