A Gait Recognition Method Based on Gait Relation Network

Abstract

This invention discloses a gait recognition method based on a gait relationship network. The gait relationship network comprises one spatial feature extraction network, 16 temporal feature extraction networks, and 16 spatial feature selection networks. The spatial feature extraction network includes one global encoding network, one image segmentation network, four local encoding networks, one fusion encoding network, and one spatial feature segmentation network. When using the gait relationship network to obtain pedestrian gait feature maps, it can fully utilize gait image data under various gait conditions, increasing recognition accuracy. The number and order of gait images in the gait image sequence to be recognized are not required. Its advantages include high recognition accuracy even when gait data is missing, strong robustness, and excellent performance.

Description

Technical Field

[0001] This invention relates to a gait recognition method, and more particularly to a gait recognition method based on a gait relationship network. Background Technology

[0002] Gait recognition technology is a technique that uses gait information to identify a person. Compared with biometric recognition methods such as fingerprints, irises, and faces, video-based gait recognition has advantages such as easy data collection, long distance, non-contact operation, difficulty in spoofing, and no need for cooperation. As a new generation of biometric recognition technology, gait recognition has received increasing attention from experts and scholars in recent years. With the rapid development of computer vision and artificial intelligence technologies, there is a wide range of technical needs for public security systems and intelligent video analysis systems that combine gait recognition technology in ensuring public safety and improving the scientific management level of smart cities. Its development is also closely watched by the industry.

[0003] Real-world surveillance scenarios present complex and varied conditions, posing a significant challenge to gait feature extraction. Gait recognition identifies pedestrian gait sequences, which consist of a series of gait images. The completeness of the gait sequence affects the accuracy of gait recognition. Data loss in the gait sequence is referred to as gait data loss, such as gait images being out of order (i.e., missing time information within the sequence) or a very small number of gait images. Data loss is a major technical bottleneck for gait recognition in practical applications. The main reasons for gait data loss are: (1) pedestrian self-occlusion caused by perspective, carrying objects, clothing, etc. (this is the most common reason for missing pedestrian gait modeling data); (2) occlusion by objects in the scene, such as occlusion between pedestrians, vehicles, buildings, etc. in actual surveillance, leading to the temporary or complete loss of the target pedestrian; (3) the algorithm cannot obtain effective source images due to complex backgrounds, lighting, weather conditions, etc., such as the obtained pedestrian contour information being blurry or containing a lot of noise.

[0004] In the existing research field of gait recognition methods, some mainstream methods are based on the order of gait sequences for gait modeling. Among these methods, some believe that short-term trunk changes are the focus of gait recognition, such as GaitPart [Fan C, Peng Y, Cao C, et al. Gaitpart: Temporal part-based model for gait recognition[C] / / Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition.2020:14225-14233.] which uses micro-motion principles for modeling. Other methods believe that long-term dependencies are essential, such as GaitGL [Lin B, Zhang S, Yu X, et al. Learning Effective Representations from Global and Local Features for Cross-View Gait Recognition[J].2020.] which uses 3D convolutional blocks for gait recognition. The aforementioned methods achieve high recognition accuracy when the gait sequence contains a large number of images and the order of the images is intact. However, their accuracy drops sharply when the number of images in the gait sequence is small or the order of the images is disrupted. Therefore, these methods are unsuitable for situations where gait images are missing or where gait data is lost due to scrambling. To address the problem of missing gait data, the literature GaitSet [Chao H, Wang K, He Y, et al. GaitSet: Cross-view Gait Recognition through Utilizing Gait as a Deep Set[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI). 2021.] proposes a deep set modeling method to solve this problem. However, this deep set modeling method uses a compression operation on the time axis, which can easily lead to the loss of temporal information. In summary, most current methods have high requirements for the continuity of the gait sequence and the number of gait images in the sequence. Their recognition accuracy is low when the order of gait images is disrupted, and also low when the number of gait images is small. Summary of the Invention

[0005] The technical problem to be solved by the present invention is to provide a gait recognition method based on gait relationship network that still has a high recognition accuracy when the order of gait images is shuffled and the number of gait images is small.

[0006] The technical solution adopted by this invention to solve the above-mentioned technical problems is as follows: a gait recognition method based on gait relationship network, comprising the following steps:

[0007] Step 1: From the Center for Biometrics and Security Technology Research A gait dataset was obtained from the CASIA-B gait dataset of the Security Research (CBSR). This gait dataset contains gait images of 124 pedestrians under 3 walking conditions and 11 viewpoints. The 124 pedestrian identities are labeled with numbers 001-124. The 3 walking conditions are normal walking, backpack walking, and walking while wearing a coat or jacket. The 11 viewpoints are 0°, 18°, 36°, 54°, 72°, 90°, 108°, 126°, 144°, 162°, and 180°. In this gait dataset, each pedestrian identity has 6 sequence sets under the normal walking condition, 2 sequence sets under the backpack walking condition, and 2 sequence sets under the coat or jacket walking condition. Each sequence set includes 11 gait image sequences. The 11 gait image sequences were taken from 11 viewpoints, and there is a one-to-one correspondence between the 11 viewpoints and 11 gait image sequences. Each gait image sequence consists of several gait images.

[0008] Step 2: Using the Takemura method proposed in the literature [Noriko Takemura, Yasushi Makihara, Daigo Muramatsu, Tomio Echigo, and Yasushi Yagi. Multi-view large population gait dataset and its performance evaluation for cross-view gait recognition. IPSJ Transactions on Computer Vision and Applications, 10, 12 2018.1, 2, 6, 7, 8], each gait image in the gait dataset is processed separately to make each gait image have a height of H and a width of W, where H = 64 and W = 44. At this point, the training dataset is obtained.

[0009] Step 3: Construct a gait relationship network, which includes one spatial feature extraction network, 16 temporal feature extraction networks, and 16 spatial feature selection networks. The spatial feature extraction network includes one global encoding network, one image segmentation network, four local encoding networks, one fusion encoding network, and one spatial feature segmentation network. The global encoding network includes one input layer, four convolutional layers, one max-pooling layer, and two output layers. The first convolutional layer of the global encoding network uses 32 convolutional kernels of size 5×5 with a zero-padding parameter of 2 and a stride of 1. The second convolutional layer of the global encoding network uses... The global coding network is implemented using 32 convolutional kernels of size 3×3, zero-padding parameter value of 1, and stride of 1. The third convolutional layer of the global coding network is implemented using 64 convolutional kernels of size 3×3, zero-padding parameter value of 1, and stride of 1. The fourth convolutional layer of the global coding network is implemented using 64 convolutional kernels of size 3×3, zero-padding parameter value of 1, and stride of 1. The stride of the max pooling layer of the global coding network is set to 2. The image segmentation network includes one input layer, one segmentation layer, and four output layers. Each local coding network includes one input layer, four convolutional layers, one max pooling layer, and two output layers. The first convolutional layer of each local coding network is implemented using 32 convolutional kernels of size 5×5, with a zero-padding parameter of 2 and a stride of 1. The second convolutional layer of each local coding network is implemented using 32 convolutional kernels of size 3×3, with a zero-padding parameter of 1 and a stride of 1. The third convolutional layer of each local coding network is implemented using 64 convolutional kernels of size 3×3, with a zero-padding parameter of 1 and a stride of 1. The fourth convolutional layer of each local coding network is implemented using 64 convolutional kernels of size 3×3, with a zero-padding parameter of 1 and a stride of 1. The max pooling layer of each local coding network... The stride of each layer is set to 2. The fusion coding network includes 10 input layers, 2 feature concatenation layers, 3 feature fusion layers, 2 convolutional layers, 1 max pooling layer, and 1 output layer. The first convolutional layer of the fusion coding network is implemented with 128 convolutional kernels of size 3×3, zero-padding parameter value of 1, and stride of 1. The second convolutional layer of the fusion coding network is implemented with 128 convolutional kernels of size 3×3, zero-padding parameter value of 1, and stride of 1. The stride of the max pooling layer of the fusion coding network is set to 2. The spatial feature segmentation network includes 1 input layer, 1 segmentation layer, 1 pooling layer, and 1 output layer.Each of the aforementioned temporal feature extraction networks includes one input layer, three mapping layers, one temporal feature selection layer, one independent mapping layer, and one output layer. The three mapping layers and the independent mapping layer of the temporal feature extraction network are each fully connected layers with 128 input channels and 128 output channels. The temporal feature selection layer of the temporal feature extraction network is implemented using the max function. Each of the aforementioned spatial feature selection networks includes one input layer, one feature selection layer, one independent mapping layer, and one output layer. The independent mapping layer of each spatial feature selection network is each a fully connected layer with 128 input channels and 128 output channels.

[0010] When training the gait relationship network, the input layer of the global encoding network receives a gait image sequence with dimensions of 30×1×64×44, where the dimension is represented as number of images × number of channels × image length × image width. The first convolutional layer of the global encoding network receives the gait image sequence with dimensions of 30×1×64×44 output from the input layer of the global encoding network, and performs feature extraction to obtain a feature map F with dimensions of 30×32×64×44. global_1 The output is the feature map F output by the first convolutional layer of the global encoding network, which is connected to the second convolutional layer of the global encoding network. global_1 Feature extraction was performed to obtain a feature map F with dimensions of 30×32×64×44. global_2 The output, the max pooling layer of the global encoding network, is connected to the feature map F output by the second convolutional layer of the global encoding network. global_2 Feature extraction was performed to obtain a feature map F with dimensions of 30×32×32×22. global_pool The output is the feature map F output by the third convolutional layer of the global encoding network, which is connected to the max pooling layer of the global encoding network. global_pool Feature extraction was performed to obtain a feature map F with dimensions of 30×64×32×22. global_3 The output is the feature map F output by the third convolutional layer of the global encoding network, which is connected to the fourth convolutional layer of the global encoding network. global_3 Feature extraction was performed to obtain a feature map F with dimensions of 30×64×32×22. global_4 The output is the feature map F output by the first output layer of the global encoding network, which is connected to the max pooling layer of the global encoding network. global_pool And output, the second output layer of the global encoding network is connected to the feature map F output by the fourth convolutional layer of the global encoding network. global_4The image segmentation network's input layer receives and outputs a gait image sequence with dimensions of 30×1×64×44. The image segmentation network's segmentation layer receives the gait image sequence output from its input layer (30×1×64×44) and segments it at the midpoints of its length and width to obtain four gait image sub-sequences with dimensions of 30×1×32×22. The image segmentation network's four output layers each receive and output the four gait image sub-sequences with dimensions of 30×1×32×22. Four local coding networks each receive and output the corresponding gait image sub-sequences. The image segmentation network described above outputs four gait image subsequences with dimensions of 30×1×32×22 from its four output layers. The input layer of the Nth local coding network is then fed into and outputs the gait image subsequence with dimensions of 30×1×32×22 from its Nth output layer. N = 1, 2, 3, 4. The first convolutional layer of the Nth local coding network is fed into and outputs the gait image subsequence with dimensions of 30×1×32×22 from its input layer, and feature extraction is performed to obtain a feature map with dimensions of 30×32×32×22. The second convolutional layer of the Nth local coding network is then fed into and outputs the gait image subsequence with dimensions of 30×1×32×22 from its input layer. The feature maps output by the first convolutional layer of the Nth local coding network are processed, and feature extraction is performed to obtain a feature map output with dimensions of 30×32×32×22. The max pooling layer of the Nth local coding network is connected to the feature map output by the second convolutional layer of the Nth local coding network, and feature extraction is performed to obtain a feature map output with dimensions of 30×32×16×11. The third convolutional layer of the Nth local coding network is connected to the feature map output by the max pooling layer of the Nth local coding network, and feature extraction is performed to obtain a feature map output with dimensions of 30×64×16×11. The fourth convolutional layer of the Nth local coding network... The feature map output from the third convolutional layer of the Nth local coding network is input into the convolutional layer, and feature extraction is performed to obtain a feature map output with dimensions of 30×64×16×11. The first output layer of the Nth local coding network is used to input the feature map output from the max pooling layer of the Nth local coding network and outputs the feature map. The second output layer of the Nth local coding network is used to input the feature map output from the fourth convolutional layer of the Nth local coding network and outputs the feature map. In the four local coding networks, the feature map output from the first output layer and the feature map output from the second output layer of the first local coding network are denoted as F. local_pool,1 and F local_4,1 Let F be the feature map output by the first output layer and the feature map output by the second output layer of the second local coding network. local_pool,2 and Flocal_4,2 Let F be the feature map output by the first output layer and the feature map output by the second output layer of the third local coding network. local_pool,3 and F local_4,3 Let F be the feature map output by the first output layer and the feature map output by the second output layer of the fourth local coding network. local_pool,4 and F local_4,4 The 10 input layers of the fusion coding network are connected one-to-one to the feature map F output by the first output layer of the global coding network. global_pool The feature map F output by the second output layer of the global coding network. global_4 The feature maps F output by the first and second output layers of the four local coding networks. local_pool,1 Feature map F local_4,1 Feature map F local_pool,2 Feature map F local_4,2 Feature map F local_pool,3 Feature map F local_4,3 Feature map F local_pool,4 and feature map F local_4,4 The first feature concatenation layer of the fusion coding network outputs the input feature map F. local_pool,1 Feature map F local_pool,2 Feature map F local_pool,3 and feature map F local_pool,4 After arranging and splicing the pieces according to the previously cut positions, a feature map F with dimensions of 30×32×32×22 is obtained. local_pool,all The output is the second feature concatenation layer of the fusion coding network connected to the feature map F. local_4,1 F local_4,2 F local_4,3 and F local_4,4 After being arranged and spliced ​​according to the previously corresponding cutting positions, a feature map F with dimensions of 30×64×32×22 is obtained. local_4,all The output is the first feature fusion layer of the fusion coding network connected to the feature map F. local_pool,all and feature map F local_4,all and feature map F local_pool,all and feature map F local_4,all Dimensional concatenation is performed, followed by feature fusion using a 1×1 convolutional kernel to obtain a feature map F with dimensions of 30×64×32×22. local The output is the second feature fusion layer of the fusion coding network accessing the feature map F. global_pool and feature map F global_4 The two are then concatenated dimensionally, and features are fused using a 1×1 convolutional kernel to obtain a feature map F with dimensions of 30×64×32×22. globalThe output, the third feature fusion layer of the fusion coding network, is connected to the feature map F output by the first feature fusion layer of the fusion coding network. local The feature map F output by the second feature fusion layer of the fusion coding network. global Then, the two are added one-to-one according to their pixel positions to obtain a feature map F with dimensions of 30×64×32×22. fuse Output, i.e., F fuse =F global +F local The first convolutional layer of the fusion coding network is connected to the feature map F output by the third feature fusion layer of the fusion coding network. fuse Feature extraction was performed to obtain a feature map F with dimensions of 30×128×32×22. fuse_1 Output: The second convolutional layer of the fusion coding network is connected to the feature map F output by the first convolutional layer of the fusion coding network. fuse_1 Feature extraction was performed to obtain a feature map F with dimensions of 30×128×32×22. fuse_2 The output, the max pooling layer of the fusion coding network, is connected to the feature map F output by the second convolutional layer of the fusion coding network. fuse_2 The spatial feature map F with dimensions of 30×128×16×11 is extracted and output. The output layer of the fusion coding network is connected to the spatial feature map F output by the max pooling layer of the fusion coding network and output. The input layer of the spatial feature cutting network is connected to the spatial feature map F with dimensions of 30×128×16×11 output by the output layer of the fusion coding network and output. The cutting layer of the spatial feature cutting network is connected to the spatial feature map F output by the input layer of the spatial feature cutting network and cut into 16 equal parts in the length and width directions to obtain 16 feature maps with dimensions of 30×128×1×11. The i-th feature map with dimensions of 30×128×1×11 is denoted as F. i Let i = 1, 2, 3, ..., 16. The pooling layer of the spatial feature segmentation network is connected to the 16 feature maps with dimensions of 30×128×1×11 output from the segmentation layer of the spatial feature segmentation network, and feature extraction is performed on each of them to obtain 16 feature map outputs with dimensions of 30×128×1×1. The feature map f obtained after feature extraction on the i-th feature map with dimensions of 30×128×1×11 is then processed. i The 16 temporal feature extraction networks are connected one-to-one with the pooling layer output of the spatial feature segmentation network, which outputs 16 feature maps of 30×128×1×1 dimensions. The input layer of the i-th temporal feature extraction network is connected to the i-th dimension 30×128×1×1 feature map f output by the pooling layer of the spatial feature segmentation network.i And output, the first mapping layer of the i-th temporal feature extraction network is connected to the feature map f output by the input layer of the i-th temporal feature extraction network. i Feature extraction was performed to obtain a feature map a with dimensions of 30×128×1×1. i The output is the feature map a output by the first mapping layer of the i-th temporal feature extraction network, which is connected to the second mapping layer of the i-th temporal feature extraction network. i First, feature extraction is performed to obtain a feature map b with dimensions of 30×128×1×1. i Then feature map a i All dimensions and feature maps b i Subtracting the corresponding dimensions yields a feature map c with dimensions 30×128×1×1. i The output is the feature map c output by the third mapping layer of the i-th temporal feature extraction network. i Then, feature extraction is performed to obtain a feature map d with dimensions of 30×128×1×1. i The output is the feature map d output by the third mapping layer of the i-th temporal feature extraction network, which is connected to the temporal feature selection layer of the i-th temporal feature extraction network. i And perform feature extraction to obtain a feature map e with dimensions of 1×128×1×1. i The output is the feature map e output by the time feature selection layer of the i-th time feature extraction network, which is connected to the independent mapping layer of the i-th time feature extraction network. i And perform feature extraction to obtain a feature map v with dimensions of 1×128×1×1. i Output: 16 feature maps of 30×128×1×1 dimensions output by the pooling layer of the spatial feature selection network, corresponding one-to-one with each of the 16 mentioned spatial feature selection networks. The input layer of the i-th mentioned spatial feature selection network is connected to the i-th dimension 30×128×1×1 feature map f output by the pooling layer of the spatial feature cutting network. i And output, the feature selection layer of the i-th spatial feature selection network is connected to the feature map f output by the input layer of the i-th spatial feature selection network. i Feature extraction is then performed to obtain a feature map A with dimensions of 1×128×1×1. i The output is the feature map A output by the feature selection layer of the i-th spatial feature selection network, which is connected to the independent mapping layer of the i-th spatial feature selection network. i And perform feature extraction to obtain a feature map s with dimensions of 1×128×1×1. i Output; Feature map vi and feature map s i That is, the gait feature map output by the gait relationship network;

[0011] Step 4: Train the constructed gait relationship network. Specifically, first, divide the training dataset obtained in Step 2 into a training set and a test set. Data under 74 pedestrian identities is used as the training set, and data under the remaining 50 pedestrian identities is used as the test set. During each training iteration, randomly select 42 gait image sequences from the training set, and then randomly select 30 gait images from each gait image sequence as input to the gait relationship network. If the number of gait images in a certain gait image sequence is less than 30, randomly repeat the selection until 30 gait images are obtained. During each training process, the parameters of the gait relationship network are optimized using a loss function until the training iteration reaches 200,000 times, resulting in a trained gait relationship network. The loss function is set to the Batch All (BA+) Triplet loss disclosed in the literature [A. Hermans, L. Beyer, B. Leibe, In defense of the triplet loss for person re-identification, CoRR abs / 1703.07737 (2017).]. The Cross-Entropy Loss method was disclosed in the literature [Hao Luo, Youzhi Gu, Xingyu Liao, Shenqi Lai, and WeiJiang. Bag of tricks and a strong baseline for deep personre-identification. In CVPR Workshops, June 2019, 2, 3].

[0012] Step 5: When the pedestrian to be identified needs to undergo gait recognition, a gait image sequence of the pedestrian to be identified is processed in the same way as in Step 2, so that the size of each gait image in the gait image sequence is 64×44 (length×width). Then, the processed gait image sequence is input into the trained gait relationship network to obtain 32 gait feature maps of the pedestrian to be identified.

[0013] Step 6: Perform the same operation as step 5 on all gait image sequences in the pre-established gait database that includes the identity of the pedestrian to be identified, to obtain 32 gait feature maps for each pedestrian identity. Then, calculate the Euclidean distance between the 32 feature maps obtained in step 5 and the 32 gait feature maps for each pedestrian identity obtained at this time, and obtain the pedestrian identity with the smallest Euclidean distance to the 32 feature maps obtained in step 5. This pedestrian identity is the identity of the pedestrian to be identified.

[0014] Compared with existing technologies, the advantages of this invention lie in constructing a gait relationship network, which includes one spatial feature extraction network, 16 temporal feature extraction networks, and 16 spatial feature selection networks. The spatial feature extraction network includes one global encoding network, one image segmentation network, four local encoding networks, one fusion encoding network, and one spatial feature segmentation network. The global encoding network includes one input layer, four convolutional layers, one max-pooling layer, and two output layers. The first convolutional layer of the global encoding network uses 32 convolutional kernels with a size of 5×5, a zero-padding parameter of 2, and a stride of 1. The second convolutional layer of the global encoding network uses kernels with a size of 3×3, a zero-padding parameter of 1, and a stride of 1. The global coding network uses 32 convolutional kernels. The third convolutional layer uses 64 kernels of size 3×3 with zero-padding of 1 and stride of 1. The fourth convolutional layer uses 64 kernels of size 3×3 with zero-padding of 1 and stride of 1. The stride of the max pooling layer in the global coding network is set to 2. The image segmentation network consists of one input layer, one segmentation layer, and four output layers. Each local coding network consists of one input layer, four convolutional layers, one max pooling layer, and two output layers. The first convolutional layer of each local coding network uses 32 kernels of size 5×5 with zero-padding of 2 and stride of 1. The first two convolutional layers each use 32 kernels of size 3×3 with zero-padding of 1 and a stride of 1. The third convolutional layer of each local coding network uses 64 kernels of size 3×3 with zero-padding of 1 and a stride of 1. The fourth convolutional layer of each local coding network uses 64 kernels of size 3×3 with zero-padding of 1 and a stride of 1. The stride of the max pooling layer in each local coding network is set to 2. The fusion coding network consists of 10 input layers, 2 feature concatenation layers, 3 feature fusion layers, 2 convolutional layers, 1 max pooling layer, and 1 output layer. The first convolutional layer of the fusion coding network uses a kernel of size 3×3 with zero-padding of 1. The first convolutional layer of the fusion coding network uses 128 convolutional kernels with a stride of 1. The second convolutional layer of the fusion coding network uses 128 convolutional kernels with a size of 3×3, a zero-padding parameter of 1, and a stride of 1. The stride of the max pooling layer of the fusion coding network is set to 2. The spatial feature segmentation network includes one input layer, one segmentation layer, one pooling layer, and one output layer. Each temporal feature extraction network includes one input layer, three mapping layers, one temporal feature selection layer, one independent mapping layer, and one output layer. The three mapping layers and the independent mapping layer of the temporal feature extraction network are each fully connected layers with 128 input channels and 128 output channels. The temporal feature selection layer of the temporal feature extraction network is implemented using the max function.Each spatial feature selection network comprises one input layer, one feature selection layer, one independent mapping layer, and one output layer. The independent mapping layer of each spatial feature selection network is a fully connected layer with 128 input channels and 128 output channels. When using gait relation networks to acquire pedestrian gait feature maps, it can fully utilize gait image data under various gait conditions, increasing recognition accuracy. The number and order of gait images in the sequence to be recognized are not required; therefore, even with missing gait data, the recognition accuracy remains high, exhibiting strong robustness and excellent performance. Detailed Implementation

[0015] The present invention will be further described in detail below with reference to the embodiments.

[0016] Example: A gait recognition method based on gait relationship networks, comprising the following steps:

[0017] Step 1: Obtain a gait dataset from the CASIA-B gait dataset of the Center for Biometrics and Security Research (CBSR). This dataset contains gait images of 124 pedestrians under 3 walking conditions and 11 viewpoints. The 124 pedestrians are labeled with numbers 001-124. The 3 walking conditions are normal walking (NM), backpack walking (BG), and walking while wearing a coat or jacket (CL). The 11 viewpoints are 0°, 18°, 36°, 54°, 72°, 9 ... The gait dataset includes 0°, 108°, 126°, 144°, 162°, and 180°. For each pedestrian identity, there are 6 sequence sets under normal walking conditions, 2 sequence sets under backpack walking conditions, and 2 sequence sets under the condition of walking while wearing a coat or jacket. Each sequence set includes 11 gait image sequences. The 11 gait image sequences were taken from 11 different viewpoints, and there is a one-to-one correspondence between the 11 viewpoints and the 11 gait image sequences. Each gait image sequence consists of several gait images.

[0018] Step 2: Using the Takemura method proposed in the literature [Noriko Takemura, Yasushi Makihara, Daigo Muramatsu, Tomio Echigo, and Yasushi Yagi. Multi-view large population gait dataset and its performance evaluation for cross-view gait recognition. IPSJ Transactions on Computer Vision and Applications, 10, 12 2018.1, 2, 6, 7, 8], each gait image in the gait dataset is processed separately to make each gait image have a height of H and a width of W, where H = 64 and W = 44. At this point, the training dataset is obtained.

[0019] Step 3: Construct a gait relationship network, which includes one spatial feature extraction network, 16 temporal feature extraction networks, and 16 spatial feature selection networks. The spatial feature extraction networks include one global encoder, one image segmentation network, four local encoders, one fusion encoder, and one spatial feature segmentation network. The global encoder includes one input layer, four convolutional layers, one max-pooling layer, and two output layers. The first convolutional layer of the global encoder uses 32 convolutional kernels of size 5×5 with a zero-padding parameter of 2 and a stride of 1. The second convolutional layer of the global encoder uses 32 convolutional kernels of size 3×3 with a zero-padding parameter of 1 and a stride of 1. The third convolutional layer of the local coding network is implemented using 64 convolutional kernels of size 3×3 with a zero-padding parameter of 1 and a stride of 1. The fourth convolutional layer of the global coding network is implemented using 64 convolutional kernels of size 3×3 with a zero-padding parameter of 1 and a stride of 1. The stride of the max pooling layer of the global coding network is set to 2. The image segmentation network includes one input layer, one segmentation layer, and four output layers. Each local coding network includes one input layer, four convolutional layers, one max pooling layer, and two output layers. The first convolutional layer of each local coding network is implemented using 32 convolutional kernels of size 5×5 with a zero-padding parameter of 2 and a stride of 1. The second convolutional layer of each local coding network is implemented using... The local coding network is implemented using 32 convolutional kernels of size 3×3 with zero-padding of 1 and stride of 1. The third convolutional layer of each local coding network uses 64 convolutional kernels of size 3×3 with zero-padding of 1 and stride of 1. The fourth convolutional layer of each local coding network uses 64 convolutional kernels of size 3×3 with zero-padding of 1 and stride of 1. The stride of the max pooling layer in each local coding network is set to 2. The fusion coding network consists of 10 input layers, 2 feature concatenation layers, 3 feature fusion layers, 2 convolutional layers, 1 max pooling layer, and 1 output layer. The first convolutional layer of the fusion coding network uses a kernel of size 3×3 with zero-padding of 1 and stride of 1. The first convolutional layer of the fusion coding network uses 128 convolutional kernels of size 3×3 with zero-padding parameter of 1 and stride of 1. The stride of the max pooling layer of the fusion coding network is set to 2. The spatial feature segmentation network includes one input layer, one segmentation layer, one pooling layer and one output layer. Each temporal feature extraction network includes one input layer, three mapping layers, one temporal feature selection layer, one independent mapping layer and one output layer. The three mapping layers and the independent mapping layer of the temporal feature extraction network are fully connected layers with 128 input channels and 128 output channels. The temporal feature selection layer of the temporal feature extraction network is implemented using the max function.Each spatial feature selection network includes one input layer, one feature selection layer, one independent mapping layer, and one output layer. The independent mapping layer of each spatial feature selection network is a fully connected layer with 128 input channels and 128 output channels.

[0020] When training the gait relation network, the input layer of the global encoder network receives a gait image sequence with dimensions of 30×1×64×44, where the dimension is represented as number of images × number of channels × image length × image width. The first convolutional layer of the global encoder network receives the gait image sequence with dimensions of 30×1×64×44 output from its input layer and performs feature extraction to obtain a feature map F with dimensions of 30×32×64×44. global_1 The output is the feature map F output by the first convolutional layer of the global encoding network, which is connected to the second convolutional layer of the global encoding network. global_1 Feature extraction was performed to obtain a feature map F with dimensions of 30×32×64×44. global_2 The output, the feature map F, is fed into the second convolutional layer of the global encoding network after the max pooling layer of the global encoding network. global_2 Feature extraction was performed to obtain a feature map F with dimensions of 30×32×32×22. global_pool The output is the feature map F output by the third convolutional layer of the global encoding network, which is connected to the max pooling layer of the global encoding network. global_pool Feature extraction was performed to obtain a feature map F with dimensions of 30×64×32×22. global_3 The output is the feature map F output by the third convolutional layer of the global encoding network, which is connected to the fourth convolutional layer of the global encoding network. global_3 Feature extraction was performed to obtain a feature map F with dimensions of 30×64×32×22. global_4 The output is the feature map F output by the first output layer of the global encoder network, which is connected to the max pooling layer of the global encoder network. global_pool And output, the feature map F output by the fourth convolutional layer of the global encoding network is connected to the second output layer of the global encoding network. global_4The image segmentation network's input layer receives and outputs a gait image sequence with dimensions of 30×1×64×44. Its segmentation layer receives this gait image sequence and segments it at the midpoints of its length and width, resulting in four gait image sub-sequences with dimensions of 30×1×32×22. The four output layers of the image segmentation network receive and output these four gait image sub-sequences. The four local coding networks also receive and output these sub-sequences. The image segmentation network outputs four gait image subsequences with dimensions 30×1×32×22 from its four output layers. The input layer of the Nth local coding network (N=1, 2, 3, 4) is fed into the gait image subsequence with dimensions 30×1×32×22 output from the Nth output layer of the image segmentation network and outputs it. The first convolutional layer of the Nth local coding network (N=1, 2, 3, 4) is fed into the gait image subsequence with dimensions 30×1×32×22 output from its input layer and performs feature extraction to obtain a feature map with dimensions 30×32×32×22. The second convolutional layer of the Nth local coding network (N=1, 2, 3, 4) is fed into the gait image subsequence with dimensions 30×1×32×22 output from its input layer. The feature map output from the first convolutional layer of N local coding networks is processed, and feature extraction is performed to obtain a feature map output with dimensions of 30×32×32×22. The max-pooling layer of the Nth local coding network then connects to the feature map output from the second convolutional layer of the Nth local coding network, and feature extraction is performed to obtain a feature map output with dimensions of 30×32×16×11. The third convolutional layer of the Nth local coding network then connects to the feature map output from the max-pooling layer of the Nth local coding network, and feature extraction is performed to obtain a feature map output with dimensions of 30×64×16×11. The fourth convolutional layer of the Nth local coding network... The feature map output from the 3rd convolutional layer of the Nth local coding network is used for feature extraction, resulting in a feature map output with dimensions of 30×64×16×11. The 1st output layer of the Nth local coding network is used to input the feature map output from the max pooling layer of the Nth local coding network and output this feature map. The 2nd output layer of the Nth local coding network is used to input the feature map output from the 4th convolutional layer of the Nth local coding network and output this feature map. In the four local coding networks, the feature maps output from the 1st and 2nd output layers of the 1st local coding network are denoted as F, F1, F2, F3, F4, F5, F6, F7, F8, F9, F10 ... local_pool,1 and F local_4,1 Let F be the feature map output by the first output layer and the feature map output by the second output layer of the second local coding network. local_pool,2 and F local_4,2 Let F be the feature map output by the first output layer and the feature map output by the second output layer of the third local coding network. local_pool,3 and Flocal_4,3 Let F be the feature map output by the first output layer and the feature map output by the second output layer of the fourth local coding network. local_pool,4 and F local_4,4 The 10 input layers of the fusion coding network are connected one-to-one to the feature map F output by the first output layer of the global coding network. global_pool The feature map F output by the second output layer of the global encoding network global_4 The feature maps F output by the first and second output layers of the four local coding networks. local_pool,1 Feature map F local_4,1 Feature map F local_pool,2 Feature map F local_4,2 Feature map F local_pool,3 Feature map F local_4,3 Feature map F local_pool,4 and feature map F local_4,4 The input feature map is then output, and the first feature concatenation layer of the fusion coding network inputs the feature map F. local_pool,1 Feature map F local_pool,2 Feature map F local_pool,3 and feature map F local_pool,4 After arranging and splicing the pieces according to the previously cut positions, a feature map F with dimensions of 30×32×32×22 is obtained. local_pool,all The output is the second feature concatenation layer of the fusion coding network, which is connected to the feature map F. local_4,1 F local_4,2 F local_4,3 and F local_4,4 After being arranged and spliced ​​according to the previously corresponding cutting positions, a feature map F with dimensions of 30×64×32×22 is obtained. local_4,all The output is the first feature fusion layer of the fusion coding network connected to the feature map F. local_pool,all and feature map F local_4,all and feature map F local_pool,all and feature map F local_4,all Dimensional concatenation is performed, followed by feature fusion using a 1×1 convolutional kernel to obtain a feature map F with dimensions of 30×64×32×22. local The output is the second feature fusion layer of the fusion coding network connected to the feature map F. global_pool and feature map F global_4 The two are then concatenated dimensionally, and features are fused using a 1×1 convolutional kernel to obtain a feature map F with dimensions of 30×64×32×22. global The output is the feature map F output by the first feature fusion layer of the fusion coding network, which is connected to the third feature fusion layer of the fusion coding network. local The feature map F output by the second feature fusion layer of the fusion coding network globalThen, the two are added one-to-one according to their pixel positions to obtain a feature map F with dimensions of 30×64×32×22. fuse Output, i.e., F fuse =F global +F local The first convolutional layer of the fusion coding network is connected to the feature map F output by the third feature fusion layer of the fusion coding network. fuse Feature extraction was performed to obtain a feature map F with dimensions of 30×128×32×22. fuse_1 Output: The feature map F output by the first convolutional layer of the fusion coding network is connected to the second convolutional layer of the fusion coding network. fuse_1 Feature extraction was performed to obtain a feature map F with dimensions of 30×128×32×22. fuse_2 The output, the feature map F, is fed into the max pooling layer of the fusion coding network and then into the second convolutional layer of the fusion coding network. fuse_2 The spatial feature map F with dimensions 30×128×16×11 is extracted and output. The output layer of the fusion coding network is connected to the spatial feature map F output by the max pooling layer of the fusion coding network and output. The input layer of the spatial feature cutting network is connected to the spatial feature map F with dimensions 30×128×16×11 output by the output layer of the fusion coding network and output. The cutting layer of the spatial feature cutting network is connected to the spatial feature map F output by the input layer of the spatial feature cutting network and cut into 16 equal parts in the length and width directions to obtain 16 feature maps with dimensions 30×128×1×11. The i-th feature map with dimensions 30×128×1×11 at this time is denoted as F. i For each i = 1, 2, 3, ..., 16, the pooling layer of the spatial feature segmentation network is connected to the 16 feature maps with dimensions of 30×128×1×11 output from the segmentation layer of the spatial feature segmentation network, and feature extraction is performed on each of them to obtain 16 feature map outputs with dimensions of 30×128×1×1. The feature map f obtained after feature extraction on the i-th feature map with dimensions of 30×128×1×11 is then processed. i The 16 temporal feature extraction networks are connected one-to-one with the pooling layer output of the spatial feature segmentation network, each containing a 30×128×1×1 feature map. The input layer of the i-th temporal feature extraction network is connected to the i-th dimension 30×128×1×1 feature map f output by the pooling layer of the spatial feature segmentation network. i And output the feature map f output by the first mapping layer of the i-th temporal feature extraction network, which is connected to the input layer of the i-th temporal feature extraction network. i Feature extraction was performed to obtain a feature map a with dimensions of 30×128×1×1. iThe output is the feature map a, which is connected to the second mapping layer of the i-th temporal feature extraction network and the output of the first mapping layer of the i-th temporal feature extraction network. i First, feature extraction is performed to obtain a feature map b with dimensions of 30×128×1×1. i Then feature map a i All dimensions and feature maps b i Subtracting the corresponding dimensions yields a feature map c with dimensions 30×128×1×1. i The output is the feature map c output by the second mapping layer of the i-th temporal feature extraction network, which is connected to the third mapping layer of the i-th temporal feature extraction network. i Then, feature extraction is performed to obtain a feature map d with dimensions of 30×128×1×1. i The output is the feature map d output by the 3rd mapping layer of the i-th temporal feature extraction network, which is connected to the temporal feature selection layer of the i-th temporal feature extraction network. i And perform feature extraction to obtain a feature map e with dimensions of 1×128×1×1. i The output is the feature map e output by the time feature selection layer of the i-th time feature extraction network, which is connected to the independent mapping layer of the i-th time feature extraction network. i And perform feature extraction to obtain a feature map v with dimensions of 1×128×1×1. i Output: 16 feature maps of 30×128×1×1 dimensions output by the pooling layer of the spatial feature selection network, corresponding one-to-one with the input layer of the spatial feature cutting network. The i-th spatial feature selection network's input layer outputs the i-th dimension 30×128×1×1 feature map f from the pooling layer of the spatial feature cutting network. i And output the feature map f output by the feature selection layer of the i-th spatial feature selection network, which is connected to the input layer of the i-th spatial feature selection network. i Feature extraction is then performed to obtain a feature map A with dimensions of 1×128×1×1. i The output is the feature map A output by the feature selection layer of the i-th spatial feature selection network, which is connected to the independent mapping layer of the i-th spatial feature selection network. i And perform feature extraction to obtain a feature map s with dimensions of 1×128×1×1. i Output; Feature map v i and feature map s i That is, the gait feature map output by the gait relationship network;

[0021] Step 4: Train the constructed gait relationship network. Specifically, first, divide the training dataset obtained in Step 2 into a training set and a test set. Data under 74 pedestrian identities is used as the training set, and data under the remaining 50 pedestrian identities is used as the test set. During each training iteration, randomly select 42 gait image sequences from the training set, and then randomly select 30 gait images from each gait image sequence as input to the gait relationship network. If the number of gait images in a certain gait image sequence is less than 30, randomly repeat the selection until 30 gait images are obtained. During each training process, the parameters of the gait relationship network are optimized using a loss function until the training iteration reaches 200,000 times, resulting in a trained gait relationship network. The loss function is set to the Batch All (BA+) Triplet loss disclosed in the literature [A. Hermans, L. Beyer, B. Leibe, In defense of the triplet loss for person re-identification, CoRR abs / 1703.07737 (2017).]. The Cross-Entropy Loss method was disclosed in the literature [Hao Luo, Youzhi Gu, Xingyu Liao, Shenqi Lai, and WeiJiang. Bag of tricks and a strong baseline for deep person re-identification. In CVPR Workshops, June 2019, 2, 3].

[0022] Step 5: When the pedestrian to be identified needs to undergo gait recognition, a gait image sequence of the pedestrian to be identified is processed in the same way as in Step 2, so that the size of each gait image in the gait image sequence is 64×44 (length×width). Then, the processed gait image sequence is input into the trained gait relationship network to obtain 32 gait feature maps of the pedestrian to be identified.

[0023] Step 6: Perform the same operation as step 5 on all gait image sequences in the pre-established gait database that includes the identity of the pedestrian to be identified, to obtain 32 gait feature maps for each pedestrian identity. Then, calculate the Euclidean distance between the 32 feature maps obtained in step 5 and the 32 gait feature maps for each pedestrian identity obtained at this time, and obtain the pedestrian identity with the smallest Euclidean distance to the 32 feature maps obtained in step 5. This pedestrian identity is the identity of the pedestrian to be identified.

[0024] To evaluate the effectiveness of the gait relation network in the gait recognition method of this invention, the gait relation network (GaitRN) of this invention was compared with several other existing recognition models. Specific comparison data are shown in Table 1:

[0025] Table 1

[0026]

[0027] Analyzing the data in Table 1, we can see that the gait relationship network of this invention, under the LT training condition, and under the NM, BG, and CL conditions, achieves average rank-1 accuracies of 97.1%, 93.7%, and 79.1% on the CASIA-B dataset, respectively. Compared with the recent method GaitPart, which also improves the spatiotemporal feature extraction method, the gait relationship network of this invention improves accuracy by 0.9%, 2.2%, and 0.4%, respectively, significantly enhancing recognition progress.

[0028] The practicality of the gait recognition method based on gait relationship networks of this invention is analyzed as follows: In real-world scenarios, due to occlusion and other reasons, the obtained gait images may exhibit the following characteristics: 1) The order of the gait images is shuffled; 2) The number of gait images is limited; 3) The gait images may originate from different gait image sequences under the same walking conditions and with the same viewpoint; 4) The gait images may originate from gait image sequences under different walking conditions and with different viewpoints. Based on the CASIA-B dataset, the recognition accuracy of the gait relationship network of this invention under these conditions is experimentally tested:

[0029] 1) In real-world scenarios, the order of obtained gait images may be uncertain. Therefore, we shuffle the order of gait images in the test gait image dataset to simulate this situation. The accuracy of the gait relationship network of this invention and several existing gait recognition models under shuffled and unshuffled gait image conditions is shown in Table 2:

[0030] Table 2

[0031]

[0032] Analysis of the data in Table 2 shows that the test results of the gait relationship network of the present invention are consistent with those of the gait image order when the order of the gait images is shuffled and when the order of the gait images is not shuffled. This indicates that the gait relationship network of the present invention successfully resists the influence of gait image disorder. Even with the input of disordered gait images, the motion features of pedestrians can still be extracted.

[0033] 2) In crime scenarios, criminals often evade cameras, resulting in very few gait images being available. To simulate this situation, a fixed number of gait images were randomly selected as test inputs, and the accuracy of GaitSet, GaitPart, and GaitRN was tested under these conditions. The average of 10 experimental results was taken as the test result, as shown in Table 3.

[0034] Table 3

[0035]

[0036] Since GaitPart requires sequential input of gait images, tests were conducted under both sequential (GaitPart_Sort) and scrambled (GaitPart_Shuffle) input conditions. Because GaitSet and GaitRN are based on unordered gait image input, their accuracy is independent of the order of the input gait images. Analysis of Table 3 shows that, with the same number of test gait images, GaitRN of this invention significantly outperforms GaitSet and GaitPart, and GaitSet performs even better than GaitPart. This is because the micro-motion model proposed by GaitPart requires complete gait image sequences for short-term modeling, making it unsuitable for situations where gait images are missing or scrambled. In contrast, GaitRN of this invention does not have gait image input order constraints and fully utilizes the changing relationships between each frame of gait images, exhibiting good robustness even with a small number of gait images. Under NM conditions, the GaitRN of this invention can achieve an accuracy of 87.4% with only 7 gait images as input, which is 3.6% higher than the GaitSet method. Furthermore, the recognition accuracy of the GaitRN of this invention also increases monotonically with the increase of the number of gait images, indicating that the GaitRN of this invention can make full use of time information and has strong practicality.

[0037] 3) Here we studied a scenario where a person's gait images were collected from different gait image sequences from the same viewpoint. Gait images of the pedestrian were selected from these different gait image sequences for recognition. The detectors and database in the test set were re-divided, with NM(04-06) used as detectors and NM(01-03) retained as the database. We obtained the samples to be predicted by combining sequences from the same viewpoint in the NM(04-06) case. Then, 10 frames of gait images were randomly selected from the samples to be predicted as the final test samples. The average of the 10 experimental results was taken as the test result, as shown in Table 4.

[0038] Table 4

[0039]

[0040] Analysis of the data in Table 4 shows that the test accuracy of this invention and GaitSet is similar to that of a single gait image sequence. This indicates that: 1. Under the same viewpoint and walking conditions, gait images from different gait image sequences can also form a gait cycle. Gait has a special periodicity, and images from different gait image sequences can supplement the data within the gait cycle; 2. This invention can make full use of gait images from multiple gait image sequences under the same viewpoint and walking conditions, without requiring the continuity and completeness of the image sequence.

[0041] 4) In real life, gait images are likely obtained from different angles, different gait image sequences, and different walking conditions. To simulate this situation, we regenerate the samples to be predicted by combining different gait image sequences, gait images from different viewpoints, and different walking conditions. We randomly select 20 frames of gait images from the samples to be predicted as the final test samples, and take the average of 50 experimental results as the test result. The test results are shown in Table 5.

[0042] Table 5

[0043]

[0044] Analysis of Table 5 shows that the experiment demonstrates that, in a gait dataset containing multiple viewpoints and gait states, the accuracy of this invention reaches 97.6%, which is 3.4% higher than the GaitSet model. When only gait image data is available under BG and CL conditions, the accuracy exceeds the GaitSet model by 5.6%. Furthermore, we found that the accuracy under multi-viewpoint conditions is significantly higher than that under single-viewpoint conditions, indicating that this invention fully utilizes multi-viewpoint gait images. Adding gait image data under BG or NM conditions to the gait dataset with only CL conditions improves the accuracy, demonstrating that this invention can utilize gait images under various gait conditions. This demonstrates that this invention can fully utilize gait image data and still exhibits strong robustness under constrained real-world conditions.

Claims

1. A gait recognition method based on gait relationship networks, characterized in that... Includes the following steps: Step 1: Obtain a gait dataset from the CASIA-B gait dataset of the Biometrics and Security Technology Research Center. This gait dataset contains gait images of 124 pedestrians under 3 walking conditions and 11 viewpoints. The 124 pedestrians are labeled with numbers 001-124. The 3 walking conditions are normal walking, backpack walking, and walking while wearing a coat or jacket. The 11 viewpoints are 0°, 18°, 36°, 54°, 72°, 90°, 108°, 126°, 144°, 162°, and 180°. In this gait dataset, each pedestrian has 6 sequence sets under the normal walking condition, 2 sequence sets under the backpack walking condition, and 2 sequence sets under the coat or jacket walking condition. Each sequence set includes 11 gait image sequences. The 11 gait image sequences were taken from 11 viewpoints, and there is a one-to-one correspondence between the 11 viewpoints and the 11 gait image sequences. Each gait image sequence consists of several gait images. Step 2: Process each gait image in the gait dataset using the Takemura method to make each gait image have a height of H and a width of W, where H = 64 and W = 44. This gives you the training dataset. Step 3: Construct a gait relationship network, which includes one spatial feature extraction network, 16 temporal feature extraction networks, and 16 spatial feature selection networks. The spatial feature extraction network includes one global encoding network, one image segmentation network, four local encoding networks, one fusion encoding network, and one spatial feature segmentation network. The global encoding network includes one input layer, four convolutional layers, one max-pooling layer, and two output layers. The first convolutional layer of the global encoding network uses 32 convolutional kernels of size 5×5 with a zero-padding parameter of 2 and a stride of 1. The second convolutional layer of the global encoding network uses... The global coding network is implemented using 32 convolutional kernels of size 3×3, zero-padding parameter value of 1, and stride of 1. The third convolutional layer of the global coding network is implemented using 64 convolutional kernels of size 3×3, zero-padding parameter value of 1, and stride of 1. The fourth convolutional layer of the global coding network is implemented using 64 convolutional kernels of size 3×3, zero-padding parameter value of 1, and stride of 1. The stride of the max pooling layer of the global coding network is set to 2. The image segmentation network includes one input layer, one segmentation layer, and four output layers. Each local coding network includes one input layer, four convolutional layers, one max pooling layer, and two output layers. The first convolutional layer of each local coding network is implemented using 32 convolutional kernels of size 5×5, with a zero-padding parameter of 2 and a stride of 1. The second convolutional layer of each local coding network is implemented using 32 convolutional kernels of size 3×3, with a zero-padding parameter of 1 and a stride of 1. The third convolutional layer of each local coding network is implemented using 64 convolutional kernels of size 3×3, with a zero-padding parameter of 1 and a stride of 1. The fourth convolutional layer of each local coding network is implemented using 64 convolutional kernels of size 3×3, with a zero-padding parameter of 1 and a stride of 1. The max pooling layer of each local coding network... The stride of each layer is set to 2. The fusion coding network includes 10 input layers, 2 feature concatenation layers, 3 feature fusion layers, 2 convolutional layers, 1 max pooling layer, and 1 output layer. The first convolutional layer of the fusion coding network is implemented with 128 convolutional kernels of size 3×3, zero-padding parameter value of 1, and stride of 1. The second convolutional layer of the fusion coding network is implemented with 128 convolutional kernels of size 3×3, zero-padding parameter value of 1, and stride of 1. The stride of the max pooling layer of the fusion coding network is set to 2. The spatial feature segmentation network includes 1 input layer, 1 segmentation layer, 1 pooling layer, and 1 output layer.Each of the aforementioned temporal feature extraction networks includes one input layer, three mapping layers, one temporal feature selection layer, one independent mapping layer, and one output layer. The three mapping layers and the independent mapping layer of the temporal feature extraction network are each fully connected layers with 128 input channels and 128 output channels. The temporal feature selection layer of the temporal feature extraction network is implemented using the max function. Each of the aforementioned spatial feature selection networks includes one input layer, one feature selection layer, one independent mapping layer, and one output layer. The independent mapping layer of each spatial feature selection network is each a fully connected layer with 128 input channels and 128 output channels. When training the gait relationship network, the input layer of the global encoding network receives a gait image sequence with dimensions of 30×1×64×44, where the dimension is represented as number of images × number of channels × image length × image width. The first convolutional layer of the global encoding network receives the gait image sequence with dimensions of 30×1×64×44 output from the input layer of the global encoding network, and performs feature extraction to obtain a feature map F with dimensions of 30×32×64×44. global_1 The output is the feature map F output by the first convolutional layer of the global encoding network, which is connected to the second convolutional layer of the global encoding network. global_1 Feature extraction was performed to obtain a feature map F with dimensions of 30×32×64×44. global_2 The output, the max pooling layer of the global encoding network, is connected to the feature map F output by the second convolutional layer of the global encoding network. global_2 Feature extraction was performed to obtain a feature map F with dimensions of 30×32×32×22. global_pool The output is the feature map F output by the third convolutional layer of the global encoding network, which is connected to the max pooling layer of the global encoding network. global_pool Feature extraction was performed to obtain a feature map F with dimensions of 30×64×32×22. global_3 The output is the feature map F output by the third convolutional layer of the global encoding network, which is connected to the fourth convolutional layer of the global encoding network. global_3 Feature extraction was performed to obtain a feature map F with dimensions of 30×64×32×22. global_4 The output is the feature map F output by the first output layer of the global encoding network, which is connected to the max pooling layer of the global encoding network. global_pool And output, the second output layer of the global encoding network is connected to the feature map F output by the fourth convolutional layer of the global encoding network. global_4 The image segmentation network's input layer receives and outputs a gait image sequence with dimensions of 30×1×64×44. The image segmentation network's segmentation layer receives the gait image sequence output from its input layer (30×1×64×44) and segments it at the midpoints of its length and width to obtain four gait image sub-sequences with dimensions of 30×1×32×22. The image segmentation network's four output layers each receive and output the four gait image sub-sequences with dimensions of 30×1×32×22. Four local coding networks each receive and output the corresponding gait image sub-sequences. The image segmentation network described above outputs four gait image subsequences with dimensions of 30×1×32×22 from its four output layers. The input layer of the Nth local coding network is then fed into and outputs the gait image subsequence with dimensions of 30×1×32×22 from its Nth output layer. N = 1, 2, 3, 4. The first convolutional layer of the Nth local coding network is fed into and outputs the gait image subsequence with dimensions of 30×1×32×22 from its input layer, and feature extraction is performed to obtain a feature map with dimensions of 30×32×32×22. The second convolutional layer of the Nth local coding network is then fed into and outputs the gait image subsequence with dimensions of 30×1×32×22 from its input layer. The feature maps output by the first convolutional layer of the Nth local coding network are processed, and feature extraction is performed to obtain a feature map output with dimensions of 30×32×32×22. The max pooling layer of the Nth local coding network is connected to the feature map output by the second convolutional layer of the Nth local coding network, and feature extraction is performed to obtain a feature map output with dimensions of 30×32×16×11. The third convolutional layer of the Nth local coding network is connected to the feature map output by the max pooling layer of the Nth local coding network, and feature extraction is performed to obtain a feature map output with dimensions of 30×64×16×11. The fourth convolutional layer of the Nth local coding network... The feature map output from the third convolutional layer of the Nth local coding network is input into the convolutional layer, and feature extraction is performed to obtain a feature map output with dimensions of 30×64×16×11. The first output layer of the Nth local coding network is used to input the feature map output from the max pooling layer of the Nth local coding network and outputs the feature map. The second output layer of the Nth local coding network is used to input the feature map output from the fourth convolutional layer of the Nth local coding network and outputs the feature map. In the four local coding networks, the feature map output from the first output layer and the feature map output from the second output layer of the first local coding network are denoted as F. local_pool,1 and F local_4,1 Let F be the feature map output by the first output layer and the feature map output by the second output layer of the second local coding network. local_pool,2 and F local_4,2 Let F be the feature map output by the first output layer and the feature map output by the second output layer of the third local coding network. local_pool,3 and F local_4,3 Let F be the feature map output by the first output layer and the feature map output by the second output layer of the fourth local coding network. local_pool,4 and F local_4,4 The 10 input layers of the fusion coding network are connected one-to-one to the feature map F output by the first output layer of the global coding network. global_pool The feature map F output by the second output layer of the global coding network. global_4 The feature maps F output by the first and second output layers of the four local coding networks. local_pool,1 Feature map F local_4,1 Feature map F local_pool,2 Feature map F local_4,2 Feature map F local_pool,3 Feature map F local_4,3 Feature map F local_pool,4 and feature map F local_4,4 The first feature concatenation layer of the fusion coding network outputs the input feature map F. local_pool,1 Feature map F local_pool,2 Feature map F local_pool,3 and feature map F local_pool,4 After arranging and splicing the pieces according to the previously cut positions, a feature map F with dimensions of 30×32×32×22 is obtained. local_pool,all The output is the second feature concatenation layer of the fusion coding network connected to the feature map F. local_4,1 F local_4,2 F local_4,3 and F local_4,4 After being arranged and spliced ​​according to the previously corresponding cutting positions, a feature map F with dimensions of 30×64×32×22 is obtained. local_4,all The output is the first feature fusion layer of the fusion coding network connected to the feature map F. local_pool,all and feature map F local_4,all and feature map F local_pool,all and feature map F local_4,all Dimensional concatenation is performed, followed by feature fusion using a 1×1 convolutional kernel to obtain a feature map F with dimensions of 30×64×32×22. local The output is the second feature fusion layer of the fusion coding network accessing the feature map F. global_pool and feature map F global_4 The two are then concatenated dimensionally, and features are fused using a 1×1 convolutional kernel to obtain a feature map F with dimensions of 30×64×32×22. global The output, the third feature fusion layer of the fusion coding network, is connected to the feature map F output by the first feature fusion layer of the fusion coding network. local The feature map F output by the second feature fusion layer of the fusion coding network. global Then, the two are added one-to-one according to their pixel positions to obtain a feature map F with dimensions of 30×64×32×22. fuse Output, i.e., F fuse =F global +F local The first convolutional layer of the fusion coding network is connected to the feature map F output by the third feature fusion layer of the fusion coding network. fuse Feature extraction was performed to obtain a feature map F with dimensions of 30×128×32×22. fuse_1 Output: The second convolutional layer of the fusion coding network is connected to the feature map F output by the first convolutional layer of the fusion coding network. fuse_1 Feature extraction was performed to obtain a feature map F with dimensions of 30×128×32×22. fuse_2 The output, the max pooling layer of the fusion coding network, is connected to the feature map F output by the second convolutional layer of the fusion coding network. fuse_2 The spatial feature map F with dimensions of 30×128×16×11 is extracted and output. The output layer of the fusion coding network is connected to the spatial feature map F output by the max pooling layer of the fusion coding network and output. The input layer of the spatial feature cutting network is connected to the spatial feature map F with dimensions of 30×128×16×11 output by the output layer of the fusion coding network and output. The cutting layer of the spatial feature cutting network is connected to the spatial feature map F output by the input layer of the spatial feature cutting network and cut into 16 equal parts in the length and width directions to obtain 16 feature maps with dimensions of 30×128×1×11. The i-th feature map with dimensions of 30×128×1×11 is denoted as F. i Let i = 1, 2, 3, ..., 16. The pooling layer of the spatial feature segmentation network is connected to the 16 feature maps with dimensions of 30×128×1×11 output from the segmentation layer of the spatial feature segmentation network, and feature extraction is performed on each of them to obtain 16 feature map outputs with dimensions of 30×128×1×1. The feature map f obtained after feature extraction on the i-th feature map with dimensions of 30×128×1×11 is then processed. i The 16 temporal feature extraction networks are connected one-to-one with the pooling layer output of the spatial feature segmentation network, which outputs 16 feature maps of 30×128×1×1 dimensions. The input layer of the i-th temporal feature extraction network is connected to the i-th dimension 30×128×1×1 feature map f output by the pooling layer of the spatial feature segmentation network. i And output, the first mapping layer of the i-th temporal feature extraction network is connected to the feature map f output by the input layer of the i-th temporal feature extraction network. i Feature extraction was performed to obtain a feature map a with dimensions of 30×128×1×1. i The output is the feature map a output by the first mapping layer of the i-th temporal feature extraction network, which is connected to the second mapping layer of the i-th temporal feature extraction network. i First, feature extraction is performed to obtain a feature map b with dimensions of 30×128×1×1. i Then feature map a i All dimensions and feature maps b i Subtracting the corresponding dimensions yields a feature map c with dimensions 30×128×1×1. i The output is the feature map c output by the third mapping layer of the i-th temporal feature extraction network. i Then, feature extraction is performed to obtain a feature map d with dimensions of 30×128×1×1. i The output is the feature map d output by the third mapping layer of the i-th temporal feature extraction network, which is connected to the temporal feature selection layer of the i-th temporal feature extraction network. i And perform feature extraction to obtain a feature map e with dimensions of 1×128×1×1. i The output is the feature map e output by the time feature selection layer of the i-th time feature extraction network, which is connected to the independent mapping layer of the i-th time feature extraction network. i And perform feature extraction to obtain a feature map v with dimensions of 1×128×1×1. i Output: 16 feature maps of 30×128×1×1 dimensions output by the pooling layer of the spatial feature selection network, corresponding one-to-one with each of the 16 mentioned spatial feature selection networks. The input layer of the i-th mentioned spatial feature selection network is connected to the i-th dimension 30×128×1×1 feature map f output by the pooling layer of the spatial feature cutting network. i And output, the feature selection layer of the i-th spatial feature selection network is connected to the feature map f output by the input layer of the i-th spatial feature selection network. i Feature extraction is then performed to obtain a feature map A with dimensions of 1×128×1×1. i The output is the feature map A output by the feature selection layer of the i-th spatial feature selection network, which is connected to the independent mapping layer of the i-th spatial feature selection network. i And perform feature extraction to obtain a feature map s with dimensions of 1×128×1×1. i Output; Feature map v i and feature map s i That is, the gait feature map output by the gait relationship network; Step 4: Train the constructed gait relationship network. Specifically, first, divide the training dataset obtained in Step 2 into a training set and a test set. Data under 74 pedestrian identities is used as the training set, and data under the remaining 50 pedestrian identities is used as the test set. During each training, randomly select 42 gait image sequences from the training set, and then randomly select 30 gait images from each gait image sequence as the input to the gait relationship network. If the number of gait images in a certain gait image sequence is less than 30, randomly repeat the selection until 30 gait images are obtained. During each training process, the parameters of the gait relationship network are optimized using a loss function until the training is iterated 200,000 times to obtain the trained gait relationship network. The loss function is set as Cross-Entropy Loss. Step 5: When the pedestrian to be identified needs to undergo gait recognition, a gait image sequence of the pedestrian to be identified is processed in the same way as in Step 2, so that the size of each gait image in the gait image sequence is 64×44 (length×width). Then, the processed gait image sequence is input into the trained gait relationship network to obtain 32 gait feature maps of the pedestrian to be identified. Step 6: Perform the same operation as step 5 on all gait image sequences in the pre-established gait database that includes the identity of the pedestrian to be identified, to obtain 32 gait feature maps for each pedestrian identity. Then, calculate the Euclidean distance between the 32 feature maps obtained in step 5 and the 32 gait feature maps for each pedestrian identity obtained at this time, and obtain the pedestrian identity with the smallest Euclidean distance to the 32 feature maps obtained in step 5. This pedestrian identity is the identity of the pedestrian to be identified.

Images