A multi-view gait recognition method and system

Abstract

The present invention discloses a multi-view gait recognition method and system based on feature extraction of two orthogonal viewing directions. The method has the following specific steps: S1. Collecting a gait data set of multi-view and multi-covariate conditions and preprocessing the data set; S2. Sending the preprocessed input to a trained feature extraction network to extract feature vectors of components of the two viewing directions of 90° and 0°; at the same time, identifying the covariate of the sample according to the input sample and obtaining the covariate vector; S3. According to the obtained covariate vector, weighting processing is performed on the feature vector of step S2 to obtain the final gait feature vector; S4. Setting a registration set, which only contains samples of the two viewing directions of 90° and 0°; S5. By comparing the gait features output by the network with the gait features of the registration set, the input identity information is classified. The present invention achieves the purpose of gait recognition in multi-view and multi-carrying states, and effectively reduces the influence of covariates on gait recognition.

Description

Technical Field

[0001] The present invention belongs to the technical field of gait recognition, and specifically relates to a multi-view gait recognition method and system based on deep learning of feature separation extraction in two orthogonal viewing directions. Background Art

[0002] The accuracy of gait recognition is easily affected by factors such as clothing, shooting angle, speed, and light intensity. The researchers' work mainly focuses on better extracting more effective gait features that are independent of covariates.

[0003] At present, gait recognition algorithms are mainly divided into two categories: non-deep learning and deep learning gait recognition.

[0004] Related research on non-deep learning gait recognition algorithms mainly relies on manually designed features. Technicians extract human body contours from images, use different human gait representation methods to build models to represent contour sequences, and then use different learning methods to extract feature information. Finally, the identity of the subject is identified through classifiers such as the K-Nearest Neighbor (KNN) algorithm.

[0005] After 2015, deep learning emerged and the concept of deep learning was also introduced into the field of gait recognition. Currently, most gait recognition models are built using deep learning methods. According to the different input types, deep learning gait recognition algorithms can be divided into three categories: algorithms based on synthetic images, algorithms based on video frame sequences, and algorithms based on skeleton models.

[0006] The input of the algorithm based on synthetic images is generally the gait energy image (GEI), which is obtained by adding and averaging all the contour images of a gait cycle of the target. The following formula is the gait energy image of a gait cycle of N frames, where X t Represents the image at time t:

[0007]

[0008] The algorithm based on synthetic graphs ignores the information of the time dimension and has low accuracy, but it integrates gait information into one graph, and the constructed model is simple and has fewer parameters.

[0009] The algorithm based on video frame sequence is the current mainstream algorithm. It contains temporal gait information and spatial gait information, and can achieve higher recognition accuracy, but there is still room for improvement, and it is easily affected by covariate factors such as clothing conditions.

[0010] The algorithm based on the skeleton model does not regard gait recognition as image classification. It obtains gait information by modeling the human body. Therefore, this type of algorithm is more robust to problems such as changes in clothing and occlusion. However, the current recognition accuracy cannot reach that of the algorithm based on video frame sequences. Summary of the invention

[0011] In view of the above-mentioned defects of the prior art, the present invention aims to improve the recognition accuracy. Based on the mathematical principle of vector decomposition, a multi-view gait recognition method and system based on deep learning with separate feature extraction in two orthogonal viewing directions are proposed, and according to different input types, a model with gait energy map as input and a model with video frame sequence as input are constructed.

[0012] The present invention provides a multi-view gait recognition method and system based on vector decomposition, which decomposes a gait video frame image under a certain viewing angle into frame images in two viewing angle directions of 90° and 0° to extract gait features, and then performs weighted splicing processing on the obtained gait features.

[0013] To achieve the above object, the present invention adopts the following technical solutions:

[0014] A multi-view gait recognition method is performed according to the following steps:

[0015] S1. Collect a gait video dataset with multiple perspectives and multiple covariates, and preprocess the dataset;

[0016] S2. Input the preprocessed data into the feature extraction network to extract and obtain the feature vectors of the two viewing angle components of 90° and 0°;

[0017] S3. In parallel with step S2, the covariates of the samples are identified according to the input samples, and the covariate vector is obtained. This step is responsible for identifying the input covariates and obtaining the corresponding covariate vector. There are two common types of covariates, namely, different perspectives and different states of the objects carried.

[0018] S4. According to the obtained covariate vector, weighted processing is performed on the feature vector obtained in step S2 to obtain the final gait feature vector;

[0019] S5. Setting the registration set to include complete gait information of each target. Different from the existing registration set including samples of various viewing angles, the registration set of the present invention only includes samples of two viewing angles of 90° and 0°.

[0020] S6. By comparing the gait features output by the network with the gait features of the registration set, the input identity information is obtained through classification. Preferably, a classifier is used to classify the feature vector to obtain the identity information (Identification, ID) of the sample.

[0021] According to the different inputs, the present invention constructs a gait recognition model whose input is a gait energy graph and a gait recognition model whose input is a video frame sequence. The block diagrams of the two models are as follows: Figure 2 and Figure 3 shown.

[0022] The technical solution of the present invention effectively realizes gait recognition in multiple viewing angles and with multiple items carried.

[0023] As a preferred solution, step S1 specifically includes:

[0024] The video frames in the collected gait dataset are decomposed at the frame level to obtain the frame-level gait images, and the frame-level gait images are binarized and cropped to obtain the gait silhouette images. For model 2, the obtained gait silhouette image sequence can be used as input to enter the model network. For model 1, the gait contour image of one cycle also needs to be GEI normalized before it can be used as input to enter the model network.

[0025] As a preferred solution, step S2 is the core of the present invention, and its task is to extract feature vectors of the two viewing angles of 90° and 0°. According to different inputs, the present invention proposes two different ideas:

[0026] For model 1, since the input is a gait energy map, the feature extraction network can be constructed relatively simply. Inspired by GaitNet, the present invention uses an encoder-decoder model to extract 256×1-dimensional feature vectors in the two viewing directions of 90° and 0°. Among them, the first 128 dimensions are the gait features of the 90° viewing direction component, and the last 128 dimensions are the gait features of the 0° viewing direction component. The training loss function of the Encoder-Decoder model includes three parts: reconstruction loss, identity recognition loss, and feature partitioning loss:

[0027] ①Reconstruction losses

[0028] This loss is a common loss in the Encoder-Decoder model. For the present invention, the input gait energy map X is passed through the encoder E to obtain the gait feature f of the 90° viewing angle direction component. 90° and the gait characteristics of the 0° visual angle direction component f 0° , the decoder is responsible for converting f 90° and f 0° The two gait features are restored to the input gait energy graph X′:

[0029] (f 90° ,f 0° )=E(X)

[0030] X′=D(f 90° ,f 0° )

[0031] The decoded output X′ should be similar to the original input X, so the reconstruction loss is expressed as:

[0032]

[0033] in, Represents the Euclidean distance metric.

[0034] ②Identity loss

[0035] In order to ensure that the Encoder-Decoder model can effectively extract identity information, the present invention adds a linear classifier C in the training stage. k Auxiliary training is performed to ensure that the extracted feature vector can extract the identity information of the target. The cross entropy loss function is used and regarded as the identity recognition loss:

[0036] L identify = -log(C K (f 90° ,f 0° ))

[0037] ③Feature partition loss

[0038] To ensure effective separation of features from 90° and 0° viewing angle components, consider:

[0039] When the input is a training sample with a 90° viewing angle, the feature vector of the last 128 dimensions is set to 0, and the decoder outputs X′:

[0040] X′=D(f 90° ,0)

[0041] Should be similar to the original input X′, that is:

[0042]

[0043] When the input is a training sample with a 0° viewing angle, the feature vector of the first 128 dimensions is set to 0, and the decoder outputs X′:

[0044] X′=D(0,f 0° )

[0045] Should be similar to the original input X′, that is:

[0046]

[0047] When the input is a training sample that is not a 90° or 0° viewing angle, the following two operations are performed: (1) the feature vector of the last 128 dimensions is set to 0; (2) the feature vector of the last 128 dimensions is set to 0. And according to the different input viewing angles v, different weights λ are assigned before the Euclidean distance measurement. 1 and λ 2 , and thus obtain the feature partition loss:

[0048]

[0049] Therefore, the training loss function of the entire feature extraction network is:

[0050] L=L recon +L identify +L divide

[0051] For model 2, since the input is a sequence of video frames, the network model is more complex than model 1. The input is sent to the feature extraction network of the 90° viewing angle and the feature extraction network of the 0° viewing angle respectively, and the component features of the two viewing angles with a dimension of 128×62 are obtained, which are then concatenated to obtain the gait features of 256×62 dimensions.

[0052] The feature extraction network of model 2 uses the GaitSet backbone network and is trained using the improved triplet loss function. The original triplet loss function is as follows:

[0053]

[0054] in, It is the characteristic expression of anchor. is the characteristic expression of the positive example, is the characteristic expression of negative examples, and α is the margin of the triplet loss function, which is a hyperparameter. The larger the value of α, the better it can distinguish similar images, but it will also cause the network to not converge.

[0055] In order to extract the feature vectors of the two viewing directions of 90° and 0°, the triplet loss function is improved as follows:

[0056]

[0057]

[0058] In addition, the anchor selection method is changed. During the training process of the 90° feature extraction network, only the features of the 90° contour map are used as anchors. During the training process of the 0° feature extraction network, only the features of the 0° contour map are used as anchors.

[0059] As a preferred solution, step S3 specifically includes:

[0060] ① Identify the covariates of the input samples. The covariates can be several categories that act independently of each other, such as the viewing angle, the carrying conditions, the light intensity, etc. of the input samples.

[0061] ② After identifying the covariate, obtain the corresponding covariate vector. Different covariates correspond to different covariate vectors.

[0062] As a preferred solution, step S4 specifically includes: performing weighted processing on the feature vector according to the difference of the identified covariates to obtain the final gait feature vector.

[0063] Step S3 is to better identify the identity of the target based on step S2. It can be seen that the change of the perspective and the difference of the carried objects will affect the gait characteristics of the input sample. The present invention assumes that the change of the perspective and the difference of the carried objects are two independent situations, that is, the effect of the difference of the perspective on the gait characteristics and the effect of the difference of the carried objects on the gait characteristics are independent of each other, and their corresponding covariate vectors are unrelated.

[0064] The covariate vector of the present invention is used to weight the feature vector of the registration set.

[0065] It should be noted that the covariate vector needs to be trained in advance. Taking the CASIA-B database as an example, the database contains 11 perspectives and 3 carrying conditions (no carrying state (NM), backpack state (BG) and coat state (CL)). Therefore, it is necessary to train the covariate vectors of 11 perspectives and 2 carrying conditions.

[0066] As a preferred solution, step S5 is to process the features of the registration set.

[0067] The registration set of the present invention is composed of feature vectors of samples with a viewing angle of 90° without interference from other covariates and feature vectors of samples with a viewing angle of 0° without interference from other covariates, as a complete gait feature of a target.

[0068] Step S5 is as follows: samples with a viewing angle of 90° and no interference from other covariates are fed into the trained 90° feature extraction network to obtain The samples with a viewing angle of 0° and no interference from other covariates are fed into the trained 0° feature extraction network to obtain Then concatenate the two to get a registration set sample Finally, we get a registration set F containing N targets:

[0069]

[0070] As a preferred solution, step S6 uses a classifier to perform identity recognition on the feature vector output by the network.

[0071] As a preferred solution, the size of the gait energy map input in model 1 is 64×64; the size of the gait contour map sequence input in model 2 is 64×44.

[0072] The present invention proposes a decomposition method for extracting gait information from multi-view gait silhouette images in two viewing directions of 90° and 0°. The present invention proposes the concept of covariate vectors, and applies the covariate vectors to the feature vectors of the registration set, so that the recognition accuracy is greatly improved.

[0073] The present invention resets the registration set and discards the registration set samples except the two viewing angles of 90° and 0°, greatly reducing the storage capacity of the registration set; at the same time, the feature information contained in the two viewing angles of 90° and 0° is the closest to being unrelated - the present invention regards the two as unrelated - and by splicing these two unrelated gait feature information as samples of the registration set, the complete gait feature information of each target can be obtained.

[0074] The present invention achieves the purpose of gait recognition in multiple viewing angles and multiple carrying object states, and effectively reduces the influence of covariates on gait recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0075] Figure 1 It is a flowchart of the method of the present invention;

[0076] Figure 2 It is a flowchart of the model 1 constructed by the present invention;

[0077] Figure 3 It is a flowchart of Model 2 constructed by the present invention;

[0078] Figure 4 It is a flowchart of the GaitSet network;

[0079] Figure 5 It is a system block diagram of the present invention. DETAILED DESCRIPTION

[0080] The technical solution of the present invention is further explained below through preferred embodiments, which are specifically divided into two types: Model 1 and Model 2.

[0081] For the convenience of description, two covariates are set: multiple perspectives and multiple carrying conditions.

[0082] The embodiment of the present invention is based on a multi-view gait recognition method for extracting features from two orthogonal viewing directions separately, for model 1, such as Figure 2 As shown, the following steps are included:

[0083] S1. Collect a gait data set with multiple perspectives and multiple covariates, perform preprocessing operations on the gait data set, and align, crop, and normalize the obtained gait silhouette image to obtain a gait energy map.

[0084] S2. Input the initial gait energy map into the trained encoder and extract the feature vector f containing two viewing direction components of 90° (first 128 dimensions) and 0° (last 128 dimensions) p ′, which is a 256×1-dimensional feature vector.

[0085] S3. Input the initial gait energy graph into the covariate classification network to obtain the input covariate information, recorded as view and condition. Select the corresponding covariate vector P according to the two covariate classification conditions. view and P condition , and adding these two unrelated vectors gives:

[0086] P view+condition =P view +P condition

[0087] This is the covariate vector of the gait energy graph taken as input. view and P condition are all 256×1 dimensional vectors, and the obtained P view+condition It is also a 256×1 dimensional vector.

[0088] S4. The feature vector f output from step S2 p 'Do weighted processing:

[0089] f p =f p ′·P view+condition

[0090] Get the final gait feature vector f p .

[0091] S5. Model 1 sets the registration set:

[0092] For the registered set samples of a certain target: the samples with 90° viewing angle and no objects are input into the encoder, and the first 128-dimensional vector of the output feature vector is taken; the samples with 0° viewing angle and no objects are input into the encoder, and the last 128-dimensional vector of the output feature vector is taken. The two are concatenated to obtain the 256-dimensional feature vector of the target. If there are N targets in total, there are N feature vectors in the registered set.

[0093] S6. Use the nearest neighbor classifier to obtain the identity information of the input sample.

[0094] The embodiment of the present invention is based on a multi-view gait recognition method for extracting features from two orthogonal viewing directions separately, for model 2, such as Figure 3 As shown, the following steps are included:

[0095] S1. Collect a multi-view, multi-covariate gait dataset, binarize the gait dataset to obtain a gait silhouette image, and align and crop the gait silhouette image to use it as the input of the entire network.

[0096] S2. The preprocessed input is fed into the trained feature extraction network to extract the feature vectors f of the two viewing angle components of 90° and 0°, both with a dimension of 128×62 p-90° ′ and f p-0° ′, and concatenate the two to obtain a feature vector f with a dimension of 256×62 p ′.

[0097] The feature extraction network uses the GaitSet backbone network. It is a flexible and efficient gait recognition network proposed by Fudan University in 2016. Figure 4 As shown in the figure, the input of GaitSet is a series of preprocessed silhouette images of size 64×44, which are regarded as a sequence group. The main branch is a convolutional neural network used to perform convolution processing on each frame of the image. In this process, the multi-feature set pooling (Set Pooling, SP) module is used to aggregate the gait features of k frames of silhouette images into sequence-level features after every two layers; the main branch is connected in parallel with a full-process pipeline (Multilayer Global Pipeline, MGP) module to integrate sequence-level features of different layers. Finally, the multi-scale discriminant features are collected through the horizontal pyramid pooling (Horizontal Pyramid Pooling, HPP) module, and the gait features are finally output through an independent fully connected layer for easy classification.

[0098] S3. In order to make the covariate classification network as simple as possible, the input of the covariate classification network is set to the gait energy map that does not contain time dimension information.

[0099] The operations after homogenizing the gait silhouette image sequence of S1 to obtain the gait energy image are consistent with the specific implementation details of S3 and S4 of model 1. It should be noted that P in model 2 view and P condition are all 256×62 dimensional vectors, and the obtained P view+condition It is also a 256×62 dimensional vector.

[0100] S5. Model 2 sets the registration set:

[0101] For the registered set samples of a certain target: the samples with 90° viewing angle and no objects are input into the 90° feature extraction network, and the samples with 0° viewing angle and no objects are input into the 0° feature extraction network, and 128×62-dimensional feature vectors are obtained respectively. The two are concatenated to obtain the 256×62-dimensional feature vector of the target. If there are N targets in total, there are N feature vectors in the registered set.

[0102] S6. Obtain the input identity information by calculating the Euclidean distance between the output and the registered set samples.

[0103] From the above, we can see that model 1 has a simple network structure, fewer training parameters, and a fast training time, but because the input is a gait energy map, the accuracy is lower than that of model 2. Model 2 is more complex, has more parameters, and takes a long time to train, but has a higher recognition accuracy.

[0104] like Figure 5 As shown, this embodiment discloses a multi-view gait recognition system, which includes the following modules connected in sequence:

[0105] Preprocessing module: collects gait data sets with multiple perspectives and multiple covariates, and preprocesses the data sets;

[0106] Covariate vector identification module: the preprocessed input is sent to the trained feature extraction network to extract the feature vectors of the two viewing angle components of 90° and 0°; at the same time, the covariate of the sample is identified according to the input sample, and the covariate vector is obtained;

[0107] Weighted processing module: according to the obtained covariate vector, the feature vector is weighted to obtain the final gait feature vector;

[0108] Registration set setting module: set the registration set, which only contains samples in the two viewing directions of 90° and 0°;

[0109] Identity information acquisition module: By comparing the gait features output by the network with the gait features of the registration set, the input identity information is classified.

[0110] For other contents of this embodiment, reference may be made to the aforementioned embodiments.

[0111] In summary, the present invention is a multi-perspective gait recognition method and system based on the separate extraction of features from two orthogonal viewing directions, and the input enters two network branches in parallel after preprocessing. The first is the feature extraction network, which is responsible for extracting feature vectors from two orthogonal viewing directions; the second is the covariate identification network, which is responsible for identifying the covariate state of the input. The feature vectors in the two viewing directions obtained by the feature extraction network are spliced ​​to obtain the input gait features. At the same time, the features of the registration set are weighted according to the identified covariates, and then the gait features of the input sample are compared with the processed registration set features, thereby performing gait recognition. The present invention achieves the purpose of gait recognition in multi-perspective and multi-carrying states, and effectively reduces the influence of covariates on gait recognition.

[0112] The above shows and describes the basic principles and main features of the present invention and the advantages of the present invention. It should be understood by those skilled in the art that the present invention is not limited by the above embodiments, and the above embodiments and descriptions are only for explaining the principles of the present invention. Without departing from the spirit and scope of the present invention, the present invention may have various changes and improvements, and these changes and improvements all fall within the scope of the present invention to be protected.

Claims

1. A multi-view gait recognition method, characterized in that: The following steps are involved: S1. Collect a gait dataset with multiple perspectives and multiple covariates, and preprocess the dataset; S2. The preprocessed input is sent to the trained feature extraction network to extract the feature vectors of the two viewing angle components of 90° and 0°. Specifically, model 1 or model 2 is used to extract the feature vectors of the two viewing angles of 90° and 0°. For model 1, the input is a gait energy map, and an encoder-decoder model is used to extract the feature vectors of the two viewing angles of 90° and 0°. For model 2, the input is a video frame sequence, and the input is sent to the feature extraction network of the 90° viewing angle and the feature extraction network of the 0° viewing angle respectively to obtain the component features of the two viewing angles, and then spliced ​​to obtain the gait features. The feature extraction network of model 2 uses the GaitSet backbone network. At the same time, the covariate of the sample is identified according to the input sample, and the covariate vector is obtained. S3. According to the obtained covariate vector, weighted processing is performed on the feature vector of step S2 to obtain the final gait feature vector; S4. Set a registration set, which only contains samples in the two viewing directions of 90° and 0°; S5. By comparing the gait features output by the network with the gait features of the registration set, the input identity information is classified.

2. A multi-view gait recognition method according to claim 1, characterized in that: Step S1 specifically includes: The video frames in the collected gait dataset are decomposed at the frame level to obtain the frame-level gait images, and the frame-level gait images are binarized and cropped to obtain the gait silhouette images.

3. A multi-view gait recognition method according to claim 2, characterized in that: For model 1, the gait profile of one cycle is GEI-normalized and entered into the model network as input.

4. The multi-view gait recognition method according to claim 2, characterized in that: For model 2, the obtained gait silhouette image sequence is input into the model network.

5. A multi-view gait recognition method according to claim 3 or 4, characterized in that: In step S2: ① Identify the covariates of the input samples. The covariates are several categories that act independently of each other, including the viewing angle, the contents carried, and the light intensity of the input samples; ② After identifying the covariate, obtain the corresponding covariate vector. Different covariates correspond to different covariate vectors.

6. A multi-view gait recognition method according to claim 5, characterized in that: Step S3 specifically includes: According to the differences in the identified covariates, the feature vectors are weighted to obtain the final gait feature vector.

7. The multi-view gait recognition method according to claim 6, characterized in that: Step S4 is specifically as follows: The samples with a viewing angle of 90° and no interference from other covariates are fed into the trained 90° feature extraction network to obtain The samples with a viewing angle of 0° and no interference from other covariates are fed into the trained 0° feature extraction network to obtain Then concatenate the two to get a registration set sample Finally, we get a registration set F containing N targets:

8. The multi-view gait recognition method according to claim 7, characterized in that: Step S5 uses a classifier to perform identity recognition on the feature vector output by the network.

9. A multi-view gait recognition system, characterized in that: Includes the following modules: Preprocessing module: collects gait data sets with multiple perspectives and multiple covariates, and preprocesses the data sets; Covariate vector identification module: the preprocessed input is sent to the trained feature extraction network to extract the feature vectors of the two viewing angle components of 90° and 0°. Specifically, model 1 or model 2 is used to extract the feature vectors of the two viewing angles of 90° and 0°. For model 1, the input is the gait energy map, and an encoder-decoder model is used to extract the feature vectors of the two viewing angles of 90° and 0°. For model 2, the input is a video frame sequence, and the input is sent to the feature extraction network of the 90° viewing angle and the feature extraction network of the 0° viewing angle respectively to obtain the component features of the two viewing angles, and then spliced ​​to obtain the gait features. The feature extraction network of model 2 uses the GaitSet backbone network. At the same time, the covariates of the samples are identified according to the input samples, and the covariate vector is obtained. Weighted processing module: according to the obtained covariate vector, the feature vector is weighted to obtain the final gait feature vector; Registration set setting module: set the registration set, which only contains samples in the two viewing directions of 90° and 0°; Identity information acquisition module: By comparing the gait features output by the network with the gait features of the registration set, the input identity information is classified.

Images