A cross-view gait recognition method combining LSTM and CNN

Abstract

The invention discloses a cross-view gait recognition method combining LSTM with CNN, comprising the following steps: 1) extracting 3D posture data of pedestrians in a video from a CASIA-B gait video dataset using OpenPose3D; 2) extracting motion constraint data and joint action data of joint points of pedestrians in the video from the 3D posture data obtained in step 1), constructing a pedestrian gait constraint matrix and establishing a sample set based on the data; 3) constructing a 3D gait recognition network LC-POSEGAIT; 4) training the 3D gait recognition network LC-POSEGAIT using the training set samples obtained in step 2); 5) extracting gait feature vectors of pedestrians in the video using the trained 3D gait recognition network LC-POSEGAIT, and completing cross-view gait recognition combining LSTM with CNN. The method can realize cross-view gait recognition, avoid the preprocessing work of pedestrian detection and tracking, and alleviate the influence of shooting angle, carried objects, illumination and other factors on gait recognition.

Description

Technical Field

[0001] The present invention belongs to the field of machine vision and relates to a cross-view gait recognition method combining LSTM with CNN. Background Art

[0002] Biometrics recognition has a wide range of applications in security monitoring, pedestrian tracking, identity recognition and other fields. At present, a large number of video surveillance equipment are installed in public places, and the collected surveillance data is mostly used for post-event evidence collection, and rarely for identity verification of specific target pedestrians in an unaware, long-distance, and uncontrolled manner to support security warnings and alarms. Gait is the performance of a person's walking posture during the process of moving. It has the characteristics of non-contact, non-invasive, difficult to hide and forge. It is the only biometric feature that can be obtained at a long distance and in an uncontrolled state, and can support long-distance identity recognition. Therefore, gait recognition has broad application prospects.

[0003] Gait can only show significant characteristics in a set of movement cycles, and is easily disturbed by external environment (viewing angle, road surface, occlusion, etc.) and internal factors (carrying objects, clothing, etc.). In addition, the non-rigid movement of the human body makes the change of observation angle have a greater impact on the intra-class characteristics of pedestrian gait. The existing cross-view gait recognition method based on appearance modeling adopts energy graph modeling, which is easy to lose the spatiotemporal information of gait, and is easily affected by the changes in pedestrian clothing, carrying objects and body shape. The recognition performance is limited and it is only suitable for situations with limited or small viewing angle changes. Due to the three-dimensional nature of gait information, the movement changes of human joints in 3D space can more accurately describe gait characteristics. However, tracking human joint points in traditional 3D gait models is a difficult point, and the use of 3D equipment or multi-camera collaboration to reconstruct 3D gait models requires complex camera parameter adjustment and modeling calculations, and its application scenarios are limited. Summary of the invention

[0004] The purpose of the present invention is to overcome the shortcomings of the above-mentioned prior art and provide a cross-view gait recognition method combining LSTM and CNN, which can realize cross-view gait recognition and omit the preprocessing steps of pedestrian detection and tracking.

[0005] To achieve the above object, the cross-view gait recognition method combining LSTM and CNN described in the present invention comprises the following steps:

[0006] 1) From the CASIA-B gait video dataset, OpenPose3D is used to extract the 3D pose data of pedestrians in the video;

[0007] 2) extracting the motion constraint data and joint action data of the video pedestrian joint points from the 3D posture data obtained in step 1), constructing the pedestrian motion constraint matrix and establishing a sample set based on it;

[0008] 3) Construct a 3D gait recognition network LC-POSEGAIT that combines LSTM and CNN in parallel;

[0009] 4) Using the training set samples obtained in step 2) to train the 3D gait recognition network LC-POSEGAIT;

[0010] 5) Use the trained 3D gait recognition network LC-POSEGAIT to extract the gait feature vector of the pedestrian in the video, and measure the gait feature vector to achieve cross-view gait recognition. The specific operation of step 1) is: from the CASIA-B gait video dataset, use OpenPose3D to extract the 3D posture data of the pedestrian in the video in, Represents the 3D coordinates of the i-th joint point of the pedestrian in the m-th frame.

[0011] The specific operations of step 2) are:

[0012] 21) According to the joint point posture data of the pedestrian in the video, calculate the joint angle constraint R joint , foot angle constraint R foot , center of gravity and lower limb joint motion constraints R gra , human body structure constraints R step and human motion constraints R forw , to construct the 3D joint constraint feature matrix of the video pedestrian motion

[0013] 22) According to the joint point posture data of the pedestrian in the video, set the displacement vector sequence of the joint point in the time series m Express the motion features of pedestrian joints in the time domain and space domain to construct the pedestrian joint motion feature matrix

[0014] 23) setting a gait segmentation cycle, and determining a gait segmentation cycle from the first contact of the left foot of the pedestrian in the video with the ground to the second contact of the left foot with the ground;

[0015] 24) Extract the data of two gait segmentation cycles from the pedestrian joint constraint feature matrix and joint action feature matrix, and use them to construct a sample set and Then the gait standard sample and Divide into training set, registration set and test set;

[0016] The specific operations of step 3) are:

[0017] The 3D gait recognition network LC-POSEGAIT is established by combining LSTM network and CNN network in parallel. The LSTM network is used to extract the joint constraint feature matrix of the pedestrian in the video. The motion timing features of the CNN network are used to extract the joint action feature matrix The action space features of the LSTM network and the features extracted by the CNN network are fused through the fully connected layer;

[0018] The 3D gait recognition network LC-POSEGAIT adopts the Softmax loss function L s With the Center loss function L C The joint multi-loss function L = L s +λL C as a loss function.

[0019] The specific operations of step 5) are:

[0020] 51) The fully connected layer vector of the 3D gait recognition network LC-POSEGAIT is used as the pedestrian 3D gait feature vector, and the 3D gait feature vectors of the test set and the registration set samples are extracted respectively. Among them, the feature vector set of the test set The feature vector set of the registration set i and j represent the pedestrian numbers in the video, θ and Indicates the observation angle of pedestrians;

[0021] 52) In the test set, select the 3D gait feature vector of pedestrian i under the viewing angle θ With the registered pedestrian j in the viewing angle The 3D gait feature vector under Similarity measurement is performed to realize cross-view gait recognition. The measurement function uses the vector space cosine similarity and sets the judgment threshold a. When the similarity calculation result ≥ a, the 3D gaits of pedestrian i and pedestrian j are similar, otherwise they are not similar.

[0022] Step 21) Joint constraint feature matrix The establishment process is:

[0023] Use OpenPose3D to obtain the joint point posture data of pedestrians in the video, calculate and establish the joint constraint feature matrix The joint angle constraint R joint , foot angle constraint R foot , center of gravity and lower limb joint motion constraints R gra , human body structure constraints R step and human motion ratio constraint R forwThe motion constraint vector matrix is ​​constructed. The calculation of the above constraints is shown in equations (1) to (5).

[0024]

[0025] Among them, the joint point vector i represents the joint point of the lower limb, j1 and j2 are the joint points adjacent to i; is the normal vector of the XOY plane, a is the ankle joint, k is the knee joint; N G represents the three-dimensional coordinates of the body's center of gravity G, i1 and i2 represent the joints of the lower limbs; z0 is the vertical distance from the head node to the XOY plane;

[0026] Get the 3D joint constraint feature matrix of the video pedestrian motion

[0027] Step 22) Joint action feature matrix The establishment process is:

[0028] Use OpenPose3D to obtain the joint pose data in the pedestrian video and establish the displacement vector sequence of joint point i in the time series m The calculation method is shown in formula (6).

[0029]

[0030] Among them, Δt represents the time interval between two consecutive frames. Represents the three-dimensional coordinates of the i-th joint point of the pedestrian in the m-th frame;

[0031] Get the joint action feature matrix

[0032] The loss function of the 3D gait recognition network LC-POSEGAIT is:

[0033]

[0034] in, represents the weight matrix, b yi represents the bias, x i represents the output value after the network, c yi Indicates the yth i The center of the class.

[0035] The present invention has the following beneficial effects:

[0036] The cross-view gait recognition method combining LSTM and CNN described in the present invention is specifically operated by directly extracting the 3D posture data of pedestrians in the video from the CASIA-B gait video data set using OpenPose3D, and using this to train the 3D gait recognition network LC-POSEGAIT, which not only simplifies the preprocessing steps of pedestrian detection and tracking in the video, but also directly obtains the spatial three-dimensional coordinates of the joint points, avoids the need to establish a 3D gait model in a 3D imaging device or a multi-camera collaborative environment. The reconstruction makes the application scenario of gait recognition more extensive. In addition, the present invention uses the 3D gait recognition network LC-POSEGAIT for recognition, wherein the LSTM branch learns the temporal characteristics of the joint motion constraints from the joint constraint feature matrix, and the CNN branch captures the spatial motion characteristics of the pedestrian joints in the action feature matrix, so as to fully explore the spatiotemporal characteristics of the 3D gait motion, and can achieve better results in cross-view gait recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 is a flow chart of the present invention;

[0038] Figure 2 A labeling diagram of human joints;

[0039] Figure 3 Flowchart for constructing the eigenvector matrix of joint constraints for pedestrians;

[0040] Figure 4 This is the structural diagram of the 3D gait recognition network LC-POSEGAIT;

[0041] Figure 5 This is a comparison chart of the average recognition rates of SPAE, GaitGANv2, PoseGait method and the present invention. DETAILED DESCRIPTION

[0042] The present invention is further described in detail below in conjunction with the accompanying drawings:

[0043] refer to Figure 1 The cross-view gait recognition method combining LSTM and CNN of the present invention comprises the following steps:

[0044] 1) From the CASIA-B gait video dataset, OpenPose3D is used to extract the 3D pose data of pedestrians in the video;

[0045] From the CASIA-B gait video dataset published by the Institute of Automation, Chinese Academy of Sciences, we collected 3D pose data of 124 pedestrians in the video extracted by OpenPose3D in, The 3D coordinates of the ith joint point of the pedestrian in the mth frame are shown, including 10 walking postures and 11 viewing angles in 3 states (backpack, coat, normal walking), with a total of 124×10×11=13640 videos. Figure 2 ;

[0046] 2) extracting the motion constraint data and joint action data of the video pedestrian joint points from the 3D posture data obtained in step 1), and constructing training set samples based on them;

[0047] The specific operations of step 2) are:

[0048] 21) According to the joint point posture data of the pedestrian in the video, calculate the joint angle constraint R joint , foot angle constraint R foot , center of gravity and lower limb joint motion constraints R gra , human body structure constraints R step and human motion constraints R forw , to construct the 3D joint constraint feature matrix of the video pedestrian motion

[0049] Step 21) Joint constraint feature matrix The establishment process is:

[0050] Use OpenPose3D to obtain the joint point posture data of pedestrians in the video, calculate and establish the joint constraint feature matrix The joint motion constraint R joint , foot angle constraint R foot , center of gravity and lower limb joint motion constraints R gra , human body structure constraints R step and human motion ratio constraint R forw The motion constraint vector matrix is ​​constructed. The above constraint calculation is shown in equations (1) to (5).

[0051]

[0052] Among them, the joint point vector i represents the joint point of the lower limb, j1 and j2 are the joint points adjacent to i; is the normal vector of the XOY plane, a is the ankle joint, k is the knee joint; N G represents the three-dimensional coordinates of the body's center of gravity G, i1 and i2 represent the joints of the lower limbs; z0 is the vertical distance from the head node to the XOY plane;

[0053] Get the 3D joint constraint feature matrix of the video pedestrian motion Among them, each video obtains an n×44-dimensional pedestrian motion 3D joint constraint feature matrix.

[0054] 22) According to the joint point posture data of the pedestrian in the video, set the displacement vector sequence of the joint point in the time series m Express the motion features of pedestrian joints in the time domain and space domain to construct the pedestrian joint motion feature matrix

[0055] Step 22) Joint action feature matrix The establishment process is:

[0056] Use OpenPose3D to obtain the joint pose data in the pedestrian video and establish the displacement vector sequence of joint point i in the time series m The calculation is shown in formula (6),

[0057]

[0058] Among them, Δt represents the time interval between two adjacent frames. Represents the three-dimensional coordinates of the i-th joint point of the pedestrian in the m-th frame, and calculates the displacement vector sequence of the 15 joint points of the pedestrian in each frame on the time series m Δt = 1 / 25 = 0.04 seconds. For each frame of pedestrian joints, 45 displacement vectors are obtained, forming an n×45-dimensional action feature matrix. Among them, n represents the number of frames in each video;

[0059] Get the pedestrian joint action feature matrix

[0060] 23) The process from the first contact of the left foot of the pedestrian in the video with the ground to the second contact of the left foot with the ground is determined as a gait segmentation cycle, wherein all video frames within a gait segmentation cycle are video frames within the gait segmentation cycle;

[0061] 24) Extract the data of two gait segmentation cycles from the pedestrian joint motion feature matrix and the 3D joint constraint feature matrix of the video pedestrian motion, and use this to construct the training set samples and Then the gait standard sample and Divide into training set, registration set and test set;

[0062] Specifically, the process from the first contact of the left foot of the pedestrian in the video with the ground to the second contact of the left foot with the ground is determined as a gait segmentation cycle. Specifically, when the left foot is in front, the left and right ankle joints ( Figure 2The video frame where the distances between the middle node 13 and the node 10) and the XOY plane (ground plane) are the shortest is the starting frame of a gait segmentation cycle. If a frame meets the condition and and Then the mth frame is the starting frame of a gait segmentation cycle.

[0063] Go through the process of "(left foot in front and right foot behind) → (left foot on the ground, right foot lifted) → (right foot in front and left foot behind) → (right foot on the ground, left foot lifted) → (left foot in front and right foot behind)", calculate the distance z between the left and right ankle joints and the XOY plane of each frame respectively. 13 、z 10 , when the left foot is in front and the distances from the left and right ankle joints to the XOY plane are the shortest again, the end frame n of a gait segmentation cycle is obtained, and all video frames during this period are video frames within a gait segmentation cycle.

[0064] From the joint constraint feature matrix and pedestrian joint action feature matrix In the example, we take the constraint vector matrix of two gait segmentation cycles and As standard gait samples, there are 124×10×11=13640 samples, where: is a 50×44 dimensional vector matrix, is a 50×45 dimensional vector matrix.

[0065] The gait standard samples are divided into training set, registration set and test set for training LC-POSEGAIT network and cross-view gait recognition. The sample set division is shown in Table 1.

[0066] Table 1

[0067]

[0068] 3) Construct 3D gait recognition network LC-POSEGAIT;

[0069] The specific process of step 3) is:

[0070] 31) Construction of 3D gait recognition network LC-POSEGAIT: The 3D gait recognition network LC-POSEGAIT is established by combining LSTM and CNN in parallel. Figure 4 As shown in the figure, the LSTM network has 2 layers, which are used to extract the pedestrian joint constraint feature matrix The joint motion timing characteristics, the model parameters are: activation function is ReLU, step size = 50, input dimension = 44, number of hidden layer nodes = 256, dropout = 0.1, number of LSTM layers = 2.

[0071] The CNN network consists of 2 convolutional layers, 2 pooling layers and a fully connected layer to extract the joint action feature matrix. The parameters of the joint action space features are shown in Table 2.

[0072] Table 2

[0073]

[0074] The features extracted by the two networks are fused in the fully connected layer.

[0075] Using Softmax loss function L s And Center loss function L C The joint multi-loss function L optimizes the network, specifically:

[0076]

[0077] in, represents the weight matrix, b yi represents the bias, x i represents the output value after the network, c yi Indicates the yth i The center of the class.

[0078] 32) Initialization of 3D gait recognition network LC-POSEGAIT:

[0079] The Adam method is used to optimize the network. The learning rate of each parameter is adjusted in an adaptive manner during the training process to make the network converge as quickly as possible. At the same time, in order to make the convergence speed of the network back propagation calculation faster, the activation function ReLU is selected, and its formula is Set the initial parameters of the network, where the learning rate of the network in back propagation error update weights is α = 0.001, the number of training samples per batch is Batch-size = 64, the number of network iterations is epoch = 800, and the sample center c yi Random initialization.

[0080] 4) Using the training set samples obtained in step 2) to train the 3D gait recognition network LC-POSEGAIT;

[0081] The specific process is: the joint constraint feature matrix of the training set data (001#-074#) and joint action feature matrix The data are sent to the LSTM network and CNN network respectively, and the network output value is obtained through forward propagation of data. The weight is adjusted along the gradient descent direction through the back propagation algorithm until the loss function value of the network reaches the minimum, and finally the parameters of the stable 3D gait recognition network LC-POSEGAIT are obtained.

[0082] 5) Use the trained 3D gait recognition network LC-POSEGAIT to extract the gait feature vector of pedestrians in the video and complete cross-view gait recognition combining LSTM and CNN.

[0083] The specific operations of step 5) are:

[0084] 51) The fully connected layer vector of the 3D gait recognition network LC-POSEGAIT is used as the pedestrian 3D gait feature vector, and the 3D gait feature vectors of the test set and the registration set samples are extracted respectively. Among them, the feature vector set of the test set The feature vector set of the registration set i and j represent the pedestrian numbers in the video, θ and Indicates the observation angle of pedestrians;

[0085] 52) In the test set, select the 3D gait feature vector of pedestrian i under the viewing angle θ With the registered pedestrian j in the viewing angle The 3D gait feature vector under Similarity measurement is performed to realize cross-view gait recognition. The measurement function uses the vector space cosine similarity and sets the judgment threshold a. When the similarity calculation result is ≥ a, the 3D gaits of pedestrians i and j are similar, otherwise they are not similar. The similarity calculation is:

[0086]

[0087] In order to verify the effectiveness of the invented method, the same-state cross-view experiments were conducted on the CASIA-B dataset under normal walking, unchanged clothing, and backpack walking. The recognition rate was taken as the average recognition rate of the test set at each viewpoint. The experimental results are shown in Table 3.

[0088] Table 3

[0089]

[0090] In order to compare the recognition results with SPAE, GaitGANv2 appearance method and PoseGait method of posture estimation, the walking state of the registration set in the test set is adjusted to nm01-04, and the test data set is nm05-06, bg01-02, cl01-02. The average recognition rate across viewpoints is shown in Figure 5 shown.

Claims

1. A cross-view gait recognition method combining LSTM and CNN, characterized in that: The following steps are involved: 1) From the CASIA-B gait video dataset, OpenPose3D is used to extract the 3D pose data of pedestrians in the video; 2) extracting the motion constraint data and joint action data of the video pedestrian joint points from the 3D posture data obtained in step 1), constructing the pedestrian motion constraint matrix and establishing a sample set based on it; 3) Construct a 3D gait recognition network LC-POSEGAIT that combines LSTM and CNN in parallel; 4) Using the sample set obtained in step 2) to train the 3D gait recognition network LC-POSEGAIT; 5) Use the trained 3D gait recognition network LC-POSEGAIT to extract the gait feature vector of the pedestrian in the video, and measure the gait feature vector to achieve cross-view gait recognition; The specific operations of step 2) are: 21) According to the joint point posture data of the pedestrian in the video, calculate the joint angle constraint R joint , foot angle constraint R foot , center of gravity and lower limb joint motion constraints R gra , human body structure constraints R step and human motion constraints R forw , to construct the 3D joint constraint feature matrix of the video pedestrian motion 22) According to the joint point posture data of the pedestrian in the video, set the displacement vector sequence of the joint point in the time series m Express the motion features of pedestrian joints in the time domain and space domain to construct the pedestrian joint motion feature matrix 23) Gait segmentation cycle: a gait segmentation cycle is defined as the process from the first contact of the left foot of the pedestrian in the video with the ground to the second contact of the left foot with the ground; 24) Extract the data of two gait segmentation cycles from the pedestrian joint constraint feature matrix and joint action feature matrix, and use them to construct training set samples and Then the gait standard sample and Split into training set, registration set and test set; Step 21) Joint constraint feature matrix The establishment process is: Use OpenPose3D to obtain the joint point posture data of pedestrians in the video, calculate and establish the joint constraint feature matrix is the joint angle constraint R joint , foot angle constraint R foot , center of gravity and lower limb joint motion constraints R gra , human body structure constraints R step and human motion ratio constraint R forw The motion constraint vector matrix is ​​constructed. The above constraint calculation method is shown in equations (1) to (5); Among them, the joint point vector i represents the joint point of the lower limb, j1 and j2 are the joint points adjacent to i; is the normal vector of the XOY plane, a is the ankle joint, k is the knee joint; N ⊙ represents the three-dimensional coordinates of the body's center of gravity G, i1 and i2 represent the joints of the lower limbs; z0 is the vertical distance from the head node to the XOY plane, α i is the angle between i and the vector formed by the adjacent joint point, (x0, y0, z0), (x 10 ,y 10 , z 10 )、(x 13 ,y 13 , z 13 ) are the coordinates of the head, right ankle, and left ankle joints respectively; z0 is the vertical distance from the head node to the horizontal plane XOY, β is the joint vector of a and k and The angle of Get the 3D joint constraint feature matrix of the video pedestrian motion 2. According to claim 1, the cross-view gait recognition method combining LSTM and CNN is characterized in that: The specific operation of step 1) is: from the CASIA-B gait video dataset, use OpenPose3D to extract the 3D posture data of pedestrians in the video in, Represents the 3D coordinates of the i-th joint point of the pedestrian in the m-th frame.

3. The cross-view gait recognition method combining LSTM and CNN according to claim 1 is characterized in that: The specific operations of step 3) are: The 3D gait recognition network LC-POSEGAIT is established by combining LSTM network and CNN network in parallel. The LSTM network is used to extract the joint constraint feature matrix of the pedestrian in the video. The motion timing features of the CNN network are used to extract the joint action feature matrix The action space features of the LSTM network and the features extracted by the CNN network are fused through the fully connected layer; The 3D gait recognition network LC-POSEGAIT adopts the Softmax loss function L s With the Center loss function L C The joint multi-loss function L = L s +λL C As a loss function, λ is a balancing parameter.

4. The cross-view gait recognition method combining LSTM and CNN according to claim 2 is characterized in that: The specific operations of step 5) are: 51) The fully connected layer vector of the 3D gait recognition network LC-POSEGAIT is used as the pedestrian 3D gait feature vector, and the 3D gait feature vectors of the test set and the registration set samples are extracted respectively. Among them, the feature vector set of the test set The feature vector set of the registration set i and j represent the pedestrian numbers in the video, θ and Indicates the observation angle of pedestrians; 52) In the test set, select the 3D gait feature vector of pedestrian i under the viewing angle θ With the registered pedestrian j in the viewing angle The 3D gait feature vector under Similarity measurement is performed to realize cross-view gait recognition. The measurement function uses the vector space cosine similarity and sets the judgment threshold a. When the similarity calculation result is ≥ a, the 3D gaits of pedestrian i and pedestrian j are similar, otherwise, the 3D gaits are not similar.

5. The cross-view gait recognition method combining LSTM and CNN according to claim 3 is characterized in that: The loss function of the 3D gait recognition network LC-POSEGAIT is: in, represents the weight matrix, b yi represents the bias, x i represents the output value after the network, c yi Indicates the yth i The center of the class.

Images