A 3D gait recognition method based on two-way similarity learning

Abstract

The invention discloses a 3D gait recognition method based on two-way similarity learning. A 3D gait recognition network model TSLN-PoseGait based on two-way similarity learning is constructed based on LSTM and Resnet-50 models. A training sample pair and a test sample pair are constructed by using a sample set, and then the training sample pair and the test sample pair are used to train and test the 3D gait recognition network model TSLN-PoseGait to obtain a trained 3D gait recognition network model TSLN-PoseGait. A 3D joint constraint matrix and a joint action matrix of a pair of video pedestrians to be identified are obtained, and are input into the trained 3D gait recognition network model TSLN-PoseGait, and the 3D gait spatiotemporal difference features of the two video pedestrians are identified to determine the gait similarity of the pair of video pedestrians to be identified. The method can realize direct video pedestrian gait comparison without the need for video preprocessing steps of pedestrian detection and tracking.

Description

Technical Field

[0001] The invention belongs to the field of machine vision and relates to a 3D gait recognition method based on two-way similarity learning. Background Art

[0002] Biometrics recognition has a wide range of applications in security monitoring, pedestrian tracking, and identity recognition. At present, a large number of video surveillance equipment are installed in public places. The collected surveillance data is mostly used for post-event evidence collection, and rarely for identity verification of specific target pedestrians in an unaware, long-distance, and uncontrolled manner to support security warnings and alarms. Gait is the performance of a person's walking posture during the process of moving. It has the characteristics of non-contact, non-invasive, difficult to hide and forge. It is the only biometric feature that can be obtained at a long distance and in an uncontrolled state, and can support long-distance identity recognition. Therefore, gait recognition has broad application prospects.

[0003] Gait can only show significant characteristics in a set of movement cycles. It is easily disturbed by external environment (viewing angle, road surface, occlusion, etc.) and internal factors (carrying objects, clothing, etc.). In addition, the non-rigid movement of the human body makes the change of observation angle have a greater impact on the intra-class characteristics of pedestrian gait. The existing cross-view gait recognition method based on appearance features adopts energy graph modeling, which is easy to lose the spatiotemporal information of gait and is easily affected by the changes in clothing, carrying objects and body shape of pedestrians. The recognition performance is limited and it is only applicable to situations with limited or small range of viewing angle changes. Due to the three-dimensional nature of gait information, the movement changes of human joints in 3D space can more accurately describe gait characteristics. However, tracking and comparing human joints in traditional 3D gait models is a difficult point, and the use of 3D equipment or multi-camera collaboration to reconstruct 3D gait models requires complex camera parameter adjustment and modeling calculations, and its application scenarios are limited. Summary of the invention

[0004] The purpose of the present invention is to overcome the shortcomings of the above-mentioned prior art and provide a 3D gait recognition method based on two-way similarity learning, which can realize direct video pedestrian gait comparison without the need for video preprocessing steps of pedestrian detection and tracking.

[0005] To achieve the above object, a two-path similarity learning 3D gait recognition method according to the present invention comprises the following steps:

[0006] 1) From the gait video dataset, OpenPose3D is used to extract the 3D posture data of pedestrians in the video;

[0007] 2) extracting the motion constraint data and joint action data of the joint points of the video pedestrian from the 3D posture data extracted in step 1), and then constructing the 3D joint constraint matrix and joint action matrix of the video pedestrian respectively according to the motion constraint data and joint action data, and then constructing a sample set with the 3D joint constraint matrix and joint action matrix of the video pedestrian;

[0008] 3) Construct a 3D gait recognition network model TSLN-PoseGait based on dual-path similarity learning based on LSTM and Resnet-50 models;

[0009] 4) Using the sample set obtained in step 2), construct training sample pairs and test sample pairs, and then use the training sample pairs and test sample pairs to train and test the 3D gait recognition network model TSLN-PoseGait, to obtain the trained 3D gait recognition network model TSLN-PoseGait;

[0010] 5) Obtain the 3D joint constraint matrix and joint action matrix of a pair of pedestrians to be identified in the video, input them into the trained 3D gait recognition network model TSLN-PoseGait, and identify the 3D gait spatiotemporal difference characteristics of the two video pedestrians to determine the gait similarity of the pair of pedestrians to be identified in the video.

[0011] The specific operations of step 1) are:

[0012] From the CASIA-B gait video dataset, OpenPose3D is used to extract the 3D pose data of pedestrians in the video. in, is the three-dimensional coordinate of the i-th joint point in the m-th frame of the pedestrian video; the CASIA-B gait video dataset contains 124 video pedestrians, who are numbered from #001 to #124, among which the normal walking data is recorded as NM#001-124, the backpack walking data is recorded as BG#001-124, and the coat or jacket walking data is recorded as CL#001-124.

[0013] The specific operations of step 2) are:

[0014] 2a) According to the 3D posture data of the pedestrian in the video, calculate the joint angle constraints R joint , foot angle constraint R foot , center of gravity and lower limb joint motion constraints R gra , human body structure constraints R step and human motion constraints R forw , and use this to construct the 3D joint constraint matrix M of the video pedestrian motion r =[R joint , R foot , R gra , R step, R frow ] T ;

[0015] 2b) Based on the 3D posture data of the pedestrian in the video, the displacement vector sequence of the joint points in the time series m Express the motion characteristics of gait in time and space domains, and construct the 3D motion matrix of pedestrian joints

[0016] 2c) The process from the first contact of the left foot of the pedestrian in the video to the second contact of the left foot with the ground is regarded as a gait segmentation cycle;

[0017] 2d) From the joint constraint matrix M of the pedestrian motion r and joint action matrix M a Extract the data of two gait segmentation cycles from the , and use them to construct the sample set {M r} and {M a}, and then divide the gait samples into training set, registration set and test set.

[0018] The specific operations of step 3) are:

[0019] The 3D gait recognition network model TSLN-PoseGait is established by parallelizing the LSTM network and the Resnet-50 network and fusing features. One of the routes is composed of two LSTM networks in series and a flat layer, which is used to extract the joint constraint matrix M of a pair of video pedestrians. r The other route is composed of a Resnet-50 network, which is used to extract the joint action matrix M of the pedestrians in the video a Finally, the dual-path difference features extracted by LSTM and Resnet-50 networks are fused through two fully connected layers, and the Softmax loss function is used to realize the gait similarity discrimination of a pair of video pedestrians.

[0020] The specific operations of step 4) are:

[0021] The training set of the 3D gait recognition network model TSLN-PoseGait is selected #001-074 video pedestrian joint constraint matrix M under all walking conditions r and 3D motion matrix M a ; The gallery set uses the NM#001-004 data of pedestrians in the #075-124 video, and the probe set uses the NM#005-006, BG#001-002 and CL#001-002 data of pedestrians in the #075-124 video.

[0022] The joint constraint matrix M in step 2a) rThe establishment process is:

[0023] Use OpenPose3D to obtain 3D pose data of video pedestrians Calculate and establish the joint constraint matrix M r , M r Constraint R for joint motion joint , foot angle constraint R foot , center of gravity and lower limb joint motion constraints R gra , human body structure constraints R step and human motion ratio constraint R forw The motion constraint vector matrix is ​​composed of

[0024]

[0025] in, is the joint point vector, i is the lower limb joint point, j1 and j2 are the joint points adjacent to i, is the normal vector of the XOY plane, a is the ankle joint, k is the knee joint, N G represents the three-dimensional coordinates of the body's center of gravity G, i1 and i2 represent the joints of the lower limbs, z0 is the vertical distance from the head node to the XOY plane, and the 3D joint constraint matrix M of the video pedestrian motion is obtained r =[R joint , R foot , R jgra , R step , R frow ] T .

[0026] The process of establishing the joint action matrix Ma in step 2b) is:

[0027] Using OpenPose3D to obtain 3D pose data from pedestrian videos Establish the displacement vector sequence of joint point i in time series m Get the joint action feature matrix in,

[0028]

[0029] Among them, Δt represents the time interval between two adjacent frames. Represents the 3D coordinates of the i-th joint point of the pedestrian in the m-th frame.

[0030] The present invention has the following beneficial effects:

[0031] The 3D gait recognition method of the dual-path similarity learning of the present invention is to directly extract the 3D posture data of pedestrians in the video from the CASIA-B gait video dataset using OpenPose3D, and use this to construct the joint constraint and action constraint matrix M of the pedestrian gait. r 、M a , and establish a sample set, then train and test the 3D gait recognition network model TSLN-PoseGait, and finally use the trained 3D gait recognition network model TSLN-PoseGait for 3D gait recognition, to achieve direct video pedestrian gait comparison and identification, with good perspective robustness, while avoiding pre-processing work such as pedestrian detection and tracking, and can alleviate the impact of shooting angle, carried objects, lighting and other factors on gait recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 This is the structural diagram of the 3D gait recognition network model TSLN-PoseGait;

[0033] Figure 2 It is a flow chart of the present invention. DETAILED DESCRIPTION

[0034] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only embodiments of a part of the present invention, not all embodiments, and are not intended to limit the scope of the present invention. In addition, in the following description, the description of well-known structures and technologies is omitted to avoid unnecessary confusion of the concepts disclosed in the present invention. Based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work should fall within the scope of protection of the present invention.

[0035] The accompanying drawings show schematic diagrams of structures according to embodiments disclosed in the present invention. These figures are not drawn to scale, and some details are magnified and some details may be omitted for the purpose of clear expression. The shapes of various regions and layers shown in the figures and the relative sizes and positional relationships therebetween are only exemplary, and may deviate in practice due to manufacturing tolerances or technical limitations, and those skilled in the art may additionally design regions / layers with different shapes, sizes, and relative positions according to actual needs.

[0036] refer to Figure 1 and Figure 2 The 3D gait recognition method based on two-way similarity learning of the present invention comprises the following steps:

[0037] 1) From the gait video dataset published by CASIA-B (Institute of Automation, Chinese Academy of Sciences), OpenPose3D is used to extract the 3D pose data of pedestrians in the video;

[0038] From the CASIA-B gait video dataset, we collected OpenPose3D to extract 3D pose data of 124 pedestrians in the video. in, It represents the 3D coordinates of the ith joint point of the pedestrian in the mth frame. The dataset contains 124 video pedestrians. Each pedestrian has 11 shooting angles and 10 walking states under 3 walking conditions, including 2 types of backpack walking (BG), 2 types of coat walking (CL), and 6 types of normal walking (NM). A total of 124×11×10 groups of posture data are obtained.

[0039] 2) Extract the joint constraint matrix M of the video pedestrian motion from the 3D posture data obtained in step 1) r and joint action matrix M a , and use this to construct a gait sample set;

[0040] The specific operations of step 2) are:

[0041] 2a) Based on the 3D pose data of the pedestrian in the video Calculate the joint angle constraint R separately joint , foot angle constraint R foot , center of gravity and lower limb joint motion constraints R gra , human body structure constraints R step and human motion constraints R forw , and use this to construct the joint constraint matrix M of the video pedestrian r =[R joint , R foot , R gra , R step , R frow ] T ;

[0042] The joint constraint matrix M in step 2a) r The establishment process is:

[0043] Use OpenPose3D to obtain 3D pose data of pedestrians in the video Calculate and establish the joint constraint matrix M r , M r The joint motion constraint R joint , foot angle constraint R foot , center of gravity and lower limb joint motion constraints R gra , human body structure constraints R step and human motion ratio constraint R forw The motion constraint vector matrix is:

[0044]

[0045] in, is the joint vector, i represents the lower limb joint point, j1 and j2 are the joint points adjacent to i, is the normal vector of the XOY plane, a is the ankle joint, k is the knee joint, N G represents the three-dimensional coordinates of the body's center of gravity G, i1 and i2 represent the joint points of the lower limbs, z0 is the vertical distance from the head node to the XOY plane, and the joint constraint matrix M of the video pedestrian gait is obtained r =[R joint , R foot , R jgra , R step , R frow ] T .

[0046] 2b) Based on the 3D pose data of the pedestrian in the video Set the displacement vector sequence of the joint point in the time series m To express the motion characteristics of pedestrian joints in the time domain and space domain, so as to construct the joint motion matrix of the video pedestrian

[0047] Step 2b) Joint action matrix M a The establishment process is:

[0048] 3D pose data of pedestrians in videos obtained using OpenPose3D Establish the displacement vector sequence of joint point i in time series m Specifically:

[0049]

[0050] Where Δt is the time interval between adjacent frames, is the three-dimensional coordinate of the i-th joint point of the pedestrian in the m-th frame, and calculates the displacement vector sequence of the 15 joint points of the pedestrian in each frame on the time series m Among them, Δt = 1 / 25 = 0.04 seconds, then 45 (15 × 3) displacement vectors can be obtained from the pedestrian joints in each frame, thus forming an n × 45-dimensional action matrix of the n-frame gait of the video pedestrian

[0051] 2c) The process from the first contact of the left foot of the pedestrian in the video with the ground to the second contact of the left foot with the ground is determined as a gait segmentation cycle, wherein all video frames within a gait segmentation cycle are video frames within the gait segmentation cycle;

[0052] 2d) The 3D joint constraint matrix M of the pedestrian from the videor Neutralize joint action matrix M a In the example, the segmented data of two gait cycles are extracted to construct the sample set of TSLN-Pose Gait network {M r} and {M a};

[0053] Specifically, the process from the first contact of the left foot of the pedestrian in the video with the ground to the second contact of the left foot with the ground is determined as a gait segmentation cycle. The pedestrian experiences the process of "(left foot in front and right foot behind) → (left foot on the ground, right foot lifted) → (right foot in front and left foot behind) → (right foot on the ground, left foot lifted) → (left foot in front and right foot behind)", and calculates the distance between the left and right ankle joints and the ground plane in each frame. When the left foot is in front and the distances between the left and right ankle joints and the ground plane are the shortest again, the end frame n of a gait segmentation cycle is obtained, and all video frames during this period are video frames within a gait segmentation cycle.

[0054] The CASIA-B gait video dataset contains 124 video pedestrians, each of whom has 11 shooting angles and 10 walking states under 3 walking conditions, namely, 2 types of backpack walking (BG), 2 types of walking with a coat (CL), and 6 types of normal walking (NM). Therefore, a total of 124×11×10 groups of gait samples are obtained, among which M r is a 50×44 dimensional vector matrix, M a It is a 50×45 dimensional vector matrix.

[0055] 3) Construct a 3D gait recognition network model TSLN-PoseGait (Two-way Similarity Learning Network for Pose Gait) based on LSTM and Resnet-50 two-way similarity learning;

[0056] Reference Figure 1 , the specific process of step 3) is:

[0057] The 3D gait similarity learning network model TSLN-PoseGait is established by using LSTM network and Resnet-50 network in parallel and feature fusion; Figure 1 The TCDFE module in the figure consists of two LSTM layers and a flat layer, and the SCDFE module consists of a Resnet-50 network. The features obtained by the flat layers of the two branches are passed through two fully connected layers FC-1, FC-2 and a Softmax layer to obtain the similarity judgment results of the input pairs.

[0058] The parameters of the TSLN-PoseGait network model are shown in Table 1.

[0059] Table 1

[0060]

[0061] 3b) The network parameters are randomly initialized, and the network is trained using back propagation. The weights are updated in 256 small batches, and all network layer parameters are adjusted. The learning rate of the network in back propagation error update weights is lr=0.001, and the number of training samples per batch is Batch-size=128.

[0062] 4) constructing training sample pairs and test sample pairs using the data obtained in step 2), and training and testing the two-way similarity learning 3D gait recognition network model TSLC-PoseGait by using the training sample pairs and the test sample pairs;

[0063] The specific process of step 4) is:

[0064] 4a) The gait sample set {M r} and {M a} is divided into training set, registration set and test set. The training set selects M pedestrians in all walking conditions in video #001-074. r 、M a The gallery set uses NM#001-004 data of pedestrians in video #075-124, and the probe set uses NM#005-006, BG#001-002, and CL#001-002 data of pedestrians in video #075-124. The training sample pairs and test sample pairs are constructed according to the sample combination principle. The data pairs of the same person are marked as positive samples, otherwise they are marked as negative samples. Each video of a person is combined with other videos of the same person to obtain 124×110×109 positive samples. After removing samples due to video freezes and no people, about 1.47 million positive samples are obtained. In order to ensure a balanced distribution of positive and negative samples, about 1.5 million negative sample pairs are generated by comprehensively considering the shooting angle and walking conditions.

[0065] 4b) The specific process of network training is: the joint constraint matrix (M r1 , M r2 ) and joint action matrix pair (M a1 , M a2 ) are sent to the TCDFE module and the SCDFE module respectively, and the network output value is obtained through forward data propagation. The weight is adjusted along the gradient descent direction through the back-propagation algorithm until the loss function value of the network reaches the minimum, and finally a stable 3D gait similarity learning network TSLN-PoseGait model is obtained.

[0066] 5) Calling the trained gait recognition network model TSLN-PoseGait can achieve end-to-end gait similarity discrimination. Use the joint constraints and action matrix of pedestrian #075-124 in the CASIA-B dataset to build test sample pairs, input TSLN-PoseGait to obtain the similarity discrimination values ​​of all sample pairs, among which the average recognition rate under cross-view and cross-walking conditions is shown in Table 2.

[0067] Table 2

[0068]

Claims

1. A 3D gait recognition method based on two-way similarity learning, characterized in that: The following steps are involved: 1) From the gait video dataset, OpenPose3D is used to extract the 3D posture data of pedestrians in the video; 2) extracting the motion constraint data and joint action data of the joint points of the video pedestrian from the 3D posture data extracted in step 1), and then constructing the 3D joint constraint matrix and joint action matrix of the video pedestrian respectively according to the motion constraint data and joint action data, and then constructing a sample set with the 3D joint constraint matrix and joint action matrix of the video pedestrian; 3) Construct a 3D gait recognition network model TSLN-PoseGait based on dual-path similarity learning based on LSTM and Resnet-50 models; 4) Using the sample set obtained in step 2), construct training sample pairs and test sample pairs, and then use the training sample pairs and test sample pairs to train and test the 3D gait recognition network model TSLN-PoseGait, to obtain the trained 3D gait recognition network model TSLN-PoseGait; 5) Obtain the 3D joint constraint matrix and joint action matrix of a pair of pedestrians to be identified in the video, input them into the trained 3D gait recognition network model TSLN-PoseGait, and identify the 3D gait spatiotemporal difference characteristics of the two video pedestrians to determine the gait similarity of the pair of pedestrians to be identified in the video; The specific operations of step 3) are: The 3D gait recognition network model TSLN-PoseGait is established by parallelizing the LSTM network and the Resnet-50 network and fusing features. One of the routes is composed of two LSTM networks in series and a flat layer, which is used to extract the joint constraint matrix M of a pair of video pedestrians. r The other route is composed of a Resnet-50 network, which is used to extract the joint action matrix M of the pedestrians in the video a Finally, the dual-path difference features extracted by LSTM and Resnet-50 networks are fused through two fully connected layers, and the Softmax loss function is used to realize the gait similarity discrimination of a pair of video pedestrians. The specific operations of step 2) are: 2a) According to the 3D posture data of the pedestrian in the video, calculate the joint angle constraints R joint , foot angle constraint R foot , center of gravity and lower limb joint motion constraints R gra , human body structure constraints R step and human motion constraints R forw , and use this to construct the 3D joint constraint matrix M of the video pedestrian motion r =[R joint ,R foot ,R gra ,R step ,R frow ] T ; 2b) Based on the 3D posture data of the pedestrian in the video, the displacement vector sequence of the joint points in the time series m Express the motion characteristics of gait in time and space domains, and construct the 3D motion matrix of pedestrian joints 2c) The process from the first contact of the left foot of the pedestrian in the video with the ground to the second contact of the left foot with the ground is regarded as a gait segmentation cycle; 2d) From the joint constraint matrix M of the pedestrian motion r and joint action matrix M a Extract the data of two gait segmentation cycles from the , and use them to construct the sample set {M r } and {M a }, and then divide the gait samples into training set, registration set and test set.

2. A 3D gait recognition method based on two-way similarity learning according to claim 1, characterized in that: The specific operations of step 1) are: From the CASIA-B gait video dataset, OpenPose3D is used to extract the 3D pose data of pedestrians in the video. in, is the three-dimensional coordinate of the i-th joint point in the m-th frame of the pedestrian video; the CASIA-B gait video dataset contains 124 video pedestrians, which are numbered from #001 to #124, among which the normal walking data is recorded as NM#001-124, the backpack walking data is recorded as BG#001-124, and the coat or jacket walking data is recorded as CL#001-124.

3. The 3D gait recognition method based on two-way similarity learning according to claim 1, characterized in that: The specific operations of step 4) are: The training set of the 3D gait recognition network model TSLN-PoseGait is selected #001-074 video pedestrian joint constraint matrix M under all walking conditions r And the action matrix M a ; The gallery set uses the NM#001-004 data of pedestrians in the #075-124 video, and the probe set uses the NM#005-006, BG#001-002 and CL#001-002 data of pedestrians in the #075-124 video.

4. The 3D gait recognition method based on two-way similarity learning according to claim 1, characterized in that: The joint constraint matrix M in step 2a) r The establishment process is: Use OpenPose3D to obtain 3D pose data of video pedestrians Calculate and establish the joint constraint matrix M r , M r Constraint R for joint motion joint , foot angle constraint R foot , center of gravity and lower limb joint motion constraints R gra , human body structure constraints R step and human motion ratio constraint R fotw The motion constraint vector matrix is ​​composed of in, is the joint point vector, i is the lower limb joint point, j1 and j2 are the joint points adjacent to i, is the normal vector of the XOY plane, a is the ankle joint, k is the knee joint, N6 represents the three-dimensional coordinates of the body center of gravity G, i1 and i2 represent the joint points of the lower limbs, z8 is the vertical distance from the head node to the XOY plane, and the 3D joint constraint matrix M of the video pedestrian motion is obtained. r =[R joint ,R foot ,R jpra ,R step ,R frow ] T .

5. The 3D gait recognition method based on two-way similarity learning according to claim 1, characterized in that: Step 2b) Joint action matrix M a The establishment process is: Using OpenPose3D to obtain 3D pose data from pedestrian videos Establish the displacement vector sequence of joint point i in time series m Get the joint action feature matrix in, Among them, ΔT represents the time interval between two adjacent frames. Represents the three-dimensional coordinates of the pedestrian's mth frame and i-th joint point.

Images