A gait recognition method and system based on fine-grained motion pattern selection

Abstract

The present invention relates to a gait recognition method and system based on fine-grained motion pattern selection, belonging to the field of computer vision and biometrics. The present invention proposes for the first time a gait recognition model with simple structure and effective fine-grained motion pattern selection, which can screen out robust point features under various external conditions from a large number of feature points, thereby effectively achieving high-precision gait recognition. The present invention effectively solves the problem of low gait recognition accuracy in existing methods, as well as the problem of insufficient gait feature expression caused by the lack of fine-grained feature extraction and motion pattern selection, and provides a solution with a relatively simple and effective network structure for control in real-world scenarios.

Description

Technical Field

[0001] The present invention relates to a gait recognition method and system based on fine-grained motion pattern selection, which is used to identify identities based on gait silhouette sequences and belongs to the technical field of computer vision and biometrics. Background Art

[0002] Gait recognition is an important technology in the field of biometric recognition. It can obtain walking videos of the same subject from multiple cameras at different angles and identify the pedestrian's identity information.

[0003] Compared to facial, fingerprint, and iris recognition, gait recognition boasts advantages such as long-range, multi-viewing angles, non-cooperativeness, and susceptibility to camouflage. Gait recognition typically has an effective range of 50 meters and has enormous application value in scenarios such as social security, video surveillance, and crime investigation.

[0004] From a kinematic perspective, the human body can be considered a multi-linked rigid body system composed of multiple joints. During walking, the angles of each joint and the center of gravity are constantly changing, requiring hundreds of kinematic parameters to fully describe the body's spatial motion. From a kinematics perspective, gait is a complex process involving the coordinated control of various muscles and central nervous systems. Muscle distribution, skeletal structure, and motor nerves vary from person to person due to differences in upbringing and individual genetics. Therefore, while the gait characteristics of the same person are relatively stable across different scenarios, they vary between individuals.

[0005] At present, there are two main types of gait recognition methods: one is the model-based gait recognition method, and the other is the appearance-based gait recognition method.

[0006] 1. Model-based gait recognition methods. These methods attempt to fit the human posture structure into the image and extract kinematic features. For example, skeleton-based gait recognition methods typically use a fixed number of key points as the basis for feature extraction.

[0007] However, there is currently no evidence that the choice of these key points is optimal.

[0008] 2. Appearance-based gait recognition methods, including template-based gait recognition methods, set-based gait recognition methods, and sequence-based gait recognition methods.

[0009] (1) Template-based gait recognition method: First, the gait prior information is obtained by using sequence compression method, such as gait energy graph and timed gait image, and then the gait features are extracted.

[0010] However, the input data of this type of method loses a lot of temporal information and the recognition accuracy is not high.

[0011] (2) Set-based gait recognition method. Assuming that the gait sequence itself contains position information, a random sampling method is used to randomly select a fixed number of frames from the sequence as input. Then, 2D convolution is used to extract the features of each image separately. Finally, the maximum pooling method is used to directly fuse the features of the sequence.

[0012] However, this type of method treats the silhouette sequence as a set and integrates features through maximum pooling, which greatly weakens the extraction of temporal features.

[0013] (3) Sequence-based gait recognition method: First, a fixed number of consecutive frames are selected as input, and then 3D convolution is used to extract the spatiotemporal features in the sequence.

[0014] However, such methods ignore the effective processing and selection of fine-grained features, lose a lot of information, and introduce complex modules.

[0015] Therefore, in order to obtain higher gait recognition accuracy, a simpler neural network, or to meet the demand for deploying ordinary cameras under realistic conditions, there is an urgent need for a gait recognition neural network method and system with high recognition accuracy and simple network structure. Summary of the Invention

[0016] The purpose of the present invention is to address the defects and shortcomings of the existing technology, and to solve technical problems such as the existing methods ignoring the understanding of the fine-grained classification problem of gait recognition, and the lack of temporal information due to direct maximum pooling of the time dimension. A gait recognition method and system based on fine-grained motion pattern selection are creatively proposed.

[0017] The innovation of the present invention lies in: for the first time, a gait recognition model with simple structure and effective fine-grained motion pattern selection is proposed, which can screen out robust point features under various changes in external conditions from a large number of feature points, thereby effectively realizing high-precision gait recognition.

[0018] In order to achieve the above objectives, the present invention adopts the following technical solutions.

[0019] A gait recognition method based on fine-grained motion pattern selection directly processes silhouette sequences and includes the following steps:

[0020] Step 1: Preprocess the acquired gait video data to obtain the original gait silhouette sequence.

[0021] Specifically, data preprocessing includes extracting the contour edges of the human figure in the image, aligning the center of the human body with the center of the image, and scaling the gait image to a uniform size.

[0022] Step 2: Using the backbone network based on fine-grained motion pattern selection, feature extraction is performed on the gait silhouette sequence to obtain the primary features of each point in the sequence.

[0023] The gait silhouette sequence consists of any number of consecutive sampled frames. There are continuous motion changes between adjacent frames. The backbone network extracts gait features from the input sequence.

[0024] Specifically, it includes a fine-grained feature extraction module and a horizontal pooling module.

[0025] First, based on the fine-grained feature extraction method, the spatiotemporal features of each point in the sequence are extracted to form a spatiotemporal feature map.

[0026] Then, through horizontal pooling, the spatiotemporal feature map is horizontally divided into a fixed number of features, and then the local features are optimized separately.

[0027] Step 3: For each local feature, motion mode selection is used to establish a global connection, and further obtain discriminative spatiotemporal features while removing interference points.

[0028] Among them, motion pattern selection is to further establish the global spatiotemporal relationship from the primary features of a large number of points, and then use the primary features to form a more distinctive gait feature.

[0029] Preferably, the time dimension of the features can be processed by maximum pooling to select the most discriminative gait features.

[0030] Step 4: Use triplet loss and cross entropy loss to constrain local features respectively, and use weighted summation to calculate the final loss. Finally, use the backpropagation algorithm to iteratively update the parameters of the gait neural network.

[0031] To achieve the purpose of the present invention, based on the above method, the present invention further proposes a gait recognition method based on fine-grained motion pattern selection, including an image preprocessing module, a fine-grained feature extraction module, a horizontal segmentation module and a motion pattern selection module.

[0032] The image preprocessing module is used to scale the size of each frame image in the gait silhouette sequence to a uniform size;

[0033] The fine-grained feature extraction module is used to establish the local spatiotemporal information of each site in the gait sequence and pool the width dimension of the gait features to obtain the primary gait spatiotemporal features.

[0034] The horizontal segmentation module is used to horizontally segment the height dimension of the feature map, dividing the complete gait feature into local features, thereby constraining each local feature.

[0035] The motion pattern selection module is used to establish global spatiotemporal features for each local feature and pool the time dimension of gait features to obtain more discriminative features.

[0036] Beneficial effects

[0037] Compared with the prior art, the present invention has the following advantages:

[0038] The present invention effectively solves the problem of low gait recognition accuracy in existing methods and the problem of insufficient gait feature expression due to the lack of fine-grained feature extraction and motion pattern selection, and provides a gait recognition neural network method and system with a relatively simple and effective network structure for control in real scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 It is the overall flow chart of the method of the present invention;

[0040] Figure 2 It is an algorithm flow chart of the method of the present invention;

[0041] Figure 3 Schematic diagram of the structure of the system of the present invention. DETAILED DESCRIPTION

[0042] The present invention is further described in detail below with reference to the accompanying drawings and embodiments.

[0043] Example

[0044] like Figure 1 As shown, a gait recognition method based on fine-grained motion pattern selection includes the following steps:

[0045] Step S10: Acquire original gait video data and obtain a gait silhouette sequence.

[0046] Raw gait video data can be obtained from ordinary cameras.

[0047] Step S20: Data preprocessing, including extracting the contour edge of the human figure in the image, aligning the center of the human body with the center of the image, and scaling the gait image to a uniform size.

[0048] In this embodiment, the size of the gait images is uniformly scaled to 64×64.

[0049] Step S30: sending the gait data to the gait recognition neural network for processing to obtain representative local features.

[0050] Specifically, if Figure 2 As shown, step S30 includes the following steps:

[0051] Step S31: Using the backbone network based on fine-grained motion pattern selection, feature extraction is performed on the input sequence to obtain the primary features of each point in the gait sequence.

[0052] Specifically, the backbone network includes a fine-grained feature extraction module and a horizontal segmentation module.

[0053] Furthermore, the fine-grained feature extraction module consists of 3D convolution and maximum pooling, and for each point in the gait sequence For , it is learned from several points in the surrounding 3D space.

[0054]

[0055] Among them, (x, y, z) represents the coordinates of the point, a and b represent the b-th feature map in the a-th layer, and LeakReLU represents the activation function. and b ab They represent the weight and bias of the 3D convolution respectively, P, Q, and R represent the length, width, and height dimensions of the convolution kernel respectively, and m represents the size of the channel dimension; p, q, and r are all variables, and their maximum values ​​are (P-1), (Q-1), and (R-1), respectively.

[0056] Furthermore, the maximum pooling layer adopts a specially designed step size of 1×1×2. This pooling method only pools the width dimension of the feature map, fully retaining the most important time and height dimensions in the feature map.

[0057] Furthermore, the horizontal segmentation module horizontally segments the gait feature map along the height dimension, dividing the global gait features into local features, and constraining and processing them separately.

[0058]

[0059] in, represents the gait features extracted by the backbone network, represents the local gait features obtained by the horizontal segmentation module, i and j represent the i-th frame and j-th local features, respectively. AvgPool2d represents two-dimensional average pooling, and MaxPool2d represents two-dimensional maximum pooling.

[0060] Step S32: For each local feature, a motion pattern selection module is used to establish a global connection, and further obtain a discriminative global spatiotemporal feature while removing a large number of interference points.

[0061] Among them, the motion pattern selection module further establishes global spatiotemporal relationships from the primary features of a large number of points, and then uses the primary features to combine into a more distinctive gait feature.

[0062] Preferably, maximum pooling can be used to process the time dimension of the features to select the most discriminative gait features.

[0063] Step S33: Use triplet loss and cross entropy loss to constrain local features respectively, and use weighted summation to calculate the final loss. Finally, use the backpropagation algorithm to iteratively update the parameters of the gait neural network.

[0064] L tp =max(d(a,p)-d(a,n)+margin,0) (3)

[0065] Among them, L tp represents the triplet loss, d(a, p) represents the Euclidean distance between the selected sample and the positive sample pair with the same label, d(a, n) represents the Euclidean distance between the selected sample and the negative sample pair with different labels, a represents the anchor point, p represents the positive sample, and n represents the negative sample; mar g in represents the Euclidean distance difference between the selected sample and the positive sample pair and the negative sample pair.

[0066]

[0067] Among them, L ce represents the cross entropy loss, y represents the label of the sample, Represents the predicted label.

[0068] Loss = L tp +βL ce (5)

[0069] Among them, the final loss function Loss is the weighted sum of triple loss and cross entropy loss; β is a hyperparameter, which is usually set to 0.1.

[0070] Step S40: The gait features extracted by the neural network are matched with the features in the database. The similarity between the sequences is calculated to match the known identity sequence and obtain the pedestrian's identity information. Finally, the pedestrian's features are updated to the database as known pedestrian features.

[0071] like Figure 3 As shown, a gait recognition system based on fine-grained motion pattern selection includes an image acquisition module M10, an image preprocessing module M20, a gait recognition module M30 and a gait retrieval module M40.

[0072] The image acquisition module M10 is used to acquire original gait video data and obtain a gait silhouette sequence.

[0073] Image preprocessing module M20 is used to extract the outline edges of the human figure in the image, align the center of the human figure with the center of the image, and scale the gait image to a uniform size. Image preprocessing module M20 further includes image positioning submodule M21 and image alignment submodule M22.

[0074] The gait recognition module M30 first uses the gait recognition backbone network to extract features from the input sequence, obtaining primary features for each point in the gait sequence. It then performs pooling on the width dimension of the feature map to obtain local gait features. Finally, it establishes global connections for each local feature, further deriving discriminative global spatiotemporal features while removing a large number of interfering points. The gait recognition module M30 includes a fine-grained feature extraction submodule M31 and a motion pattern selection submodule M32.

[0075] The gait retrieval module M40 is used to match the features of the unknown identity sequence with the features in the storage library, obtain the pedestrian's identity information, and then update and store the features in the library. The gait retrieval module M40 includes a pedestrian identity retrieval submodule M41 and a gait feature storage submodule M42.

[0076] The connection relationship between the above modules is:

[0077] The output of the image acquisition module M10 is connected to the input of the image pre-processing module M20. Inside the image pre-processing module M20, the output of the image de-positioning sub-module M21 is connected to the input of the image alignment sub-module M22.

[0078] The output of the image preprocessing module M20 is connected to the input of the gait recognition module M30. Inside the gait recognition module M30, the output of the fine-grained feature extraction submodule M31 is connected to the input of the motion mode selection submodule M32.

[0079] The output of the gait recognition module M30 is connected to the input of the gait retrieval module M40. Inside the gait retrieval module M40, the output of the pedestrian identity retrieval submodule M41 is connected to the input of the gait feature storage submodule M42.

[0080] Through the above technical means, the problems of low gait recognition accuracy in existing technologies and insufficient gait feature expression due to the lack of fine-grained feature extraction and motion pattern selection are solved. This can provide a gait recognition solution with a simple and effective network structure for control in real scenarios.

Claims

1. A gait recognition method based on fine-grained motion pattern selection, characterized in that: The following steps are involved: Step 1: Preprocess the acquired gait video data to obtain the original gait silhouette sequence; The preprocessing includes extracting the outline edges of the human figure in the image, aligning the center of the human body with the center of the image, and scaling the gait image to a uniform size. Step 2: Using a backbone network based on fine-grained motion pattern selection, we extract features from the gait silhouette sequence and obtain the primary features of each point in the sequence. The gait silhouette sequence consists of any number of consecutive sampling frames, and there are continuous movement changes between adjacent frames; the backbone network extracts gait features from the input sequence; First, based on fine-grained feature extraction, the spatiotemporal features of each point in the sequence are extracted to form a spatiotemporal feature map; Then, through horizontal pooling, the spatiotemporal feature map is horizontally divided into a fixed number of features, and the local features are optimized separately; The backbone network includes a fine-grained feature extraction module and a horizontal segmentation module; Among them, the fine-grained feature extraction module consists of 3D convolution and maximum pooling. For each point in the gait sequence, For example, it is learned from several points in the surrounding 3D space: Among them, (x, y, z) represents the coordinates of the point, a and b represent the b-th feature map in the a-th layer, and LeakReLU represents the activation function. b ab Represent the weight and bias of 3D convolution respectively, P, Q, R represent the height, width and channel dimension of the convolution kernel respectively, and m represents the size of the channel dimension; p, q, r are all variables, and their maximum values ​​are P-1, Q-1 and R-1 respectively; The horizontal segmentation module segments the gait feature graph horizontally along the height dimension, dividing the global gait features into local features, and constraining and processing them separately: in, represents the local gait features obtained by the horizontal segmentation module, represents the gait features extracted by the backbone network, i and j represent the i-th frame and j-th local features respectively; AvgPool2d represents two-dimensional average pooling, and MaxPool2d represents two-dimensional maximum pooling; Among them, motion pattern selection is to further establish the global spatiotemporal relationship from the primary features of a large number of points, and then use the primary features to form a more discriminative gait feature; Step 3: For each local feature, motion pattern selection is used to establish a global connection, and further obtain discriminative spatiotemporal features while removing interference points; Step 4: Use triplet loss and cross entropy loss to constrain local features respectively, and use weighted summation method to calculate the final loss; Finally, the back-propagation algorithm is used to iteratively update the parameters of the gait neural network.

2. The gait recognition method based on fine-grained motion pattern selection according to claim 1, characterized in that: In step 3, the maximum pooling method is used to process the time dimension of the features and select the most discriminative gait features.

3. The gait recognition method based on fine-grained motion pattern selection according to claim 1, characterized in that: The maximum pooling layer uses a specially designed 1×1×2 step size. This pooling method only pools the width dimension of the feature map, fully retaining the most important time and height dimensions in the feature map.

4. The gait recognition method based on fine-grained motion pattern selection according to claim 1, characterized in that: Step 4 is implemented as follows: L tp =max(d(a,p)-d(a,n)+margin,0) Among them, L tp represents the triplet loss, d(a, p) represents the Euclidean distance between the selected sample and the positive sample pair with the same label, d(a, n) represents the Euclidean distance between the selected sample and the negative sample pair with different labels, a represents the anchor point, p represents the positive sample, and n represents the negative sample; margin represents the Euclidean distance difference between the selected sample and the positive sample pair and the negative sample pair; Among them, L ce represents the cross entropy loss, y represents the label of the sample, represents the predicted label; Loss=L tp +βL ce Among them, the final loss function Loss is the weighted sum of triple loss and cross entropy loss, and β is a hyperparameter.

5. A gait recognition system based on fine-grained motion pattern selection, characterized in that: It includes image acquisition module, image preprocessing module, gait recognition module and gait retrieval module; The image acquisition module is used to obtain the original gait video data and obtain the gait silhouette sequence; The image preprocessing module is used to extract the outline edges of the human figure in the image, align the center of the human body with the center of the image, and scale the gait image to a uniform size. The image preprocessing module includes an image positioning submodule and an image alignment submodule. The gait recognition module first uses the gait recognition backbone network to extract features from the input sequence, obtaining primary features for each point in the gait sequence. It then pools the width dimension of the feature map to obtain local gait features. Finally, it establishes global connections for each local feature and further obtains discriminative global spatiotemporal features while removing interference points. The gait recognition module includes a fine-grained feature extraction submodule and a motion pattern selection submodule. The backbone network includes a fine-grained feature extraction module and a horizontal segmentation module. Among them, the fine-grained feature extraction module consists of 3D convolution and maximum pooling. For each point in the gait sequence, For example, it is learned from several points in the surrounding 3D space: Among them, (x, y, z) represents the coordinates of the point, a and b represent the b-th feature map in the a-th layer, and LeakReLU represents the activation function. b ab Represent the weight and bias of 3D convolution respectively, P, Q, R represent the height, width and channel dimension of the convolution kernel respectively, and m represents the size of the channel dimension; p, q, r are all variables, and their maximum values ​​are P-1, Q-1 and R-1 respectively; The horizontal segmentation module segments the gait feature graph horizontally along the height dimension, dividing the global gait features into local features, and constraining and processing them separately: in, represents the local gait features obtained by the horizontal segmentation module, represents the gait features extracted by the backbone network, i and j represent the i-th frame and j-th local features respectively; AvgPool2d represents two-dimensional average pooling, and MaxPool2d represents two-dimensional maximum pooling; The gait retrieval module is used to match the features of the unknown identity sequence with the features of the library, obtain the pedestrian's identity information, and then update and store the features in the library; the gait retrieval module includes a pedestrian identity retrieval submodule and a gait feature storage submodule; The connection relationship between the above modules is: The output end of the image acquisition module is connected to the input end of the image preprocessing module; inside the image preprocessing module, the output end of the image de-positioning submodule is connected to the input end of the image alignment submodule; The output of the image preprocessing module is connected to the input of the gait recognition module; within the gait recognition module, the output of the fine-grained feature extraction submodule is connected to the input of the motion pattern selection submodule; The output end of the gait recognition module is connected to the input end of the gait retrieval module; inside the gait retrieval module, the output end of the pedestrian identity retrieval submodule is connected to the input end of the gait feature storage submodule.

Images