Dynamic local fusion network gait recognition method and system

Abstract

The invention belongs to the field of biological feature recognition, and discloses a dynamic local fusion network gait recognition method and system, the dynamic local fusion network gait recognition method comprises a dynamic local fusion network model, and the dynamic local fusion network model is composed of a feature extractor, a global perception flow and a local detail flow; the method comprises the following steps: firstly, taking a skeleton sequence extracted from an attitude estimator as input, and performing spatio-temporal feature extraction through a feature extractor consisting of an ST-GCN module with a residual error and a bottleneck structure; the extracted data respectively flows into a global perception stream and a local detail stream to obtain global feature representation and local feature representation of the gait; and splicing the obtained global features and local features to obtain the gait of the target person, and obtaining a gait recognition result according to the similarity with a known skeleton sequence. The method not only pays attention to global skeleton motion information, but also performs network model modeling on local limb parts, so that gait features with higher identification degree can be extracted.

Description

Technical Field

[0001] The present invention belongs to the technical field of biometric identification, and in particular relates to a dynamic local fusion network gait recognition method and system. Background Art

[0002] At present, gait recognition is a biometric technology that identifies human gait patterns based on posture characteristics. Compared with other biometric methods such as facial recognition, iris scanning and fingerprint recognition, gait recognition has unique advantages. This technology has been widely used in fields such as video surveillance and crime investigation. As an aspect of computer vision research, the essence of gait recognition lies in deciphering the most unique changes in gait patterns. However, gait is very sensitive to factors such as clothing, carried objects, cluttered backgrounds and occlusions in complex environments. The main challenge of gait recognition is to obtain robust features that are not affected by these factors.

[0003] In recent years, many researchers have proposed appearance-based gait recognition models, using silhouette images as input and achieving excellent recognition accuracy. The paper "Hanqing Chao, Yiwei He, Junping Zhang, and Jianfeng Feng. Gaitset: Regarding gait as a set for cross-view gait recognition. In Proceedings of the AAAI conference on artificial intelligence, pages 8126–8133, 2019" introduces a method called GaitSet that treats gait sequences as sets. This simple and efficient method was adopted by subsequent studies. The paper "Chao Fan, Yunjie Peng, Chunshui Cao, Xu Liu, Saihui Hou, Jiannan Chi, Yongzhen Huang, Qing Li, and Zhiqiang He. Gaitpart: Temporal part-based model for gait recognition. In Proceedings of the IEEE / CVF conference on computer vision and pattern recognition, pages 14225–14233, 2020." introduces a method called GaitPart that emphasizes local motion details and temporal dependencies. The paper "Beibei Lin, Shunli Zhang, and Xin Yu. Gait recognition via effective global-local feature representation and local temporal aggregation. In Proceedings of the IEEE / CVF International Conference on Computer Vision, pages 14648–14656, 2021." introduces a method called GaitGL that solves the problem of lack of fine details in spatial global gait representation.The paper "Zhen Huang, Dixiu Xue, Xu Shen, Xinmei Tian, ​​Houqiang Li, Jianqiang Huang, and Xian-Sheng Hua. 3D local convolutional neural networks for gait recognition. In Proceedings of the IEEE / CVF International Conference on Computer Vision, pages 14920–14929, 2021." introduces a method called 3DLocal to extract limb features through three-dimensional local operations with adaptive scales. These methods focus not only on extracting global features, but also on capturing detailed limb motion features. In addition, with the advancement of posture estimation networks, the accuracy of extracting skeleton sequences from video sequences is also improving. Skeleton-based gait recognition methods are expected to show stronger robustness in complex environments in the real world. In recent years, this model-based approach has received increasing attention. The document "Liao R, Yu S, An W, et al (2020) A model-based gait recognition method with body pose and human prior knowledge. Pattern Recognition 98: 107, 069." introduces a method called PoseGait that uses three-dimensional human pose and human prior knowledge to alleviate the problem of clothing changes. The document "Torben Teepe, Ali Khan, Johannes Gilg, Fabian Herzog, Stefan. and Gerhard Rigoll.GaitGraph:Graphconvolutional network for skeleton-based gait recognition.In ICIP,pages2314–2318,2021.》Introduced a method called GaitGraph, which introduced a graph convolutional network to learn two-dimensional skeleton gait representation. However, compared with appearance-based methods, current model-based methods still have shortcomings in performance. These methods mainly focus on the global features of bone sequences, extract features of the entire body skeleton, and then directly pool these features at the joint level. This ignores limb-specific motion information and the motion relationship between limbs. During human walking, different limb parts show unique motion patterns, such as arm swings and leg steps. The diverse motion patterns of these local limb parts provide additional information for the gait recognition process. It is worth noting that in a complete gait cycle, the motion information carried by parts such as arms and legs is often richer and more diverse than other parts.

[0004] Therefore, in order to fully explore the subtle differences in gait patterns and the synergy between limb parts, modeling of local limb parts becomes crucial. By focusing on and capturing the unique motion characteristics of different limb parts, gait patterns between individuals can be more accurately identified and distinguished. By combining global and local features, the nature of gait can be more fully understood, thereby improving the performance and robustness of gait recognition systems.

[0005] Through the above analysis, the problems and defects of the prior art are as follows:

[0006] (1) Existing skeleton-based gait recognition modeling methods only focus on the motion information of the global skeleton, while ignoring the unique motion characteristics of local limbs.

[0007] (2) After obtaining the movement characteristics of local limbs, how to integrate them is also a problem that exists.

[0008] (3) Existing appearance-based methods are very sensitive to complex environment datasets, cluttered backgrounds, and occlusion factors. Skeleton-based methods are gaining more and more attention because they are very robust to these influencing factors. However, the performance of current skeleton-based methods is far inferior to that of appearance-based methods. Summary of the invention

[0009] In view of the problems existing in the prior art, the present invention provides a dynamic local fusion network gait recognition method and system.

[0010] The present invention is implemented as follows: a dynamic local fusion network gait recognition method, the dynamic local fusion network gait recognition method includes a dynamic local fusion network model: the dynamic local fusion network model is composed of a feature extractor, a global perception stream and a local detail stream; firstly, the skeleton sequence extracted from the posture estimator is used as input, and a feature extractor composed of an ST-GCN module with a residual and bottleneck structure is used to extract spatiotemporal features; the extracted data flows into the global perception stream and the local detail stream respectively, and the global feature representation and local feature representation of the gait are obtained; the obtained global features and local features are spliced ​​to obtain the gait of the target person and the gait recognition result is obtained according to the similarity with the known skeleton sequence; the process of the overall network model is represented by the following formula:

[0011] Z i =F(X i )

[0012] Y i =concat(F L (Z i ),F G (Z i ))

[0013] Among them, F(·) represents the key motion pattern feature Z extracted from the skeleton sequence i The feature extractor, F G (·) indicates GAS, F L (·) indicates LDS, concat(·) indicates channel concatenation operation, and Y i Represents the final gait features used for loss calculation.

[0014] Furthermore, the dynamic local fusion network gait recognition method comprises the following steps:

[0015] S1: Initialize the model parameters, obtain the skeleton sequence data extracted by the pose estimator, and divide the data into training set and test set;

[0016] S2: Input the skeleton sequence data into the feature extractor for spatiotemporal feature extraction;

[0017] S3: After passing through the feature extractor, the extracted features flow into the global perception stream and the local detail stream respectively;

[0018] S4: The extracted global features and local features are concatenated as the final gait representation and the gait recognition result is obtained based on the similarity with the known skeleton sequence.

[0019] Furthermore, the feature extractor of the dynamic local fusion network gait recognition method is composed of a basic ST-GCN module and an ST-GCN module with a residual and bottleneck structure, and adopts a multi-branch input method; first, the input skeleton sequence is calculated into joint relative position information, speed information and bone information; the three types of information are respectively passed through a basic ST-GCN module and two ST-GCN modules with a residual and bottleneck structure to extract spatiotemporal features; and then the three obtained features are spliced ​​and passed through six ST-GCN modules with a residual and bottleneck structure in turn.

[0020] Furthermore, the calculation of joint relative position information, velocity information and skeleton information in the multi-branch input method of the feature extractor is expressed as:

[0021] R = {v t,n -v t,c |t,n∈N 0 ,t <T,n<N}

[0022] F={v t+i,n -v t,n |t,n∈N 0 ,i∈{1,2},t <T-2,n<N}

[0023]

[0024] Among them, v t,c represents the central joint of the entire skeleton, Indicates that v t,n At the other end of the connected bone, R is the relative position of the joint, F is the movement speed, L is the bone length, and A is the joint angle.

[0025] Furthermore, the local detail flow of the dynamic local fusion network gait recognition method consists of a skeleton limb mapping module and a dynamic fusion module. First, the feature Z i First, the limb features f are calculated through the skeleton limb mapping module p , the process is expressed by the formula:

[0026] f p =δ(FCs(D(TP(Z i ))))

[0027] Among them, δ(·) represents the activation function, D(·) represents the process of dividing local limb features, FCs(·) represents the fully connected layer of each limb part, and TP(·) represents the maximum pooling in the time dimension.

[0028] Then the limb part feature f p The local motion feature f is calculated by the dynamic fusion module l , the process is expressed by the formula:

[0029]

[0030] Among them, MP(·) represents maximum pooling, represents the addition of elements, represents element-wise multiplication, A(f p ) represents the generated attention weight.

[0031] Attention weight A(f p ) is expressed as:

[0032]

[0033] Among them, σ(·) represents the sigmoid function, L(f p ) and G(f p ) represents local context features and global context features.

[0034] The context feature calculation process is expressed by the formula:

[0035] L(f p )=PWConv 1 (δ(PWConv 2 (f p )))

[0036] G(f p )=PWConv 1 (δ(PWConv 2 (MP(f p ))))

[0037] Among them, PWConv 1 (·) and PWConv 2 The kernel sizes of (·) are and Point-wise convolution.

[0038] Furthermore, the global perception flow of the dynamic local fusion network gait recognition method consists of global average pooling and a fully connected layer. The input is mapped through the global average pooling and then through the fully connected layer to obtain the output of the global perception flow. The process is expressed by the formula:

[0039] fx=F G (Z i )=FC(GP(Z i ))

[0040] Among them, GP(·) represents the global pooling process, FC(·) represents the fully connected layer, and f g Represents the global motion characteristics.

[0041] Another object of the present invention is to provide a dynamic local fusion network gait recognition system for implementing the dynamic local fusion network gait recognition method, the dynamic local fusion network gait recognition system comprising:

[0042] Initialize the model parameter module to obtain the skeleton sequence data extracted by the posture estimator and divide the data into training set and test set;

[0043] A feature extraction module, connected to the initialization model parameter module, is used to input the skeleton sequence data into the feature extractor for spatiotemporal feature extraction;

[0044] The global perception module is connected to the feature extraction module and is used to extract global features;

[0045] A local detail module, connected to the feature extraction module, is used to extract local features;

[0046] The splicing recognition module, connected with the global perception module and the local detail module, is used to splice the global features and the local features as the final gait representation, and obtain the gait recognition result according to the similarity with the known skeleton sequence.

[0047] Another object of the present invention is to provide a computer device, comprising a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor executes the steps of the dynamic local fusion network gait recognition method.

[0048] Another object of the present invention is to provide a computer-readable storage medium storing a computer program, which, when executed by a processor, enables the processor to perform the steps of the dynamic local fusion network gait recognition method.

[0049] Another object of the present invention is to provide an information data processing terminal, which is used to implement the dynamic local fusion network gait recognition system.

[0050] The advantages and positive effects of the technical solution to be protected by the present invention are:

[0051] First, the present invention uses a feature extractor composed of an ST-GCN module with a residual and bottleneck structure to extract spatiotemporal features, and calculates multiple branches for input. It not only pays attention to the global skeleton motion information, but also models the network model of local limb parts, so as to extract more recognizable gait features.

[0052] A skeleton limb mapping module is proposed. This module comprehensively models each limb part of the human skeleton. It first divides the overall skeleton sequence into five predefined limb parts, and then performs a maximum pooling operation on the joint motion information contained in each partition. This step aims to extract the unique motion features of each limb part. In addition, the features obtained from each limb part are individually mapped to the corresponding feature space through a fully connected layer. This process transforms limb-specific features into structured and informative representations. Unlike traditional methods that apply pooling at the joint level, this method aims to improve the ability to extract complex gait features. It can well extract the unique motion information of limb parts.

[0053] In order to promote information exchange and integration between different limb parts, the present invention proposes dynamic feature fusion. This method models global and local limb contextual motion information, and in particular, by utilizing global and local channel aggregation features, a dynamic mechanism for adaptively selecting relevant limb-specific features is realized. This complex framework effectively eliminates unnecessary redundant information while retaining the salient and unique properties of the data. This method helps to understand human motion from both global and local perspectives. It cleverly solves the challenges faced by previous methods and improves the ability to extract gait features. The skeleton-based gait recognition method achieves better results on large field datasets than the contour-based method. It can make the gait recognition network more robust to complex environments.

[0054] Second, the feature extractor of the present invention utilizes the advanced ST-GCN module, which can capture the gait characteristics of the human body very accurately. The present invention also integrates global and local information, not only focusing on the overall body movements, but also paying special attention to the movement of limb parts, so it can more comprehensively identify the gait of individuals. The skeleton limb mapping module can extract specific motion features from different limb parts and obtain unique limb motion features. The dynamic feature fusion module uses the global and local contextual information of the limb motion features to obtain a dynamic attention weight, effectively fusing the motion features of different limbs and eliminating unnecessary redundant information.

[0055] Third, as auxiliary evidence of the inventiveness of the claims of the present invention, it is also reflected in the following important aspects:

[0056] (1) The expected benefits and commercial value of the technical solution of the present invention after transformation are:

[0057] After the implementation of the present invention, more accurate identity recognition can be performed through the gait of pedestrians. Gait recognition will have extensive commercial value in the future, mainly in the following aspects:

[0058] Security and monitoring field: Gait recognition technology can be applied to various security and monitoring systems, including but not limited to smart access control systems, public transportation stations, airports, commercial centers, etc. By analyzing the gait of pedestrians, a higher level of identity authentication and recognition can be achieved, which can improve security and reduce illegal intrusions and fraud.

[0059] For example, in a highly secure research laboratory, a gait recognition system ensures that only authorized personnel enter. When a researcher approaches the entrance, the system analyzes their gait and compares it with pre-stored authorized user data. If the gait matches, the access control system will automatically unlock the door and allow the researcher to enter. If the gait does not match or there is an anomaly, the system will trigger a security alarm.

[0060] Financial sector: Financial institutions use gait recognition technology to improve the security of customer identity verification. This reduces the risk of fraudulent transactions and account access, improves customer trust, and reduces financial losses.

[0061] For example, a user conducts a cash withdrawal transaction in a bank branch. The bank's self-service terminal is equipped with gait recognition technology, and the user uses his or her unique gait for identity verification without having to remember complex passwords or carry ID documents. Once the gait is matched successfully, the user is free to conduct the transaction, which improves the security of the transaction.

[0062] Entertainment and Sports: Gait recognition technology can also be used in the fields of entertainment and sports. Virtual reality games better track players' body movements and gaits to provide a more immersive gaming experience. In sports training, gait analysis is used to improve athletes' skills and reduce sports injuries.

[0063] For example, a virtual reality game player is playing an adventure game. The game system uses gait recognition technology to capture the player's actual walking movements and map them to the character movement in the game. In this way, the player explores the game world by actually walking, which increases the interactivity and immersion of the game.

[0064] (2) The technical solution of the present invention fills the technical gap in the industry at home and abroad:

[0065] Skeleton-based gait recognition methods are robust to complex environments, but many traditional methods focus on global body movements and ignore the motion information of limb parts, which leads to low recognition accuracy. Ordinary graph convolutions have difficulty effectively extracting detailed limb motion information, which limits the capture of individual differences. As a result, the performance of traditional skeleton-based gait recognition methods in the wild and complex environments is not robust enough, and is even inferior to methods based on contour maps.

[0066] (3) The technical solution of the present invention solves the technical problems that people have been eager to solve but have never been able to solve successfully:

[0067] The present invention has a further solution to the problem of low robustness of gait recognition in complex environments, which is significantly improved compared with previous technologies. For the skeleton limb mapping module, in order to extract local limb movement information, the dynamic feature fusion mechanism is used to dynamically fuse different limb parts, eliminate redundant information, and make the gait recognition network show strong robustness in complex environments.

[0068] (4) Whether the technical solution of the present invention overcomes technical prejudice:

[0069] Contour-based methods show superior performance on small datasets, which leads to technical bias to some extent. However, this bias is conditional because small datasets are usually made in controlled environments and are not easily disturbed by noise and changes in the real world. When these methods are applied to large wild datasets, contour-based methods lose robustness due to environmental uncertainty and noise.

[0070] The present invention chooses to adopt a skeleton-based method and successfully overcomes this technical bias by introducing key technologies such as the advanced ST-GCN module with residual and bottleneck structures, the skeleton limb mapping module and the dynamic feature fusion mechanism. The skeleton-based method can capture the motion information of the human body more comprehensively and is not constrained by the environment, so it shows stronger robustness in large field data sets. This technical solution effectively solves the limitations of the contour-based method in complex environments, allowing the gait recognition system to play a role in a wider range of real-world applications.

[0071] Fourth, the significant technical progress of the technical solution of the present invention:

[0072] 1. Network model: This method uses a feature extractor composed of ST-GCN modules with residual and bottleneck structures to extract spatiotemporal features, and calculates multiple branches for input. It not only focuses on the global skeleton motion information, but also models the local limb parts.

[0073] Such a network structure can pay attention to both global motion information and local motion information at the same time, thus improving the recognition accuracy of the network.

[0074] 2. Skeleton limb mapping module: This module comprehensively models each limb part of the human skeleton and maps the movement characteristics of each limb separately.

[0075] This module allows the network to focus on the movement characteristics of local limbs and improve the ability to extract complex gait features.

[0076] 3. Dynamic feature fusion module: This method models the global and local limb contextual motion information and implements a dynamic mechanism for adaptively selecting relevant limb-specific features by leveraging global and local channel aggregation features.

[0077] This module effectively eliminates unnecessary redundant information while retaining the salient and unique attributes of the data. It helps to understand human motion both globally and locally, and improves the ability to extract gait features.

[0078] 4. Limb parts: Provide prior knowledge to the network by pre-defining the limb parts to which each joint belongs

[0079] This enables the network to extract the left hand, right hand, left leg, right leg, and head separately when extracting local limb information. BRIEF DESCRIPTION OF THE DRAWINGS

[0080] Figure 1 It is a flow chart of a dynamic local fusion network gait recognition method provided by an embodiment of the present invention.

[0081] Figure 2 This is a diagram of a dynamic local fusion network structure provided by an embodiment of the present invention.

[0082] Figure 3 It is a feature extractor structure diagram provided by an embodiment of the present invention.

[0083] Figure 4 It is a structural diagram of a skeleton limb mapping module provided in an embodiment of the present invention.

[0084] Figure 5 It is a structural diagram of a dynamic feature fusion module provided in an embodiment of the present invention.

[0085] Figure 6 It is a structural diagram of a dynamic local fusion network gait recognition system provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0086] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0087] The Dynamic Local Fusion Network (GaitDLF) gait recognition method extracts richer gait feature representations from global and local features. This method artificially sets the limb part to which each joint in the skeleton belongs, divides and maps the features according to different limb parts, and thus obtains unique gait features for each limb part. In addition, the global context information and local context information of the gait features of the local limb are used to dynamically fuse the unique motion features of each limb. By effectively aggregating the unique feature representations of different limb parts, the ability to distinguish the gait patterns of different subjects is enhanced.

[0088] like Figure 1 As shown, the dynamic local fusion network gait recognition method provided by the embodiment of the present invention includes the following steps:

[0089] S1: Initialize the model parameters, obtain the skeleton sequence data extracted by the pose estimator, and divide the data into training set and test set;

[0090] S2: Input the skeleton sequence data into the feature extractor for spatiotemporal feature extraction;

[0091] S3: After passing through the feature extractor, the extracted features flow into the global perception stream and the local detail stream respectively;

[0092] S4: The extracted global features and local features are concatenated as the final gait representation and the gait recognition result is obtained based on the similarity with the known skeleton sequence.

[0093] The specific implementation scheme of the dynamic local fusion network gait recognition method provided by the present invention is as follows:

[0094] S1: Initialize model parameters, obtain skeleton sequence data extracted by the pose estimator, and divide the data into training set and test set

[0095] 1. Initialize model parameters: This includes initializing the weights and biases of the neural network model. Usually, the initial values ​​of these parameters are random, but there are some specific initialization strategies that improve the efficiency and stability of model training, such as the Xavier initialization method or the He initialization method.

[0096] 2. Obtain skeleton sequence data extracted by the pose estimator: The pose estimator is a deep learning model that is used to extract the position and pose information of the human skeleton from the input video or image. This information is usually expressed in the form of a series of coordinate points, forming a time series, representing the dynamic changes of the human skeleton over a period of time.

[0097] 3. Data partitioning: Divide the data set into training set and test set. Use 80% of the data for training and the remaining 20% ​​for testing.

[0098] S2: Input skeleton sequence data into the feature extractor for spatiotemporal feature extraction

[0099] 1. The feature extractor is a deep neural network, such as a convolutional neural network (CNN) or a long short-term memory network (LSTM). The task of this network is to extract useful features from the skeleton sequence data. In particular, LSTM is particularly suitable for processing skeleton sequence data because it processes sequence data.

[0100] S3: After passing through the feature extractor, the extracted features flow into the global perception stream and the local detail stream respectively.

[0101] 1. The global perception stream and the local detail stream are two independent networks that process global and local features respectively.

[0102] 2. The global perception flow uses a larger field of view to observe the layout and overall movement pattern of the entire skeleton.

[0103] 3. Local detail flow focuses on a smaller field of view, focusing on a certain part of the skeleton, such as the movement of an arm or leg.

[0104] S4: The extracted global features and local features are concatenated as the final gait representation and the gait recognition result is obtained based on the similarity with the known skeleton sequence.

[0105] 1. Global features and local features are combined into a complete gait feature vector through splicing or other fusion methods (such as weighted averaging).

[0106] 2. The gait feature vector is then compared with known gait features to identify the gait by calculating the distance or similarity between them. Similarity calculation methods include Euclidean distance and cosine similarity.

[0107] 3. Finally, select the gait with the highest similarity as the recognition result.

[0108] By processing global and local features separately and then fusing these features, a more comprehensive and accurate gait representation is obtained. Then, gait recognition is performed by comparing it with known gaits.

[0109] like Figure 2As shown, the dynamic local fusion network model provided by the embodiment of the present invention is composed of a feature extractor, a global perception stream and a local detail stream; first, the skeleton sequence extracted from the posture estimator is used as input, and a feature extractor composed of an ST-GCN module with a residual and bottleneck structure is used to extract spatiotemporal features. The main purpose of this key step is to convert the original skeleton data into more expressive features; the extracted data flows into the global perception stream and the local detail stream respectively to obtain the global feature representation and local feature representation of the gait; the obtained global features and local features are spliced ​​to obtain the gait of the target person and the gait recognition result is obtained according to the similarity with the known skeleton sequence. During training, a triplet loss function is used to make the Euclidean distance between the anchor point and the positive sample closer, and the Euclidean distance between the anchor point and the negative sample farther. The process of the overall network model. The calculation of gait features is expressed by the following formula:

[0110] Z i =F(X i )

[0111] Y i =concat(F L (Z i ),F G (Z i ))

[0112] Among them, F(·) is responsible for extracting key motion patterns and features Z from the skeleton sequence i The feature extractor F G (·) indicates GAS, F L (·) indicates LDS, concat(·) indicates channel concatenation operation, and Y i Represents the final gait features used for loss calculation.

[0113] 1. Feature Extractor

[0114] like Figure 3 As shown in the figure, the feature extractor consists of a basic ST-GCN module and an ST-GCN module with residual and bottleneck structures, and adopts a multi-branch input method; first, the input skeleton sequence is calculated into joint relative position information, velocity information and bone information; the above three types of information are respectively extracted from the basic ST-GCN module and two ST-GCN modules with residual and bottleneck structures to extract spatiotemporal features; then the three obtained features are concatenated and sequentially passed through six ST-GCN modules with residual and bottleneck structures to extract spatiotemporal features. By using this extractor, the basic motion patterns and features in the skeleton sequence can be effectively captured, thus laying the foundation for the subsequent gait recognition task.

[0115] 2. Multi-branch computing

[0116] Multi-branch computation provides rich feature input for the entire network. The skeleton is described as G = (V, E), where V = {v 1 ,...,v N} represents the set of N joint point features, E represents the adjacency matrix A∈R N×N The calculated edge set of the skeleton, if the edge from v i Connect to v j , then A i,j =1, otherwise A i,j = 0. Each node consists of three channels v n =(x n ,y n ,c n ) with estimated x, y coordinates and keypoint confidence c. The skeleton sequence is defined as X = {v t,n ∈R 3 |t,n∈N 0 ,t <T,n<N},X∈R T×N×C

[0117] The calculation of joint relative position information, velocity information and bone information are expressed as follows:

[0118] R = {v t,n -v t,c |t,n∈N 0 ,t <T,n<N}

[0119] F={v t+i,n -v t,n |t,n∈N 0 ,i∈{1,2},t <T-2,n<N}

[0120]

[0121]

[0122] Among them, v t,c represents the central joint of the entire skeleton, Indicates that v t,n At the other end of the connected bone, R is the relative position of the joint, F is the movement speed, L is the bone length, and A is the joint angle.

[0123] 3. Local detail flow

[0124] Past methods usually use graph convolutional networks and temporal convolutional networks (TCN) to extract spatial and temporal features from skeleton sequences, and finally perform pooling operations on the joint dimension to achieve significant recognition accuracy. However, since most people's gait movements are very similar, accurate recognition usually requires subtle limb movements. Therefore, local limb behaviors in skeleton sequences are modeled to improve recognition accuracy. The key to the LDS module is to effectively divide the complex features extracted from the skeleton sequence into motion features related to different limb parts, and then dynamically fuse them to pay more attention to specific limb movements. The local detail stream consists of a skeleton limb mapping module and a dynamic fusion module.

[0125] 4. Skeleton limb mapping module

[0126] like Figure 4 As shown in the figure, the skeleton limb mapping module first performs temporal maximum pooling on the input, and then combines the limb parts according to the limb parts to which the joints belong, and combines them into five parts: left hand, right hand, left leg, right leg and head. The five parts are then subjected to maximum pooling to obtain the motion features of each part, so as to obtain the motion feature representation of each limb part. Then, a fully connected layer is used to map the features of each limb part to a distinguishable space, highlighting the motion information of the limb part. The process of this module is expressed by the formula:

[0127] f p =δ(FCs(D(TP(Z i ))))

[0128] Among them, δ(·) represents the activation function, D(·) represents the process of dividing local limb features, Fcs(·) represents the fully connected layer of each limb part, TP(·) represents the maximum pooling in the time dimension, and f p Indicates the characteristics of a body part.

[0129] 5. Dynamic Fusion Module

[0130] like Figure 5 As shown, the dynamic fusion module combines the limb part features f p Through the attention matrix A(f p ) is used for fusion, aiming to focus on important limb movement information, multiplying the obtained attention weight with the input point by point, and then adding the input as the residual to the result to obtain the output of the local detail stream. The process is expressed as:

[0131]

[0132] Among them, MP(·) represents maximum pooling, represents the addition of elements, represents element-wise multiplication, A(f p) represents the generated attention weight.

[0133] The calculation of the attention matrix utilizes the unique motion characteristics of each limb part to generate global channel features and local channel aggregation features to dynamically fuse information from different limb parts. p ) and G(f p ) are added point by point, and the attention weight is obtained through the sigmoid activation function. The purpose is to create an adaptive fusion weight for each limb part and channel. The attention weight A(f p ) is expressed as:

[0134]

[0135] Among them, σ(·) represents the sigmoid function, L(f p ) and G(f p ) represents local context features and global context features.

[0136] The calculation of local context features and global context features is to fuse the information of the five limb parts and keep it lightweight. Point-wise convolution is chosen as the channel context aggregator, which only aggregates the point-wise channels at each spatial position. p The local context features are obtained by point-by-point convolution, RELU activation function, point-by-point convolution in parallel, and the global context features are obtained by maximum pooling, point-by-point convolution, RELU activation function, and point-by-point convolution. The process is expressed as follows:

[0137] L(f p )=PWConv 1 (δ(PWConv 2 (f p )))

[0138] G(f p )=PWConv 1 (δ(PWConv 2 (MP(f p ))))

[0139] in,

[0140] PWConv 1 (·) and PWConv 2 The kernel sizes of (·) are and It is worth noting that L(f p ) retains the same shape as the input features, thereby preserving and emphasizing subtle details in low-level features.

[0141] 6. Global Awareness Flow

[0142] In addition to the local detail flow, the perception of the global skeleton structure is also crucial. Make full use of the global skeleton framework and local limb information to provide rich features for gait recognition. The global perception flow aims to extract global motion features from the skeleton sequence, thereby giving the system global recognition capabilities. The global perception flow consists of global average pooling and a fully connected layer. After the input is passed through the global average pooling, the dimension of the high-level features of the input is reduced to obtain an overall representation of the global motion, which is then mapped through the fully connected layer to make it more separable and discriminable. This process can capture the overall motion pattern in the skeleton sequence and provide information support from a global perspective for the recognition task. The process is also expressed by the formula:

[0143] f g =F G (Z i )=PC(GP(Z i ))

[0144] Among them, GP(·) represents the global pooling process, FC(·) represents the fully connected layer, and f g Represents the global motion characteristics.

[0145] like Figure 6 As shown, the dynamic local fusion network gait recognition system includes:

[0146] Initialize the model parameter module to obtain the skeleton sequence data extracted by the posture estimator and divide the data into training set and test set;

[0147] A feature extraction module, connected to the initialization model parameter module, is used to input the skeleton sequence data into the feature extractor for spatiotemporal feature extraction;

[0148] The global perception module is connected to the feature extraction module and is used to extract global features;

[0149] A local detail module, connected to the feature extraction module, is used to extract local features;

[0150] The splicing recognition module, connected with the global perception module and the local detail module, is used to splice the global features and the local features as the final gait representation, and obtain the gait recognition result according to the similarity with the known skeleton sequence.

[0151] 1. Personal identity verification

[0152] In this application example, the dynamic local fusion network model system is applied to the field of biometrics to identify and verify the identity of a person. The following are the specific steps and application descriptions:

[0153] Data collection and preparation: First, obtain pedestrian gait data from security cameras or sensors. This data includes information such as gait cycle, stride length, and pace. In addition, it is also necessary to collect identity information related to each pedestrian, such as name, photo, etc.

[0154] Model construction: A dynamic local fusion network model system is established, which includes a gait recognition network and an identity verification process. The gait recognition network is used to extract the final gait features from the gait data. The identity verification module compares the high-dimensional features with the previously stored identity information to verify the identity of the pedestrian.

[0155] Model training: A large amount of gait data and corresponding identity information are used to train the model. By optimizing the extraction of gait features and the weight parameters of the identity verification module, the model can accurately identify the identity of pedestrians.

[0156] Validation and testing: Use test data from different pedestrians to validate the model's performance. Evaluate the model's accuracy, false positive rate, and robustness to ensure it can reliably authenticate in real-world scenarios.

[0157] Application: This method is applied to scenarios that require highly secure identity authentication, such as access control systems in financial institutions, secure access control in government agencies, and access control for highly sensitive data.

[0158] 2. Health monitoring and diagnosis

[0159] In this application example, the dynamic local fusion network model system is applied to the health field to monitor and diagnose the health status of individuals. The following are the specific steps and application descriptions:

[0160] Data collection and preparation: Use security cameras or sensors to collect individual gait data, including gait cycle, stride length, pace, etc. At the same time, collect individual health information such as heart rate, respiratory rate, posture, etc.

[0161] Model construction: A dynamic local fusion network model system is established, which includes a gait feature extraction network and a health monitoring module. The gait feature extraction network extracts features from gait data. The health monitoring module combines gait features with health information to monitor the health status of individuals in real time.

[0162] Model training: Use a large amount of individual gait data and health information to train the model. By optimizing the extraction of gait features and the weight parameters of the health monitoring module, the model can accurately diagnose the health status of individuals.

[0163] Verification and testing: Evaluate the performance of the model in health monitoring and diagnosis by verifying the test data of different individuals. Consider indicators such as accuracy, sensitivity, and specificity to ensure that the model has good performance in health monitoring.

[0164] Application: This method is applied to individual health monitoring systems to monitor the individual's gait and health status in real time. It can be used in medical institutions, fitness fields, elderly care, etc. to provide timely health assessment and diagnosis for individuals.

[0165] 3. Pedestrian tracking and safety monitoring

[0166] In this application example, the dynamic local fusion network model system is applied to the field of security monitoring for pedestrian tracking and security monitoring. The following are the specific steps and application descriptions:

[0167] Data collection and preparation: Install security cameras and sensors in public places such as shopping malls, airports, or large events to collect pedestrian gait data and identity information.

[0168] Model construction: Using the dynamic local fusion network model system, a pedestrian tracking and safety monitoring system is established. The gait recognition network is responsible for extracting features from gait data, while the identity verification module compares it with previously stored pedestrian identity information.

[0169] Model training: The model is trained using gait data and identity information captured by multiple cameras. By continuously optimizing the extraction of gait features and the parameters of the identity verification module, the system can accurately identify and track pedestrians.

[0170] Verification and testing: Use the system in real public places and verify it with test data of different pedestrians. Evaluate the system's tracking accuracy, identity verification accuracy, and ability to handle multiple pedestrians in parallel.

[0171] Application: This system can be applied to enhance security surveillance in public places, monitor pedestrian activities and identify potential threats or suspicious behavior. This application can be used to improve the security of shopping malls, transportation hubs, large event venues, etc., ensure pedestrian safety and provide emergency response.

[0172] The dynamic local fusion network gait recognition method provides three specific embodiments and corresponding implementation schemes:

[0173] These embodiments require certain hardware and software resources, such as high-performance computers, data storage devices, professional data processing and analysis software, etc. The training and testing process of the model takes several days to several weeks, depending on the amount of data and the complexity of the model.

[0174] The dynamic local fusion network gait recognition method provided by the application embodiment of the present invention is applied to a computer device, wherein the computer device includes a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor executes the steps of the dynamic local fusion network gait recognition method.

[0175] The dynamic local fusion network gait recognition method provided by the application embodiment of the present invention is applied to an information data processing terminal, and the information data processing terminal is used to implement the dynamic local fusion network gait recognition system.

[0176] It should be noted that the embodiments of the present invention are implemented by hardware, software, or a combination of software and hardware. The hardware part is implemented using dedicated logic; the software part is stored in a memory and executed by an appropriate instruction execution system, such as a microprocessor or dedicated design hardware. A person of ordinary skill in the art understands that the above-mentioned devices and methods are implemented using computer executable instructions and / or contained in a processor control code, such as such as a carrier medium such as a disk, CD or DVD-ROM, a programmable memory such as a read-only memory (firmware), or a data carrier such as an optical or electronic signal carrier. Such code is provided on a carrier medium such as a disk, CD or DVD-ROM, a programmable memory such as a read-only memory (firmware), or a data carrier such as an optical or electronic signal carrier. The device and its modules of the present invention are implemented by hardware circuits such as very large-scale integrated circuits or gate arrays, semiconductors such as logic chips, transistors, etc., or programmable hardware devices such as field programmable gate arrays, programmable logic devices, etc., and are also implemented by software executed by various types of processors, and are also implemented by a combination of the above-mentioned hardware circuits and software, such as firmware.

[0177] The present invention has achieved positive results in the research and development process and has great advantages over the prior art. In order to verify the effectiveness of GaitDLF for gait recognition, it was verified on the GREW field dataset and compared with the traditional model. This experiment was conducted on a single NVIDIA GeForce GTX 3080 GPU using the PyTorch framework. The threshold of the loss function was set to 0.2. The embedding dimension of the fully connected layer was specified as 128. For training, the batch size was configured as (N, S), where N corresponds to the number of subjects and S represents the number of samples collected from each subject. The training batch was set to (48, 12). During the optimization process, the SGD optimizer was used, with an initial learning rate of 0.1, a momentum of 0.9, a weight decay of 0.0005, and a learning rate decay of 10 times every 30K, 60K, 90K, and 120K steps, for a total of 150K rounds of training.

[0178] Table 1 shows the comparison of the accuracy of the proposed model and other advanced models on the GREW dataset:

[0179] Table 1

[0180]

[0181] Table 1 shows that among similar skeleton-based methods, the method of the present invention has achieved the best results on the GREW dataset, which fully demonstrates the effectiveness of the method of the present invention. In contour-based methods, the contour image contains not only gait features, but also other recognizable visual clues (such as clothes, bags), and the skeleton-based method only extracts gait features from the dynamic bone sequence of the subject. Therefore, for small and medium-sized data sets under constrained environments, the contour-based method still has the highest gait recognition accuracy. Although the skeleton-based method has gradually caught up with the contour-based method on the field dataset GREW in the past two years, the superiority of the skeleton-based method has not been fully demonstrated. The method of the present invention is compared with the most advanced method on the GREW dataset, and the method of the present invention has also achieved the best results, which fully demonstrates that the method of the present invention can learn rich and unique gait features from large datasets.

[0182] Table 2 shows the comparison of the accuracy of the baseline model and the addition of the skeleton limb mapping module and the dynamic fusion module on the GREW dataset:

[0183] Table 2

[0184]

[0185] In Table 2, Baseline represents the two modules without adding local detail streams, A represents the skeleton limb mapping module, and B represents the dynamic fusion module. The results show that the network model with the addition of the skeleton limb mapping module and the dynamic fusion module can better extract more recognizable gait features, thereby improving the accuracy.

[0186] The above description is only a specific implementation mode of the present invention, but the protection scope of the present invention is not limited thereto. Any modifications, equivalent substitutions and improvements made by any technician familiar with the technical field within the technical scope disclosed by the present invention and within the spirit and principle of the present invention should be covered by the protection scope of the present invention.

Claims

1. A dynamic local fusion network gait recognition method, characterized in that: The dynamic local fusion network model takes the skeleton sequence extracted from the posture estimator as input, and performs spatiotemporal feature extraction through a feature extractor composed of an ST-GCN module with a residual and bottleneck structure; the extracted data flows into the global perception stream and the local detail stream respectively to obtain the global feature representation and local feature representation of the gait; the obtained global features and local features are spliced ​​to obtain the gait of the target person and the gait recognition result is obtained based on the similarity with the known skeleton sequence.

2. The dynamic local fusion network gait recognition method according to claim 1, characterized in that: The process of the overall network model is expressed by the following formula: Z i =F(X i Y i =concat(F L (Z i ),F G (Z i )) Among them, F(·) is responsible for extracting key motion patterns and features Z from the skeleton sequence i The feature extractor, F G (·) indicates GAS, F L (·) indicates LDS, concat(·) indicates channel concatenation operation, and Y i Represents the final gait features used for loss calculation.

3. The dynamic local fusion network gait recognition method as claimed in claim 1, characterized in that: The dynamic local fusion network gait recognition method comprises the following steps: S1: Initialize the model parameters, obtain the skeleton sequence data extracted by the pose estimator, and divide the data into training set and test set; S2: Input the skeleton sequence data into the feature extractor for spatiotemporal feature extraction; S3: After passing through the feature extractor, the extracted features flow into the global perception stream and the local detail stream respectively; S4: The extracted global features and local features are concatenated as the final gait representation and the gait recognition result is obtained based on the similarity with the known skeleton sequence.

4. The dynamic local fusion network gait recognition method as claimed in claim 3, characterized in that: Implementation method of S1: Initialize model parameters; Get the skeleton sequence data extracted by the pose estimator: The pose estimator is a deep learning model that is used to extract the position and pose information of the human skeleton from the input video or image. This information is usually expressed in the form of a series of coordinate points, forming a time series, representing the dynamic changes of the human skeleton over a period of time. Data partitioning: Divide the dataset into training and testing sets; use 80% of the data for training and the remaining 20% ​​for testing.

5. The dynamic local fusion network gait recognition method as claimed in claim 3, characterized in that: S2 implementation method: The feature extractor extracts useful features from the skeleton sequence data. In particular, LSTM is particularly suitable for processing skeleton sequence data because it processes sequence data.

6. The dynamic local fusion network gait recognition method as claimed in claim 3, characterized in that: S3 implementation method: The global perception stream and the local detail stream are two independent networks that process global and local features respectively; The global perception stream uses a larger field of view to observe the layout and overall movement pattern of the entire skeleton; Local detail flow focuses on a smaller field of view, concentrating on the movement of a certain part of the skeleton, such as an arm or leg.

7. The dynamic local fusion network gait recognition method as claimed in claim 3, characterized in that: S4 implementation method: The global features and local features are combined into a complete gait feature vector through splicing or other fusion methods; The gait feature vector is then compared with known gait features to identify the gait by calculating the distance or similarity between them; similarity calculation methods include Euclidean distance and cosine similarity; Finally, the gait with the highest similarity is selected as the recognition result.

8. The dynamic local fusion network gait recognition method as claimed in claim 3, characterized in that: The feature extractor of the dynamic local fusion network gait recognition method is composed of a basic ST-GCN module and an ST-GCN module with a residual and bottleneck structure, and adopts a multi-branch input method; first, the input skeleton sequence is calculated into joint relative position information, speed information and bone information; The three types of information are respectively passed through the basic ST-GCN module and two ST-GCN modules with residual and bottleneck structures to extract spatiotemporal features; After concatenating the three features, they are passed through six ST-GCN modules with residual and bottleneck structures in sequence; The calculation of joint relative position information, velocity information and skeleton information in the multi-branch input method of the feature extractor is expressed as: R={v t,n -v t,c |t,n∈N0,t<T,n<N} F={v t+i,n -v t,n |t,n∈N0,i∈{1,2},t<T-2,n<N} Among them, v t,c represents the central joint of the entire skeleton, Indicates that v t,n At the other end of the connected bone, R is the relative position of the joint, F is the movement speed, L is the bone length, and A is the joint angle.

9. The dynamic local fusion network gait recognition method as claimed in claim 3, characterized in that: The local detail flow of the dynamic local fusion network gait recognition method consists of a skeleton limb mapping module and a dynamic fusion module. First, the feature Z i First, the limb features f are calculated through the skeleton limb mapping module p , the process is expressed by the formula: f p =δ(FCs(D(TP(Z i )))) Among them, δ(·) represents the activation function, D(·) represents the process of dividing local limb features, FCs(·) represents the fully connected layer of each limb part, and TP(·) represents the maximum pooling in the time dimension; Then the limb part feature f p The local motion feature f is calculated by the dynamic fusion module l , the process is expressed by the formula: Among them, MP(·) represents maximum pooling, ⊕ represents element addition, represents element-wise multiplication, A(f p ) represents the generated attention weight; Attention weight A(f p ) is expressed as: A(f p )=σ(L(f p )⊕G(f p )) Among them, σ(·) represents the sigmoid function, L(f p ) and G(f p ) represents local context features and global context features; The context feature calculation process is expressed by the formula: L(f p )=PWConv1(δ(PWConv2(f p ))) G(f p )=PWConv1(δ(PWConv2(MP(f p )))) The kernel sizes of PWConv1(·) and PWConv2(·) are and Point-wise convolution of ; The global perception flow of the dynamic local fusion network gait recognition method consists of global average pooling and a fully connected layer. The input is mapped through the global average pooling and then through the fully connected layer to obtain the output of the global perception flow. The process is expressed as follows: f g =F G (From i )=FC(GP(Z i )) Among them, GP(·) represents the global pooling process, FC(·) represents the fully connected layer, and f g Represents the global motion characteristics.

10. A dynamic local fusion network gait recognition system implementing the dynamic local fusion network gait recognition method according to any one of claims 1 to 7, characterized in that: The dynamic local fusion network gait recognition system comprises: Initialize the model parameter module to obtain the skeleton sequence data extracted by the posture estimator and divide the data into training set and test set; A feature extraction module, connected to the initialization model parameter module, is used to input the skeleton sequence data into the feature extractor for spatiotemporal feature extraction; The global perception module is connected with the feature extraction module and is used to extract global features; A local detail module, connected to the feature extraction module, is used to extract local features; The splicing recognition module, connected with the global perception module and the local detail module, is used to splice the global features and the local features as the final gait representation, and obtain the gait recognition result according to the similarity with the known skeleton sequence.

Images