Gait recognition method based on video human body analysis time sequence consistency

Abstract

A gait recognition method based on video human body analysis time sequence consistency comprises the steps that firstly, human body part analysis is conducted on a first frame of a video sequence, initial masks are generated, then the masks serve as prompt information, follow-up frames are tracked and segmented, and time sequence consistency is ensured. According to the method, dynamic features of human body parts changing along with time can be captured, so that rich gait features including time sequence features (TP) and human body part spatial features (HP) are extracted. The features are input into a gait recognition model for identity recognition. By combining a human body analysis technology and image tracking, the problem of inconsistency of a traditional method during multi-frame video processing is solved, the analysis precision is improved, and the feature extraction capability is enhanced through fine-grained analysis. Besides, the method also optimizes the synergistic effect of human body analysis and gait recognition, so that the analyzed part features can directly provide input with higher discriminative power for gait recognition, and more efficient cooperative work is realized.

Description

Technical Field

[0001] The present invention relates to computer vision technology, and in particular to a gait recognition method based on temporal consistency of video human body analysis. Background Art

[0002] Gait recognition is a computer vision technology that uses human gait characteristics to identify individuals. It identifies individuals by analyzing their posture and motion characteristics when they walk. It uses the unique gait patterns of humans to effectively identify individuals even at long distances or under occlusion. As a contactless and long-distance biometric method, gait recognition has a wide range of applications in security monitoring, public safety, contactless authentication and other fields. A core challenge of gait recognition technology is how to effectively extract stable gait features from videos, which requires taking into account the dynamic changes of the human body during walking and the temporal consistency in video sequences.

[0003] Traditional gait recognition methods usually rely on static images or single-frame-based feature extraction techniques. Although these methods can recognize human features to a certain extent, they ignore the temporal information in the video, resulting in inaccurate capture of dynamic changes in gait and inability to effectively distinguish the subtle gait differences between different individuals.

[0004] In addition, the importance of human parsing in gait recognition has gradually been recognized. Through detailed analysis of various parts of the human body, the recognition system can better extract key information (such as the movement trajectory of the legs and arms), thereby improving the accuracy of recognition.

[0005] At present, the development of gait recognition technology faces several major challenges:

[0006] 1. Lack of timing information: Most analysis methods are only applicable to single-frame image processing and cannot effectively utilize the timing information of video sequences, resulting in inaccurate feature extraction in gait recognition.

[0007] 2. Insufficient accuracy in parsing human body parts: Existing parsing methods cannot accurately distinguish different parts of the human body when dealing with complex backgrounds or blurred videos, which in turn affects the performance of gait recognition.

[0008] 3. The need for efficient tracking capabilities: In order to ensure the consistency of gait analysis, the system needs to be able to track the same human body part in multiple frames of video, which poses new challenges to traditional analysis and gait recognition models.

[0009] Existing gait recognition technologies are mainly divided into template-based methods and deep learning-based methods.

[0010] Template-based methods: Early gait recognition technologies relied on image processing methods to extract dynamic changes in human body contours in videos for recognition. Among these methods, GEI (Gait Energy Image) is one of the more classic template methods. GEI averages the human body contours within a gait cycle to generate an energy image for subsequent classifiers. Although this method is simple and intuitive, it relies too much on static image features and lacks the ability to capture dynamic features of the human body, making it difficult to handle gait recognition tasks in complex backgrounds or from multiple perspectives.

[0011] Methods based on deep learning: With the development of deep learning, gait recognition has gradually turned to feature extraction and classification methods based on neural networks. This type of method extracts local and global features of the human body in gait videos through convolutional neural networks (CNNs), and combines time series models such as long short-term memory networks (LSTMs) to capture time series information in videos. However, although this method solves the problem of insufficient static features to a certain extent, its ability to finely analyze various parts of the human body is limited. Common deep learning-based gait recognition systems such as GaitSet and GaitPart have achieved significant improvements in results, but these models still focus on the extraction of overall gait features and lack the ability to extract fine-grained features of human body parts.

[0012] Human body parsing technology: In the field of human body parsing, existing technologies mainly focus on the segmentation of human body parts in static images. M2FP is a pre-trained model for static images that can effectively segment different parts of the human body. However, when the parsing task is extended to video sequences, M2FP cannot guarantee consistency between multiple frames, resulting in unstable parsing results when processing videos. The recent SegmentAnything2 (SAM2) model has excellent target tracking and segmentation capabilities and can maintain consistent target tracking between multiple video frames. However, SAM2 still focuses mainly on the segmentation of a single object, and its potential has not yet been fully explored for parsing tasks involving complex human body parts.

[0013] Disadvantages of existing technology:

[0014] 1. Lack of consistency in temporal information: Existing human parsing methods based on single-frame images, such as the M2FP model, perform well on static images, but cannot guarantee the consistency of human body part parsing results between consecutive frames when processing video sequences. This results in obvious jitter and inconsistency in the parsing results of various parts of the human body when parsing multiple frames of video, and fails to capture important temporal information in gait recognition. This inconsistency not only reduces the accuracy of human parsing, but also affects the ability of the gait recognition system to extract stable features.

[0015] 2. Insufficient analysis of human body parts: Gait recognition tasks require high levels of detail in human body parts, but most existing gait recognition methods rely only on overall contour features or coarse-grained local features, lacking detailed analysis of human body parts (such as arms and legs). In particular, in scenes with complex backgrounds, lighting changes, or large diversity of human body postures, existing methods often cannot provide detailed segmentation results, resulting in a decrease in gait recognition accuracy.

[0016] 3. Inadequate use of dynamic changes in human body parts: Traditional gait recognition methods, such as GaitSet and GaitPart, mainly rely on overall features extracted from video sequences, and fail to fully utilize the dynamic features of various parts of the human body that change over time. The movement trajectory of human body parts (such as arms and legs) in gait plays an important role in identity recognition, but existing methods cannot accurately capture these dynamic changes, resulting in limited robustness and accuracy of gait recognition.

[0017] 4. Lack of efficient tracking capabilities of parts in video frames: Existing parsing methods cannot efficiently track various parts of the human body between video frames. Especially in the process of gait recognition, the motion characteristics of various parts of the human body change greatly over time, and traditional parsing methods cannot achieve accurate tracking, resulting in unstable parsing and recognition results.

[0018] It should be noted that the information disclosed in the above background technology section is only used for understanding the background of the present application, and therefore may include information that does not constitute prior art known to ordinary technicians in the field. Summary of the invention

[0019] The main purpose of the present invention is to overcome the defects existing in the above-mentioned background technology and provide a gait recognition method based on the temporal consistency of video human body analysis.

[0020] To achieve the above object, the present invention adopts the following technical solutions:

[0021] A gait recognition method based on temporal consistency of video human body analysis includes the following steps:

[0022] S1. parsing the first frame of the video sequence for human body parts to generate initial masks for various human body parts, wherein the initial mask includes a plurality of region identifiers corresponding to different human body parts;

[0023] S2. Using the initial human body part mask as prompt information, tracking and segmenting subsequent frame images of the video sequence to maintain temporal consistency with the initial frame parsing result, and generating a series of temporal masks corresponding to different human body parts;

[0024] S3. Input the temporal mask into a gait recognition model to extract gait features, wherein the gait features include temporal features (TP) representing temporal changes of the human body during gait and spatial features of human body parts (HP) representing morphological information of various parts of the human body; use the temporal features (TP) and spatial features of human body parts (HP) to perform identity recognition to determine the identity of the individual in the video.

[0025] Furthermore, in step S1, the first frame image in the video is analyzed using the M2FP model to generate initial masks of various parts of the human body, wherein the contours of the main parts of the human body in the video, namely the head, torso, and limbs, are extracted and corresponding area identifiers are generated.

[0026] Furthermore, in step S1, the first frame image in the video is analyzed using the M2FP model to generate initial masks of various parts of the human body, specifically including:

[0027] The M2FP model is used to identify and segment human body parts in the first frame of the video sequence.

[0028] Generate initial masks representing the main parts of the human body based on the recognition results;

[0029] The part contours and region identifiers in the initial mask are used as reference information for subsequent frame parsing.

[0030] Furthermore, in step S2, the parsing results are tracked and segmented in subsequent frames through the SAM2 model, which includes using the initial mask as prompt information to track and segment the human body parts for each frame in the video to ensure the parsing consistency between frames.

[0031] Furthermore, in step S2, the parsing results are tracked and segmented in subsequent frames by the SAM2 model, which specifically includes the following sub-steps:

[0032] Use the image encoder of the SAM2 model to encode the features of the input image;

[0033] Combining memory attention and mask decoder to generate masks consistent with the initial frame parsing;

[0034] The memory encoder and memory bank are used to store and manage the intermediate information in the parsing process to ensure the continuity and consistency of the parsing results.

[0035] Furthermore, step S2 also includes: performing human body analysis on the video sequence using a network architecture consisting of an encoder and a transformer decoder, the encoder is used to extract features of the input image sequence, and integrate features from different scales through multi-layer fusion operations, and then restore high-resolution feature representation through upsampling, and finally flattening the decoded feature map and adding position encoding for processing by the transformer decoder.

[0036] Furthermore, the specific operations of the encoder include: using the backbone network to extract features from the input RGB image sequence; integrating features of different scales through fusion operations; applying upsampling technology to improve the resolution of the feature map; flattening the feature map and combining it with position encoding to generate a feature representation containing spatial position information; the specific operations of the Transformer decoder include: using the attention mechanism to fuse and interact the input features with the query vectors representing different parsing targets; calculating the mask attention, in which the mask matrix is ​​used to cover irrelevant parts; after each query vector is processed by the attention mechanism, a corresponding mask and a category prediction are generated, and the mask and category prediction are integrated through matrix multiplication to form the final semantic segmentation result.

[0037] Furthermore, in step S3, the process of feature extraction and recognition specifically includes the following sub-steps:

[0038] The parsed human body part mask is input into the gait recognition model GaitBase;

[0039] The input data is processed through the data augmentation (DA) module to enrich the data representation;

[0040] The ResNet9 model is used to extract gait-related temporal features (TP) and human body part spatial features (HP);

[0041] The extracted features are further processed through fully connected layers (Separate FCs) to generate feature embedding for gait recognition;

[0042] Use the BNNeck module to standardize the features;

[0043] The feature embedding is optimized by the triplet loss function (L_triplet) and the cross entropy loss (L_ce) to improve the accuracy of gait recognition and the ability to distinguish different individuals.

[0044] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the gait recognition method based on temporal consistency of video human body analysis.

[0045] A computer program product comprises a computer program, wherein when the computer program is executed by a processor, the gait recognition method based on temporal consistency of video human body analysis is implemented.

[0046] The present invention has the following beneficial effects:

[0047] The present invention proposes a gait recognition method based on the temporal consistency of video human body analysis. By combining human body analysis and image tracking, the temporal consistency of human body part analysis in a video sequence can be achieved. The method of the present invention not only solves the inconsistency problem that may occur in the traditional analysis method when processing multi-frame videos, but also improves the accuracy of the analysis. In addition, the present invention enhances the feature extraction capability of gait recognition by performing fine-grained analysis on various parts of the human body, especially in capturing the dynamic characteristics of human body parts that change over time, and effectively improves the accuracy of gait recognition. By combining the consistency of multi-frame analysis with fine-grained analysis, the present invention effectively solves the problem of ignoring the dynamic changes of human body parts in the existing gait recognition technology, and significantly improves the performance of gait recognition. At the same time, the present invention also optimizes the synergy between human body analysis and gait recognition, so that the analyzed part features can directly provide more discriminative input for gait recognition, and realize more efficient collaborative work. The above advantages make the present invention have obvious advantages over the prior art in terms of analysis consistency and gait recognition accuracy.

[0048] Advantages of the embodiments of the present invention include:

[0049] 1. Achieve temporal consistency of human body analysis: By combining M2FP and Segment Anything2 (SAM2) models, the present invention can consistently analyze various parts of the human body in a video frame sequence, ensuring the continuity and stability of the analysis results between multiple frames. By utilizing the efficient segmentation and tracking capabilities of SAM2, the present invention can maintain the consistency of human body analysis results throughout the entire video sequence, avoiding the problem of reduced recognition accuracy due to inconsistency in multiple frames in the prior art.

[0050] 2. Improve the accuracy of human body part analysis: The present invention improves the feature extraction capability of gait recognition by performing fine-grained analysis of various human body parts during gait recognition, especially accurate segmentation of legs, arms and other parts. By analyzing the dynamic changes of various parts of the human body, the system can capture more key gait information, thereby improving the accuracy of gait recognition.

[0051] 3. Make full use of the dynamic change information of human body parts: The present invention captures the motion trajectory and dynamic changes of various parts of the human body through a time-series consistent human body analysis method. These part characteristics play a key role in gait recognition, especially the movement patterns of parts such as legs and arms. By accurately analyzing the time-series changes of these parts, the present invention can better utilize the dynamic characteristics of the human body during gait recognition and improve the robustness of the recognition results.

[0052] 4. Improve the tracking capability of human body parts in video frames: The present invention can accurately track various parts of the human body in multiple consecutive video frames through the efficient tracking mechanism of SAM2. With the help of SAM2's ability to segment and track different human body parts, the present invention can effectively avoid the discontinuity of parsing results between frames, ensuring the stability and accuracy of parsing and gait recognition results.

[0053] Other beneficial effects of the embodiments of the present invention will be further described below. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] Figure 1 The present invention is a flow chart of a human body parsing algorithm of an upstream video sequence of a gait recognition method based on temporal consistency of video human body parsing according to an embodiment of the present invention.

[0055] Figure 2 This is a flow chart of downstream feature extraction and gait recognition algorithm of a gait recognition method based on temporal consistency of video human body analysis in an embodiment of the present invention. DETAILED DESCRIPTION

[0056] The following is a detailed description of the embodiments of the present invention. It should be emphasized that the following description is only exemplary and is not intended to limit the scope and application of the present invention.

[0057] See also Figure 1 and Figure 2 The embodiment of the present invention provides a gait recognition method based on temporal consistency of video human body analysis, comprising the following steps:

[0058] Step S1. parse the first frame image of the video sequence for human body parts to generate initial masks of various human body parts, wherein the initial masks include a plurality of region identifiers corresponding to different human body parts.

[0059] In a preferred embodiment, in step S1, the first frame image in the video is analyzed by using the M2FP model to generate initial masks of various parts of the human body, wherein the contours of the main parts of the human body in the video, such as the head, torso, limbs, etc., are extracted and corresponding region identifiers are generated. Specifically, it may include: using the M2FP model to identify and segment the first frame image of the video sequence; generating an initial mask representing the main parts of the human body according to the identification result; using the part contours and region identifiers in the initial mask as reference information for subsequent frame analysis.

[0060] Step S2. Using the initial human body part mask as prompt information, subsequent frame images of the video sequence are tracked and segmented to maintain temporal consistency with the initial frame parsing result, and generate a series of temporal masks corresponding to different human body parts.

[0061] In a preferred embodiment, in step S2, the parsing results are tracked and segmented in subsequent frames by the SAM2 model, including using the initial mask as prompt information to track and segment the human body parts for each frame in the video to ensure the parsing consistency between frames. Specifically, it may include: using the image encoder of the SAM2 model to encode the features of the input image; combining the memory attention and the mask decoder to generate a mask consistent with the initial frame parsing; using the memory encoder and the memory bank to store and manage the intermediate information in the parsing process to ensure the continuity and consistency of the parsing results.

[0062] In a preferred embodiment, step S2 also includes: performing human body analysis on the video sequence using a network architecture consisting of an encoder and a transformer decoder, the encoder is used to extract features of the input image sequence, and integrate features from different scales through multi-layer fusion operations, and then restore high-resolution feature representation through upsampling, and finally flattening the decoded feature map and adding position coding for processing by the transformer decoder.

[0063] In a further preferred embodiment, the specific operations of the Encoder include: using the backbone network to extract features from the input RGB image sequence; integrating features of different scales through fusion operations; applying upsampling technology to improve the resolution of the feature map; flattening the feature map and combining it with position encoding to generate a feature representation containing spatial position information; the specific operations of the Transformer Decoder include: using the attention mechanism to fuse and interact the input features with query vectors representing different parsing targets; calculating mask attention, in which the mask matrix is ​​used to cover irrelevant parts; after each query vector is processed by the attention mechanism, a corresponding mask and a category prediction are generated, and the mask and category prediction are integrated through matrix multiplication to form the final semantic segmentation result.

[0064] Step S3. Input the temporal mask into a gait recognition model to extract gait features, wherein the gait features include temporal features (TP) representing temporal changes of the human body during gait and spatial features of human body parts (HP) representing morphological information of various parts of the human body; use the temporal features (TP) and spatial features of human body parts (HP) to perform identity recognition to determine the identity of the individual in the video.

[0065] In a preferred embodiment, in step S3, the process of feature extraction and recognition specifically includes: inputting the parsed human body part mask into the gait recognition model GaitBase; processing the input data through the data enhancement (DA) module to enrich the data representation; extracting the temporal features (TP) and human body part spatial features (HP) related to gait using the ResNet9 model; further processing the extracted features through the fully connected layer (Separate FCs) to generate feature embedding for gait recognition; standardizing the features using the BNNeck module; optimizing the feature embedding through the triplet loss function (L_triplet) and the cross entropy loss (L_ce) to improve the accuracy of gait recognition and the ability to distinguish different individuals.

[0066] The present invention proposes an innovative gait recognition method, which significantly improves the accuracy and stability of gait recognition by achieving temporal consistency of human body part analysis in a video sequence. The method first performs human body part analysis on the first frame of the video to generate an initial mask, and then uses these masks as prompt information to track and segment subsequent frames to ensure that the analysis results of human body parts in the entire video sequence are continuous and stable. The present invention overcomes the inconsistency problem that may occur in traditional methods when processing multiple frames of video, and improves the analysis accuracy. By performing fine-grained analysis on various parts of the human body, the present invention enhances the feature extraction capability of gait recognition, especially in capturing the dynamic features of human body parts that change over time, and effectively improves the accuracy of gait recognition. In addition, the present invention also optimizes the synergy between human body analysis and gait recognition, so that the analyzed part features can directly provide more discriminative input for gait recognition, and achieve more efficient collaborative work. These innovations make the present invention have obvious advantages over the prior art in terms of analysis consistency and gait recognition accuracy.

[0067] The following further describes specific embodiments of the present invention and examples of algorithm implementation thereof.

[0068] In a specific embodiment, a video sequence human body parsing method combining M2FP and SegmentAnything2 (SAM2) model is used to solve the problems of temporal consistency and human body part parsing accuracy in gait recognition. First, M2FP is used to parse the human body in the first frame of the video, and then SAM2 is used to efficiently track and segment the parsing results in subsequent frames, and finally the gait features are extracted and the identity is recognized through the gait recognition model GaitBase. Each key step of the scheme will be described in detail below with reference to the accompanying drawings.

[0069] Upstream: Human body parsing in video sequences

[0070] First, for a gait video sequence input, the present invention uses the M2FP model to perform human body analysis on the first frame image in the video and generate initial masks of various parts of the human body. These masks represent the contours of various major parts of the human body (such as the head, torso, limbs, etc.) and provide analysis hints for subsequent frames.

[0071] Figure 1 In the figure, several frames of the video sequence are shown on the left, and the corresponding part segmentation mask is generated by the initial frame parsed by M2FP (as shown in the figure, different colors correspond to different human body parts).

[0072] The parsed mask is used to guide the subsequent operations of the SegmentAnything2 (SAM2) model. The SAM2 model uses the initial mask as a prompt and uses its powerful image tracking and segmentation capabilities to parse and track the same parts in all subsequent frames. The image encoder of SAM2 encodes the input image and combines memory attention and mask decoder to generate a mask that is consistent with the initial frame parsing, ensuring consistency between frames.

[0073] In addition, the memory encoder and memory bank are used to store and manage the intermediate information in the parsing process to ensure the continuity and consistency of the parsing results. By parsing the human body of multiple frames of video images, the system generates parsing diagrams corresponding to various parts of the human body, providing more accurate input data for gait recognition. Specifically, the network adopts the network architecture of Encoder and TransformerDecoder.

[0074] Encoder is responsible for extracting features of the input image sequence. For the input image sequence, it first passes through the backbone network (usually ResNet-50) and fuses features from different scales:

[0075] F fused = fuse({Backbone i (I)})

[0076] Where I∈R B×H×W×3 Represents the input RGB image sequence, Backbone i is the operation of the i-th layer backbone network, and fuse is the fusion operation of multiple layers. The extracted feature map is upsampled to restore the higher resolution feature representation:

[0077] F decoded =BilinearUpsample(F fused )

[0078] Before entering TransformerDecoder, the decoded feature map needs to be flattened and positional encoding added:

[0079] F flattened =Flatten(F decoded )

[0080]

[0081] F pos =F flattened +P pos

[0082] ”'

[0083] Among them, F flattened ∈R B×(H×W)×C , x and y are spatial positions, i is the channel index, C is the feature dimension, and F pos It is the feature after adding position encoding.

[0084] TransformerDecoder uses a series of attention mechanisms to fuse and interact the input features with the query vector, which represents different parsing targets (including background queries, part queries, and human queries). M×C Compute mask attention:

[0085]

[0086] Among them, K and V are F pos Features, d k is the dimension of the feature vector, and M is the mask matrix used to cover irrelevant parts. Each query vector will eventually generate a mask and a category prediction, and the final semantic segmentation result will be generated through matrix multiplication.

[0087] like Figure 1 As shown, the model consists of the following parts:

[0088] 1.M2FPParsing

[0089] The image sequence is first input into the M2FP model. M2FP is a pre-trained human body parsing model that can parse the first frame of the video sequence for human body parts and generate initial human body parsing masks (different colors in the figure represent different human body parts). These masks contain information about key parts of the human body and will be used to guide the parsing process of subsequent frames.

[0090] 2. Image Encoder

[0091] The initial human parsing mask generated from M2FP is input into the image encoder. The image encoder extracts features from the input initial frame image and encodes these features into a deep feature map. The encoding result will be used as the basic feature information for other frames in the subsequent parsing process.

[0092] 3.MemoryAttention

[0093] This module uses the attention mechanism to combine the previous image encoding information and the memory information of the intermediate steps to determine the parsing focus of each frame in the video sequence. The introduction of the attention mechanism ensures that the model can effectively track and parse various parts of the human body between video frames to ensure temporal consistency.

[0094] 4.Mask Decoder

[0095] The mask decoder is responsible for decoding the attention information from the memory attention module into a specific human body part parsing mask for each frame in the video. These parsing masks are generated based on the initial prompt to ensure consistency with the parsing results of the first frame.

[0096] 5. Prompt Encoder

[0097] This module is a key part of SAM2, which uses the initial mask generated from M2FP as a hint to guide SAM2 to parse and track human body parts in subsequent frames of the video. By hinting the encoder, subsequent frames can be segmented consistently based on the part information parsed in the initial frame.

[0098] 6.Memory Encoder and MemoryBank

[0099] The parsing results of each human body part in the parsing process are stored in the memorybank. The memory encoder is responsible for encoding the parsing results to maintain the consistency and transmission of information in the parsing of subsequent frames. This process ensures that the model does not lose important information between frames when processing long video sequences, while improving the stability and consistency of parsing.

[0100] 7. GaitBase gait recognition

[0101] After parsing, the parsed mask of human body parts is input into the GaitBase model. GaitBase is a gait recognition model that uses the human body part information obtained from video parsing to extract gait features and ultimately use them for identity recognition. By combining the human body dynamic information in the parsing results, GaitBase can accurately identify the gait features of individuals.

[0102] Downstream: gait recognition feature extraction

[0103] After the video is parsed, the human body segmentation results of each frame are input into the gait recognition model GaitBase for feature extraction and recognition. Figure 2The processing flow of gait recognition is shown.

[0104] The images after gait analysis first pass through the DA (data enhancement) module to further enrich the data representation. Then, the data is input into the ResNet9 model, which is used as a feature extraction network to extract two key features related to gait:

[0105] TP (temporal pooling): Temporal features, which represent the characteristics of human body changing over time during gait;

[0106] HP (horizontal pooling): spatial features of human body parts, representing the morphological information of various parts of the human body.

[0107] Next, these features are further processed through Separate FCs (fully connected layers) to generate feature embeddings for gait recognition. Then through the BNNeck module, the features are normalized and optimized using two loss functions:

[0108] L_triplet: triplet loss function, used to increase the feature distance between different individuals;

[0109] L_ce: Cross entropy loss, used to optimize classification accuracy.

[0110] like Figure 2 As shown, the model consists of the following parts:

[0111] 1. Gait analysis result input

[0112] After the video sequence is parsed by the M2FP and SAM2 models, a human body segmentation map is obtained, as shown in the color human body image in the figure. This parsing result contains the segmentation information of various parts of the human body (such as head, arms, legs, etc.) and maintains the temporal consistency of the video frame sequence. The parsing result is used as input to the downstream gait recognition process.

[0113] 2. DA (Data Augmentation)

[0114] The Data Augmentation (DA) module is used to perform various enhancement operations on the input parsing results. These enhancement operations may include image flipping, rotation, scaling, noise addition, etc., in order to increase the diversity of data and improve the generalization ability of the model. The image after data enhancement is input to the next feature extraction module.

[0115] 3.ResNet9 feature extraction

[0116] ResNet9 is a classic convolutional neural network (CNN) architecture used to extract gait features from parsed images. The model processes the input image through multiple convolutional layers to extract global and local features related to gait. Two features are marked in the figure:

[0117] TP (temporal pooling): Temporal features, which represent the dynamic characteristics of human body parts changing over time during gait.

[0118] HP (horizontal pooling): spatial features of human body parts, representing the static and dynamic morphological information of different parts.

[0119] 4.Separate FCs (fully connected layers)

[0120] The extracted features then enter the Separate FCs (fully connected layers), where the fully connected layers further process and combine the features from ResNet9 to generate high-dimensional feature embeddings for gait recognition. These embedded features contain the temporal information of the gait and the shape characteristics of the human body parts, which are the key features for recognition.

[0121] 5.BNNeck(Batch NormalizationNeck)

[0122] BNNeck is a common feature normalization layer. Its main function is to normalize the features that have passed through the fully connected layer so that the feature values ​​of different data are distributed in a unified range, which is convenient for subsequent feature learning and optimization. The features that have passed through BNNeck will be used to calculate two loss functions:

[0123] L_triplet (triplet loss): This loss function is used to increase the distance between gait features of different individuals and reduce the distance between gait features of the same individual, ensuring that the model can accurately distinguish the gaits of different individuals.

[0124] L_ce (cross entropy loss): This loss function is used for classification tasks to help the model better distinguish different individual identities.

[0125] 6.Inference

[0126] Finally, the optimized feature embedding will be used in the inference stage of gait recognition. In the inference stage, the person in the input video is identified based on the gait features learned by the model, and the identification result is output. By introducing TP and HP features, the present invention can more accurately capture the timing information of gait and the dynamic changes of human body parts, significantly improving the accuracy of gait recognition.

[0127] Alternative Embodiments

[0128] 1. Alternative parsing and tracking models

[0129] Use other human parsing models instead of M2FP: In addition to M2FP, you can also consider using other excellent pre-trained human parsing models, such as HRNet or Deeplabv3+. These models can also perform high-precision segmentation of human parts and can be used to generate initial masks, and then parse and track subsequent frames through SAM2 or other similar target tracking models (such as TrackAnything (TAT)). This alternative method can still maintain consistency between video frames and achieve high parsing accuracy.

[0130] Use other tracking models instead of SAM2: In addition to SAM2, you can also use models designed specifically for multi-target tracking, such as DeepSORT or FairMOT. These models are good at handling continuous tracking of multiple targets and can replace SAM2 to track and analyze various parts of the human body in video frames. Although these models are not completely targeted at human body analysis, they can be adapted to similar tasks through adjustment and training.

[0131] 2. Alternative gait recognition models

[0132] Use other gait recognition models instead of GaitBase: For gait recognition, you can consider using other excellent deep learning models, such as GaitSet, GaitPart, or more recent 3D convolutional neural network (3D CNN) models. These models can replace GaitBase to achieve feature extraction and gait recognition of parsing results. In particular, models such as 3D CNN can directly process video sequences and use their convolution operations in the time dimension to extract richer temporal features.

[0133] 3. Different data processing and optimization strategies

[0134] Utilizing a time series model: A time series model (such as LSTM, GRU or Transformer) can be introduced to enhance the modeling capability of time series information in the gait recognition process. Unlike the direct feature extraction method used in the present invention, the time series model can better capture the changes of human body parts over time during gait by memorizing and analyzing sequence data, further improving the recognition accuracy.

[0135] End-to-end deep learning method: In addition to the step-by-step processing (parsing, tracking, and recognition) method, you can also consider building an end-to-end deep learning model and directly input the video sequence. The model automatically optimizes the synergy between human body parsing and recognition by jointly learning the two tasks of parsing and gait recognition. This method can simplify the entire process and may improve the efficiency and recognition accuracy of the model.

[0136] 4. Alternative memory management and timing optimization methods

[0137] Using video enhancement technology: Optical flow technology or temporal consistency optimization can be introduced to replace the temporal consistency management implemented by memory encoder and memory bank. Optical flow technology can effectively track the movement trajectory of objects in the video, and combined with human body analysis, it can achieve similar effects.

[0138] The technical innovations of the specific embodiments of the present invention include:

[0139] 1. Temporal consistency of human body analysis based on video sequences

[0140] The technical solution combining M2FP and SegmentAnything2 (SAM2) model is used to analyze the human body for each frame in the video sequence, and the image tracking capability of SAM2 is used to ensure the temporal consistency of the analysis results of various parts of the human body in the entire video sequence. This is one of the key points to solve the inconsistency problem of traditional analysis methods when processing multiple frames of video.

[0141] 2. The initial parsing prompt generated by M2FP is used to guide the SAM2 model to track and segment in subsequent frames

[0142] M2FP is used to generate a human body part parsing mask for the first frame as a prompt for the SAM2 model, guiding SAM2 to segment and track human body parts in subsequent frames of the video, thereby achieving accurate human body part parsing and time-series tracking. This method greatly improves the accuracy and consistency of parsing.

[0143] 3. Detailed analysis of human body parts and capture of dynamic changes

[0144] The present invention can achieve fine analysis of various parts of the human body (such as the head, arms, legs, etc.), and can capture the dynamic characteristics of these parts changing over time, especially improving the accuracy of gait recognition by tracking the motion trajectories of human body parts in gait videos. This capture of dynamic changes is a very important innovation in gait recognition tasks.

[0145] 4. Combined application of multi-frame consistency and fine analysis

[0146] The present invention combines the consistency of multi-frame analysis with fine-grained analysis, effectively solving the problems of ignoring dynamic changes of human body parts and multi-frame inconsistency in existing gait recognition technology, thereby significantly improving the performance of gait recognition.

[0147] 5. Collaborative optimization of human body analysis and gait recognition

[0148] The technical solution of the present invention not only performs human body analysis on each frame in the video, but also optimizes the synergy between the analysis results and downstream gait recognition. The analyzed part features directly provide more discriminative input for gait recognition, making the whole system achieve more efficient synergy between analysis and recognition.

[0149] Compared with the prior art, the embodiments of the present invention have significant advantages in the following aspects:

[0150] First, by combining the M2FP and SegmentAnything2 (SAM2) models, the temporal consistency of human body part analysis in video sequences is achieved. Compared with the traditional single-frame analysis method, the present invention can maintain the consistency of analysis of human body parts in multiple frames of video, effectively solving the problem of jitter and instability of analysis results between multiple frames in the prior art. This innovation significantly improves the accuracy of analysis, especially in applications that require high temporal information, such as gait recognition.

[0151] Secondly, the efficient tracking capability of SAM2 and the gait recognition performance of GaitBase are utilized to achieve an effective combination of refined human body part analysis and gait feature extraction. By introducing M2FP to generate initial prompt information and using SAM2 to track and segment human body parts, the present invention can capture the dynamic changes of the human body in the video, further improve the accuracy of gait recognition, and overcome the shortcomings of insufficient fine-grained feature analysis of human body parts in traditional gait recognition methods.

[0152] In summary, the present invention has obvious advantages over the prior art in terms of analytical consistency and gait recognition accuracy.

[0153] An embodiment of the present invention further provides a storage medium for storing a computer program, which at least performs the above method when executed.

[0154] An embodiment of the present invention further provides a control device, comprising a processor and a storage medium for storing a computer program; wherein the processor is configured to execute at least the method described above when executing the computer program.

[0155] An embodiment of the present invention further provides a processor, wherein the processor executes a computer program and at least executes the method described above.

[0156] The storage medium can be implemented by any type of non-volatile storage device, or a combination thereof. Among them, the non-volatile memory can be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a magnetic random access memory (FRAM), a ferromagnetic random access memory (FRAM), a flash memory, a magnetic surface memory, an optical disc, or a compact disc read-only memory (CD-ROM); the magnetic surface memory can be a disk memory or a tape memory. The storage medium described in the embodiments of the present invention is intended to include, but is not limited to, these and any other suitable types of memory.

[0157] In the several embodiments provided by the present invention, it should be understood that the disclosed systems and methods can be implemented in other ways. The device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as: multiple units or components can be combined, or can be integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the components shown or discussed can be through some interfaces, and the indirect coupling or communication connection of the devices or units can be electrical, mechanical or other forms.

[0158] The units described above as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units; some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0159] In addition, all functional units in the embodiments of the present invention may be integrated into one processing unit, or each unit may be separately used as a unit, or two or more units may be integrated into one unit; the above-mentioned integrated units may be implemented in the form of hardware or in the form of hardware plus software functional units.

[0160] Those skilled in the art can understand that: all or part of the steps of implementing the above method embodiments can be completed by hardware related to program instructions, and the aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it executes the steps of the above method embodiments; and the aforementioned storage medium includes: mobile storage devices, read-only memories (ROM, Read-Only Memory), random access memories (RAM, Random Access Memory), disks or optical disks, etc. Various media that can store program codes.

[0161] Alternatively, if the above-mentioned integrated unit of the present invention is implemented in the form of a software function module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiment of the present invention can be essentially or partly reflected in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the methods described in each embodiment of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as mobile storage devices, ROM, RAM, magnetic disks or optical disks.

[0162] The methods disclosed in the several method embodiments provided by the present invention can be arbitrarily combined without conflict to obtain new method embodiments.

[0163] The features disclosed in several product embodiments provided by the present invention can be arbitrarily combined without conflict to obtain new product embodiments.

[0164] The features disclosed in several method or device embodiments provided by the present invention can be arbitrarily combined without conflict to obtain new method embodiments or device embodiments.

[0165] The above contents are further detailed descriptions of the present invention in combination with specific preferred embodiments, and it cannot be determined that the specific implementation of the present invention is limited to these descriptions. For those skilled in the art of the present invention, several equivalent substitutions or obvious variations can be made without departing from the concept of the present invention, and the performance or use is the same, which should be regarded as belonging to the protection scope of the present invention.

Claims

1. A gait recognition method based on temporal consistency of video human body analysis, characterized in that: The following steps are involved: S1. parsing the first frame of the video sequence for human body parts to generate initial masks for various human body parts, wherein the initial mask includes a plurality of region identifiers corresponding to different human body parts; S2. Using the initial human body part mask as prompt information, tracking and segmenting subsequent frame images of the video sequence to maintain temporal consistency with the initial frame parsing result, and generating a series of temporal masks corresponding to different human body parts; S3. Input the temporal mask into a gait recognition model to extract gait features, wherein the gait features include temporal features (TP) representing temporal changes of the human body during gait and spatial features of human body parts (HP) representing morphological information of various parts of the human body; use the temporal features (TP) and spatial features of human body parts (HP) to perform identity recognition to determine the identity of the individual in the video.

2. The gait recognition method based on temporal consistency of video human body analysis as claimed in claim 1, characterized in that: In step S1, the first frame image in the video is analyzed using the M2FP model to generate initial masks of various parts of the human body, wherein the contours of the main parts of the human body in the video, namely the head, torso, and limbs, are extracted and corresponding area identifiers are generated.

3. The gait recognition method based on temporal consistency of video human body analysis as claimed in claim 2, characterized in that: In step S1, the first frame image in the video is analyzed using the M2FP model to generate initial masks of various parts of the human body, specifically including: The M2FP model is used to identify and segment human body parts in the first frame of the video sequence. Generate initial masks representing the main parts of the human body based on the recognition results; The part contours and region identifiers in the initial mask are used as reference information for subsequent frame parsing.

4. The gait recognition method based on temporal consistency of video human body analysis as claimed in claim 1, characterized in that: In step S2, the parsing results are tracked and segmented in subsequent frames through the SAM2 model, which includes using the initial mask as prompt information to track and segment the human body parts for each frame in the video to ensure the parsing consistency between frames.

5. The gait recognition method based on temporal consistency of video human body analysis as claimed in claim 4, characterized in that: In step S2, the parsing results are tracked and segmented in subsequent frames using the SAM2 model, which specifically includes the following sub-steps: Use the image encoder of the SAM2 model to encode the features of the input image; Combining memory attention and mask decoder to generate masks consistent with the initial frame parsing; The memory encoder and memory bank are used to store and manage the intermediate information in the parsing process to ensure the continuity and consistency of the parsing results.

6. The gait recognition method based on temporal consistency of video human body analysis according to any one of claims 1 to 5, characterized in that: Step S2 also includes: using a network architecture consisting of an encoder and a transformer decoder to perform human body analysis on the video sequence, the encoder is used to extract features of the input image sequence, and integrate features from different scales through multi-layer fusion operations, and then restore high-resolution feature representation through upsampling, and finally flatten the decoded feature map and add position encoding for processing by the transformer decoder.

7. The gait recognition method based on temporal consistency of video human body analysis as claimed in claim 6, characterized in that: The specific operations of the encoder include: using the backbone network to extract features from the input RGB image sequence; integrating features of different scales through fusion operations; applying upsampling technology to improve the resolution of feature maps; flattening the feature maps and combining them with position encoding to generate feature representations containing spatial position information; The specific operations of the Transformer decoder include: using the attention mechanism to fuse and interact the input features with the query vectors representing different parsing targets; calculating the mask attention, where the mask matrix is ​​used to cover irrelevant parts; after each query vector is processed by the attention mechanism, a corresponding mask and a category prediction are generated, and the mask and category prediction are integrated through matrix multiplication to form the final semantic segmentation result.

8. The gait recognition method based on temporal consistency of video human body analysis according to any one of claims 1 to 7, characterized in that: In step S3, the process of feature extraction and recognition specifically includes the following sub-steps: The parsed human body part mask is input into the gait recognition model GaitBase; The input data is processed through the data augmentation (DA) module to enrich the data representation; The ResNet9 model is used to extract gait-related temporal features (TP) and human body part spatial features (HP); The extracted features are further processed through fully connected layers (Separate FCs) to generate feature embedding for gait recognition; Use the BNNeck module to standardize the features; The feature embedding is optimized by the triplet loss function (L_triplet) and the cross entropy loss (L_ce) to improve the accuracy of gait recognition and the ability to distinguish different individuals.

9. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the gait recognition method based on temporal consistency of video human body analysis is implemented as described in any one of claims 1 to 8.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the gait recognition method based on temporal consistency of video human body analysis is implemented as described in any one of claims 1 to 8.

Images