Real scene-oriented gait-text multi-modal gait recognition method

Abstract

The invention discloses a real scene-oriented gait-text multi-modal gait recognition method, which comprises the following steps of: generating corresponding gait characteristic natural language modal description for each gait sequence, and constructing a real scene-oriented large-scale multi-modal gait recognition data set; a real scene-oriented multi-modal gait recognition model is constructed and trained, and multi-modal alignment and fusion of gait features and natural languages are realized through the multi-modal gait recognition model; and performing target person gait recognition on a to-be-recognized gait sequence by using the trained multi-modal gait recognition model. According to the invention, through the multi-modal feature fusion module, text features are expanded to spatial dimensions and are aligned with image features, so that the method has a good gait recognition effect in a real complex environment, and meanwhile, a spatial attention mechanism and a self-adaptive residual connection module are used to focus and filter key information of input features, so that the accuracy of gait recognition is improved. The utilization rate of natural language modal information is improved, and higher recognition precision is achieved.

Description

Technical Field

[0001] This invention relates to the field of gait recognition technology, and is applied to gait-text multimodal gait recognition assisted by natural language modality information. Specifically, it relates to a gait-text multimodal gait recognition method for real-world scenarios. Background Technology

[0002] Gait recognition has been widely applied in various fields. In real-world scenarios, existing representation learning methods typically focus on gait contours or skeletons, but these studies are easily affected by complex environments or factors such as clothing changes. Furthermore, gait recognition under a single modality has certain limitations, and common skeletons are difficult to escape the similarity between modalities, making recognition performance and adaptability easily constrained by the single modality. Although gait recognition technology has achieved considerable recognition results under a single modality, in real-world scenarios, due to complex and variable environmental factors such as lighting changes, clothing changes, perspective differences, and environmental occlusion, single-modality gait recognition research still faces many challenges that urgently need to be addressed. For example, there is still considerable room for improvement in recognition performance and adaptability, and the existing gait representation data is insufficient.

[0003] Previous single-modal gait recognition methods focused primarily on laboratory settings and have achieved significant success in the industry. However, in recent years, more and more scholars have begun to focus on multimodal gait recognition in real-world scenarios, and this area has now become the core of research in the field of gait recognition. However, in current practical applications of gait recognition, there is a large amount of information in textual and verbal forms. Existing gait recognition technologies cannot effectively utilize this type of information to assist in gait recognition tasks. Most gait recognition research uses computer vision techniques to perform gait silhouette features, ignoring the abundant natural language modal information present in the real world. Summary of the Invention

[0004] To address the shortcomings of existing technologies, this invention provides a gait-text multimodal gait recognition method for real-world scenarios. This invention establishes a large-scale benchmark for gait recognition tasks, using natural language for gait description and retrieval. Through an original, large-scale natural language modality dataset and an adaptive gait-text spatiotemporal fusion module, gait features in both text and image modalities become increasingly distinct, thereby improving the performance of gait-text multimodal gait recognition and solving the problems mentioned in the background.

[0005] To address the aforementioned technical problems, this invention provides the following technical solution: a gait-text multimodal gait recognition method for real-world scenarios, comprising the following operational steps:

[0006] Step 1: Data collection and generation are carried out using a large language model. The Gait3D gait dataset and the CCPG gait dataset are used as gait sequence data. A corresponding gait-featured natural language modal description is generated for each gait sequence to build a multimodal gait recognition dataset for real-world scenarios.

[0007] Preferably, an initial gait text description is generated using a large language model, and each generated initial gait text description is checked and data augmentation is performed to obtain the final gait text description of the natural language modality.

[0008] Preferably, generating an initial gait text description includes:

[0009] Step a1: Add simulated character commands and observation angle restrictions to the system prompts, adjust the prompts, and use the large language model after the prompts are adjusted to simulate two different observers;

[0010] Step a2: Analyze and describe the gait of the same observer from multiple angles, and remove relevant irrelevant noise interference.

[0011] The multiple perspectives include: descriptions from the perspectives of the observer's hand movements, leg movements, special gait, walking speed, and stride frequency;

[0012] Step a3: Based on the gait sequences in the dataset, perform data cleaning on the observer's gait text description, filter out noise in the text, and check and clean characters, clauses, content and semantics to eliminate the illusion of a large language model;

[0013] Step a4: Perform data augmentation using back-translation to obtain the final gait text description as the dataset.

[0014] Step 2: Build and train a multimodal gait recognition model for real-world scenarios, and perform multimodal alignment and fusion of gait features and natural language through the multimodal gait recognition model.

[0015] Preferably, the operation process of the multimodal gait recognition model includes:

[0016] Step b1: Encode the gait text description into text features using a text encoder, and simultaneously use digital image processing technology to convert the gait sequence into a gait silhouette sequence, and extract gait silhouette features through a silhouette learning network;

[0017] Step b2: The text features and gait silhouette features are concatenated through the adaptive gait-text spatiotemporal fusion module, and fine-grained feature fusion is performed through the spatiotemporal attention module and the adaptive gated residual connection module.

[0018] Step b3: Perform feature aggregation based on temporal pooling and horizontal pyramid mapping to obtain embedding vectors and classification logits. Sort the embedding vectors of the query samples and the candidate samples according to the Euclidean distance and take the Top-1 as the recognition result.

[0019] Step b4: Calculate the cross-entropy loss and triplet loss, and use the final loss obtained by weighting the two losses in a 1:1 ratio for training.

[0020] Step 3: Use the trained multimodal gait recognition model to output the gait recognition of the target person from the gait sequence to be recognized.

[0021] Preferably, in step 3, an initial gait text description of the gait sequence to be identified is generated by using a large language model. The initial gait text description is then cleaned and augmented. The cleaned and augmented initial gait text description is used as the input of the trained multimodal gait recognition model to identify the gait sequence to be identified and the target person in the gait sequences of the candidate samples.

[0022] Compared with existing technologies, the beneficial effects achieved by this invention are as follows: This invention establishes a large-scale benchmark for gait description and retrieval using natural language. Through an original large-scale natural language modality dataset and an adaptive gait-text spatiotemporal fusion module, gait features in both text and image modalities become increasingly distinct, thereby improving the performance of gait-text multimodal gait recognition and enabling it to achieve better gait recognition results in complex real-world environments. The proposed adaptive gait-text spatial fusion module uses a spatiotemporal attention module and an adaptive gated residual connection module to focus on and filter key information from the input features. Compared with existing models, this improves the utilization rate of natural language modality information, broadens the practical scenarios for gait recognition, and achieves higher recognition accuracy. Attached Figure Description

[0023] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used in conjunction with embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings:

[0024] Figure 1 This is a diagram illustrating the multimodal gait recognition dataset provided in an embodiment of the present invention.

[0025] Figure 2 This is a bar chart showing the length statistics of the gait text description dataset provided in this embodiment of the invention.

[0026] Figure 3 This is a high-frequency word cloud diagram of gait text description provided in an embodiment of the present invention.

[0027] Figure 4This is a diagram of the architecture of the multimodal gait recognition model provided in an embodiment of the present invention.

[0028] Figure 5 This is a flowchart of the multimodal gait recognition process provided in an embodiment of the present invention. Detailed Implementation

[0029] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0030] Example 1

[0031] Combination Figures 1-5 As shown, to learn effective features from gait sequences and text data, this embodiment of the invention employs the state-of-the-art large language model GPT-4o to assist in data collection and generation. First, the Gait3D gait dataset and the CCPG gait dataset are used as gait sequence data. The GPT-4o large language model is then used to assist in generating textual descriptions of gait features in a natural language modality based on the image data. Through manual refinement and modification, the first large-scale natural language modality dataset in the field of gait recognition is proposed. Next, through a designed multimodal feature fusion module, text features are extended to the spatial dimension and aligned with image features, resulting in better gait recognition performance in complex real-world environments.

[0032] This invention provides a technical solution: a gait text multimodal gait recognition method for real-world scenarios, comprising the following steps:

[0033] Step 1: Generate corresponding gait-featured natural language modal descriptions for each gait sequence, and construct a large-scale multimodal gait recognition dataset for real-world scenarios;

[0034] In this embodiment, the present invention uses the large language model GPT-4o to assist in data collection and generation, and uses the Gait3D gait dataset and CCPG gait dataset as gait sequence data, collecting a total of 4200 individuals and 41674 gait sequences as the objects of description, ensuring the authenticity and diversity of the data. For each gait sequence, two corresponding gait-featured natural language modal descriptions are generated and checked, and the data are organized into a large-scale multimodal gait recognition dataset for real-world scenarios.

[0035] For example, the gait text description in this invention focuses on the perspective of a person's gait. It generates gait text descriptions in natural language modality from the perspective of a real eyewitness, focusing on the observer's specific gait and body movements. The specific operation process is as follows: An initial gait text description is generated using a large language model. Each generated initial gait text description is checked and polished to achieve data augmentation, resulting in a final gait text description of 83,348 sentences in English form, ensuring the authenticity and accuracy of the description. The multimodal gait recognition dataset is shown in the figure below. Figure 1 As shown in the figure. Based on the different data sources of the gait sequences, this invention divides the gait text description dataset into two parts: Gait3D-Caption and CCPG-Caption. The bar chart showing the statistical length of the gait text descriptions for each part is shown below. Figure 2 As shown, the high-frequency word cloud diagram for all gait text descriptions is as follows: Figure 3 As shown.

[0036] For example, the specific operations for generating gait text descriptions in this invention are as follows:

[0037] Step a1: Based on the observer gait image dataset, in order to accurately describe the gait of a single observer from multiple perspectives, simulated role instructions and observation angle restrictions are added to the system prompts, the prompts are adjusted, and the large language model with the adjusted prompts is used to simulate two different observers.

[0038] Step a2: Analyze and describe the gait of the same observer from multiple angles, and remove relevant useless noise interference. The specific description includes: a detailed description from multiple angles such as the observer's hand movements, leg movements, special gait, walking speed, and stride frequency, to ensure the diversity of the text data.

[0039] Step a3: To ensure the accuracy of gait text description, based on the gait sequences in the dataset, perform data cleaning on the observer gait text description, filter out noise in the text, and check and clean characters, clauses, content and semantics to eliminate the illusion of a large language model;

[0040] Step a4: To highlight the target information in the text, the English text is translated into Chinese and then back into English. Data augmentation is performed using back-translation to increase the amount of information in the gait text description, allowing the model to better learn gait features in real-world scenarios. The resulting gait text description serves as the dataset. By supplementing the data with multimodal information from the observer's perspective, the original single-modal gait recognition is assisted, enabling multimodal gait recognition for real-world complex environments and improving gait recognition performance.

[0041] Step 2: Construct a multimodal gait recognition model based on real-world scenarios, train it using the obtained dataset, and achieve multimodal alignment and fusion of gait features and natural language through the multimodal gait recognition model. Explore the feasibility of gait-text multimodal recognition in gait recognition tasks and improve gait recognition performance in real-world scenarios.

[0042] In this embodiment, to fully explore the multimodal correlation between gait images and natural language descriptions, this invention designs a multimodal feature fusion method, constructs and trains a robust multimodal gait recognition model, and expands the design by incorporating the natural language modality to integrate more effective multimodal feature fusion, achieving accurate matching of gait data under different modalities, while also possessing good noise tolerance. This invention constructs as follows... Figure 4 The multimodal gait recognition model architecture shown includes the following operational processes:

[0043] Step b1: Encode the gait text description into text features using a text encoder, and simultaneously use digital image processing techniques to convert the gait sequence into a gait silhouette sequence, and extract gait silhouette features using a silhouette learning network;

[0044] Step b2: The text features and gait silhouette features are concatenated through the adaptive gait-text spatiotemporal fusion module, and fine-grained feature fusion is achieved through the spatiotemporal attention module and the adaptive gated residual connection module.

[0045] For example, the implementation process of the adaptive gait-text spatiotemporal fusion module includes:

[0046] Step b21: Align the obtained text features with the gait silhouette features in the channel dimension using a neural network, and then extend the text features to the spatial dimension. Achieve fine-grained alignment between natural language description and gait silhouette in the spatiotemporal dimension using a neural network.

[0047] Step b22: After feature alignment, concatenate the text features and silhouette features;

[0048] Step b23: Utilize the spatiotemporal attention module to generate attention weights through a multi-layered grid structure, which are then applied to the spliced ​​features. This allows for more granular capture of local key information of the spliced ​​features in the spatial dimension, enabling the spliced ​​features to achieve multi-level alignment of intermodal differences in space, thus facilitating similarity measurement.

[0049] Step b24: Use the adaptive gated residual connection module to further filter and fuse the spliced ​​features focused by the spatiotemporal attention module;

[0050] For example, in the adaptive gated residual connection module, the spliced ​​features are generated into adaptive gate weights through the gate unit. The obtained gate weights are used to weight the spliced ​​features and gait silhouette features focused by the spliced ​​features and gait silhouette features respectively, and adaptive gated residual connection is performed to obtain more effective gait features. Irrelevant information and feature noise interference in the features are filtered out, making the model more robust and able to more robustly integrate two complementary information with large differences in real scene.

[0051] Step b3: Perform feature aggregation based on temporal pooling and horizontal pyramid mapping to obtain embedding vectors and classification logits. Sort the embedding vectors of the query samples and the candidate samples according to the Euclidean distance and take the top-1 as the recognition result.

[0052] For example, effective gait features are obtained through the adaptive gait-text spatiotemporal fusion module. Temporal pooling and horizontal pyramid mapping are used to efficiently aggregate the gait features to obtain the embedding vector and classification logits of the input sample. The embedding vector of the query sample is sorted according to the Euclidean distance between the embedding vector of the query sample and the embedding vectors of all candidate samples. The top-1 of the sorted results is taken as the recognition result.

[0053] Step b4: Calculate the cross-entropy loss and triplet loss, and use the final loss obtained by weighting the two losses in a 1:1 ratio for training.

[0054] For example, during training, triplet loss and cross-entropy loss are used as the model's loss functions, and they are weighted and summed in a 1:1 ratio to obtain the final loss value. The final loss value is then used for iterative training of the model. Wherein, the triplet loss l... triplet and cross-entropy loss L C-E The calculation formulas are shown in equation (1) and equation (2).

[0055]

[0056] Where a represents the anchored gait feature vector, p represents the positive sample feature vector, n represents the negative sample feature vector, margin is a hyperparameter controlling the minimum margin between positive and negative sample pairs, C represents the total number of sample classes, and y i p represents the sample. i This represents the predicted probability of a sample.

[0057] Step 3: Use the trained multimodal gait recognition model to identify the gait sequence of the target person.

[0058] In this embodiment, as Figure 5As shown, a large language model is used to generate an initial gait text description of the gait sequence to be identified. Based on the gait sequence, the initial gait text description is cleaned and augmented, and then used as input to a trained multimodal gait recognition model to identify the gait sequence to be identified and the target person in the gait sequences of the candidate samples.

[0059] Example 2

[0060] In this embodiment, based on the performance of the gait-text multimodal gait recognition method for real-world scenarios proposed in this invention, comparative experiments are conducted with existing methods in two different practical scenarios: real complex scenarios and people changing clothes, according to the dataset divided by the data source in step 1. Accuracy is used as the performance evaluation index to quantitatively analyze the gait recognition results. The following experimental examples demonstrate the effectiveness of this invention. The experimental results show that this invention can improve the recognition accuracy of gait recognition compared to existing methods. It fully utilizes gait information from natural language modalities in the gait recognition task, introduces gait text descriptions to assist the gait recognition task, broadens the practical application scenarios of gait recognition, and solves the limitation that in the real world, a large amount of auxiliary information exists in text or oral description form and cannot assist in gait recognition.

[0061] For example, experiments were conducted using the Gait3D-Caption and CCPG-Caption datasets created in this invention, along with gait sequences from the Gait3D and CCPG datasets. The results were compared with five existing single-modal gait recognition methods based on gait silhouettes: GaitGL, GaitPart, GLN, GaitSet, and GaitBase. Ablation experiments were also performed based on the multimodal gait recognition model designed in this invention.

[0062] Table 1 compares the results of this invention with five existing single-modal gait recognition methods based on gait silhouettes. The larger the value of the result, the higher the accuracy of gait recognition. As can be seen from the table, this invention utilizes multimodal information assistance to effectively improve the accuracy of gait recognition, enabling the model to learn more diverse gait features and semantic information.

[0063] Table 2 compares the results of ablation experiments based on the multimodal gait recognition model designed in this invention. As can be seen from the table, the gait-text multimodal gait recognition model designed in this invention can effectively utilize gait feature information in silhouette features and text features. Furthermore, through the adaptive gait-text spatiotemporal fusion module, it achieves alignment and fusion between two highly different modal features, extracts effective gait feature information for efficient model training, and achieves accurate gait recognition.

[0064] Table 1 Comparison of accuracy results of gait recognition methods

[0065]

[0066] Table 2 Comparison of ablation experimental results of multimodal gait recognition models

[0067]

[0068] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus.

[0069] Finally, it should be noted that the above descriptions are merely preferred embodiments of the present invention and are not intended to limit the present invention. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some of the technical features. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A gait-text multimodal gait recognition method for real-world scenarios, characterized in that: The following steps are included: Step 1: Data collection and generation are carried out using a large language model. The Gait3D gait dataset and the CCPG gait dataset are used as gait sequence data. A corresponding gait-featured natural language modal description is generated for each gait sequence to build a multimodal gait recognition dataset for real-world scenarios. Step 2: Build and train a multimodal gait recognition model for real-world scenarios, and perform multimodal alignment and fusion of gait features and natural language through the multimodal gait recognition model; Step 3: Use the trained multimodal gait recognition model to output the gait recognition of the target person from the gait sequence to be recognized.

2. The gait-text multimodal gait recognition method for real-world scenarios according to claim 1, characterized in that: In step 1, an initial gait text description is generated by using a large language model. The generated initial gait text descriptions are checked one by one and data augmentation is performed to obtain the final gait text description of the natural language modality.

3. The gait-text multimodal gait recognition method for real-world scenarios according to claim 2, characterized in that: The generation of the initial gait text description includes: Step a1: Add simulated character commands and observation angle restrictions to the system prompts, adjust the prompts, and use the large language model after the prompts are adjusted to simulate two different observers; Step a2: Analyze and describe the gait of the same observer from multiple angles, and remove relevant irrelevant noise interference. The multiple perspectives include: descriptions from the perspectives of the observer's hand movements, leg movements, special gait, walking speed, and stride frequency; Step a3: Based on the gait sequences in the dataset, perform data cleaning on the observer's gait text description, filter out noise in the text, and check and clean characters, clauses, content and semantics to eliminate the illusion of a large language model; Step a4: Perform data augmentation using back-translation to obtain the final gait text description as the dataset.

4. The gait-text multimodal gait recognition method for real-world scenarios according to claim 3, characterized in that: The operation of the multimodal gait recognition model constructed in step 2 includes: Step b1: Encode the gait text description into text features using a text encoder, and simultaneously use digital image processing technology to convert the gait sequence into a gait silhouette sequence, and extract gait silhouette features through a silhouette learning network; Step b2: The text features and gait silhouette features are concatenated through the adaptive gait-text spatiotemporal fusion module, and fine-grained feature fusion is performed through the spatiotemporal attention module and the adaptive gated residual connection module. Step b3: Perform feature aggregation based on temporal pooling and horizontal pyramid mapping to obtain embedding vectors and classification logits. Sort the embedding vectors of the query samples and the candidate samples according to the Euclidean distance and take the Top-1 as the recognition result. Step b4: Calculate the cross-entropy loss and triplet loss, and use the final loss obtained by weighting the two losses in a 1:1 ratio for training.

5. The gait-text multimodal gait recognition method for real-world scenarios according to claim 4, characterized in that: The operation of the adaptive gait-text spatiotemporal fusion module in step b2 includes: Step b21: Align the obtained text features with the gait silhouette features in the channel dimension using a neural network, and then extend the text features to the spatial dimension; Step b22: After feature alignment, concatenate the text features and silhouette features; Step b23: Use the spatiotemporal attention module to generate attention weights through a multi-layer grid structure, and apply them to the spliced ​​features; Step b24: Use the adaptive gated residual connection module to further filter and fuse the spliced ​​features focused by the spatiotemporal attention module.

6. The gait-text multimodal gait recognition method for real-world scenarios according to claim 5, characterized in that: In the adaptive gated residual connection module, the stitched features are generated into adaptive gate weights through the gate unit. The obtained gate weights are used to weight the stitched features and gait silhouette features focused by the spatiotemporal attention module, respectively, to perform adaptive gated residual connection, resulting in more effective gait features and filtering out irrelevant information and feature noise interference in the features.

7. The gait-text multimodal gait recognition method for real-world scenarios according to claim 6, characterized in that: In step b4, the model is iteratively trained by using triplet loss and cross-entropy loss as loss functions for the multimodal gait recognition model.

8. The gait-text multimodal gait recognition method for real-world scenarios according to claim 7, characterized in that: Triple loss L triplet The calculation formula is: Where a represents the anchored gait feature vector, p represents the positive sample feature vector, n represents the negative sample feature vector, and margin is a hyperparameter that controls the minimum edge spacing between positive and negative sample pairs.

9. A gait-text multimodal gait recognition method for real-world scenarios according to claim 8, characterized in that: Cross-entropy loss L C-E The calculation formula is: Where C represents the total number of sample categories, y i p represents the sample. i This represents the predicted probability of a sample.

10. A gait-text multimodal gait recognition method for real-world scenarios according to claim 9, characterized in that: Step 3 includes: generating an initial gait text description of the gait sequence to be identified using a large language model; cleaning and augmenting the initial gait text description; using the cleaned and augmented initial gait text description as input to a trained multimodal gait recognition model; and identifying the gait sequence to be identified and the target person in the gait sequences of the candidate samples.

Images