A gait recognition method and system based on multi-modal feature fusion

Abstract

The application discloses a gait recognition method and system based on multi-modal feature fusion, and comprises the following steps: acquiring a video stream containing a target object; extracting a gait sequence graph of the target object from the video stream; selecting a face optimal frame and a pedestrian optimal frame from the gait sequence graph of the target object; extracting a face feature of the target object from the face optimal frame; extracting a body feature of the target object from the pedestrian optimal frame; extracting a gait feature of the target object from the gait sequence graph; weighting and fusing the face feature, the body feature and the gait feature of the target object to obtain multi-modal fusion features of the target object; and performing identity recognition on the target object according to the multi-modal fusion features of the target object to obtain an identity recognition result of the target object. The accuracy of identity recognition is improved.

Description

Technical Field

[0001] This invention relates to the field of identity recognition technology, and in particular to a gait recognition method and system based on multimodal feature fusion. Background Technology

[0002] The statements in this section are merely background information related to the present invention and do not necessarily constitute prior art.

[0003] Gait is the change in posture during walking. Gait recognition is a technology that identifies individuals based on the biological and behavioral characteristics contained in gait. It is a type of biometric identification method. Its basic goal is to identify individuals by comparing a video of a pedestrian walking normally with a pre-stored collection of pedestrian walking videos. Because pedestrians vary in muscle strength, tendon and bone length, bone density, and center of gravity, these differences can uniquely identify an individual. These characteristics can then be used to build human motion models or directly extract features from the human body contour to achieve gait recognition.

[0004] While gait recognition technology has achieved some success under long-distance, uncontrolled conditions, its effectiveness in practical applications remains limited by numerous constraints. Current gait recognition technologies primarily rely on gait cycle characteristics such as stride length, stride speed, and gait rhythm. Although these features reflect the basic patterns of human walking, they can be significantly affected by complex backgrounds, severe occlusion, different camera angles, and excessively bright or dim lighting conditions, leading to low recognition accuracy. Furthermore, gait recognition technology is limited by data samples and algorithms; technological improvements require extensive data training. Especially in situations with high population diversity, the acquisition and processing of data samples are challenging, resulting in lower accuracy in gait recognition. Summary of the Invention

[0005] To address the aforementioned problems, this invention proposes a gait recognition method and system based on multimodal feature fusion, which improves the accuracy of identity recognition.

[0006] To achieve the above objectives, the present invention adopts the following technical solution:

[0007] Firstly, a gait recognition method based on multimodal feature fusion is proposed, including:

[0008] Obtain the video stream containing the target object;

[0009] Extracting gait sequence maps of target objects from video streams;

[0010] Select the optimal face frame and the optimal pedestrian frame from the gait sequence map of the target object;

[0011] Extract facial features of the target object from the best face frame;

[0012] Extract human body features of the target object from the best pedestrian frame;

[0013] Extract gait features of the target object from the gait sequence image;

[0014] The facial features, body features, and gait features of the target object are weighted and fused to obtain the multimodal fusion features of the target object;

[0015] The target object is identified based on its multimodal fusion features, and the identification result of the target object is obtained.

[0016] Secondly, a gait recognition system based on multimodal feature fusion is proposed, including:

[0017] The video acquisition module is used to acquire video streams containing the target object;

[0018] The gait sequence map acquisition module is used to extract the gait sequence map of the target object from the video stream;

[0019] The optimal frame determination module is used to select the optimal face frame and the optimal pedestrian frame from the gait sequence map of the target object.

[0020] The feature extraction module is used to extract facial features of the target object from the best face frame; extract human body features of the target object from the best pedestrian frame; and extract gait features of the target object from the gait sequence image.

[0021] The feature fusion module is used to weightedly fuse the facial features, body features, and gait features of the target object to obtain the multimodal fusion features of the target object;

[0022] The identity recognition module is used to identify the target object based on its multimodal fusion features and obtain the identity recognition result.

[0023] Thirdly, an electronic device is proposed, including a memory and a processor, as well as computer instructions stored in the memory and running on the processor. When the computer instructions are executed by the processor, they complete the steps described in a gait recognition method based on multimodal feature fusion.

[0024] Fourthly, a computer-readable storage medium is proposed for storing computer instructions, which, when executed by a processor, complete the steps of a gait recognition method based on multimodal feature fusion.

[0025] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0026] 1. This invention obtains multimodal fusion features by fusing gait features, human body features and facial features, and then uses the fusion features to identify the target object, thereby improving the accuracy of target object identification.

[0027] 2. This invention determines the optimal face frame and the optimal pedestrian frame from the gait sequence image, and then extracts face features and body features from the optimal face frame and the optimal pedestrian frame respectively, ensuring the accuracy of the extracted face features and body features; when the face features and body features are fused with the gait features to perform target object identification, the accuracy of target object identification is guaranteed.

[0028] 3. This invention evaluates the quality scores of facial features, body features, and gait features, and then determines the weight of each feature based on the quality scores. Based on these weights, the facial features, body features, and gait features are weighted and fused. The resulting multimodal fusion feature takes into account the quality of each feature and does not give excessive weight to features with low quality scores, thereby further improving the accuracy of the identity recognition results when using the multimodal fusion feature for target object identification.

[0029] Advantages of additional aspects of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description

[0030] The accompanying drawings, which form part of this application, are used to provide a further understanding of this application. The illustrative embodiments of this application and their descriptions are used to explain this application and do not constitute an undue limitation of this application.

[0031] Figure 1 The flowchart is shown in the embodiment.

[0032] Figure 2 The quality judgment flowchart is disclosed in the embodiment. Detailed Implementation

[0033] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0034] It should be noted that the following detailed descriptions are illustrative and intended to provide further explanation of this application. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains.

[0035] Example 1

[0036] Current gait recognition technology primarily relies on gait cycle features such as stride length, stride speed, and gait rhythm. While these features can reflect the basic patterns of human walking, they can be significantly affected by real-world environments, such as complex backgrounds, severe occlusion, different camera angles, and excessively bright or dim lighting conditions, leading to low recognition accuracy. Furthermore, gait recognition technology is limited by data samples and algorithms; technological improvements require extensive data training, especially given the diversity of populations, where data acquisition and processing are challenging.

[0037] Furthermore, current gait recognition technology neglects the influence of other biometrics, such as height, weight, and age. These biometrics affect human walking; for example, height and weight influence stride length and cadence, while age and health status affect gait rhythm and walking speed. The fact that current gait recognition technology does not consider the impact of these features leads to lower recognition accuracy.

[0038] To address the aforementioned technical problems, this embodiment discloses a gait recognition method based on multimodal feature fusion, such as... Figure 1 As shown, it includes:

[0039] S1: Obtain the video stream containing the target object.

[0040] This embodiment uses a camera or other device to acquire a video stream containing the target object.

[0041] Furthermore, the acquired video stream may contain one or more target objects.

[0042] S2: Extract the gait sequence map of the target object from the video stream.

[0043] When a video stream contains multiple target objects, extract the gait sequence map of each target object from the video stream.

[0044] The process of extracting the gait sequence map of the target object from the video stream is as follows: detect and track the target object in the video stream to obtain the image sequence of the target object; and obtain the gait sequence map of the target object based on the image sequence of the target object.

[0045] Specifically, for each element in the video stream containing the target object x i The video frames are preprocessed, including resizing, normalizing, and adjusting the channel order, to obtain the preprocessed image.

[0046] An object detection model is used to detect objects in the preprocessed images to determine the target objects in each preprocessed image.

[0047] Preferably, the target detection model is constructed using a YOLOv7 network, with the preprocessed image as input and the identified target object as output.

[0048] The preprocessed image is input into the object detection model. The model first extracts semantic and spatial information from the preprocessed image using a convolutional neural network. It then fuses features from different levels to obtain a more comprehensive and accurate feature representation. On the feature map after feature fusion, convolutional and fully connected layers are used to classify and locate the target object. Post-processing of the detection results includes non-maximum suppression (NMS) and confidence thresholding. Finally, the target object x is output. i The test results.

[0049] After detecting the target object x i Next, the deep sort method is used for target tracking. A unique ID is generated for each detected target object, and a deep learning model is used to extract its feature vector. Then, target matching is performed. For each detected target object x... i The algorithm extracts the object's appearance features, calculates its similarity score with all tracked objects in the previous frame, and selects the one with the highest score as the matching target. Next, for matching targets with similarity scores exceeding a threshold, a Kalman filter is used to predict their position in the next frame, and their state information (position, velocity, etc.) is updated. The deep sort method can still mark the target with its initial ID even after it has been occluded and reappeared, improving the accuracy of moving target tracking. By detecting and tracking target objects in the video stream, an image sequence of the target objects is obtained.

[0050] Identify the human-shaped region of the target object in the image sequence of the target object, and use the human-shaped region to obtain the gait sequence of the target object.

[0051] S3: Select the optimal face frame and the optimal pedestrian frame from the gait sequence map of the target object.

[0052] Obtain target object x i After obtaining the gait sequence diagram, select the target object x from the current sequence. i The frame with the best quality for the face and the body is selected as the optimal frame for the face and the optimal frame for the pedestrian, respectively.

[0053] Preferably, the maximum motion stability method is used to select the optimal face frame and the optimal pedestrian frame from the gait sequence map of the target object. Specifically:

[0054] In target object x i During target tracking, for target object x iThe face region and body region are tracked separately, and the motion speed and direction of the face region and body region in different frames are calculated. For the face region and body region, the motion speed difference and direction change between each frame and the adjacent frame are calculated. The motion speed difference and direction change are weighted and summed to obtain the motion change between each frame and the adjacent frames. The frame with the smallest motion change with the adjacent frames is selected as the most stable frame and is selected as the optimal frame. Among them, the optimal frame determined based on the face region is the face optimal frame, and the optimal frame determined based on the body region is the pedestrian optimal frame.

[0055] The maximum motion stabilization method can avoid situations where the target moves rapidly or changes its direction of motion, thus improving the robustness of tracking.

[0056] S4: Extract facial features of the target object from the best face frame; extract human body features of the target object from the best pedestrian frame; extract gait features of the target object from the gait sequence image.

[0057] In this embodiment, after selecting the optimal frames of the target object's face and pedestrians, facial and human features are extracted, and these features are used to assist in gait recognition of the target object.

[0058] When extracting facial and body features, the facial region of the target object in the optimal face frame and the body region of the target object in the optimal pedestrian frame are determined; facial features f of the target object are extracted from the facial region of the target object. i1 Extracting human features of the target object from its human body region. i2 .

[0059] Preferably, a face feature extraction network is used to extract face features from the face region, and a body feature extraction network is used to extract body features from the body region. Both the face feature extraction network and the body feature extraction network are constructed using deep neural networks and obtained after training with corresponding training data.

[0060] Among them, facial features f i1 It includes the target audience's gender, age, and ethnicity.

[0061] Human characteristics f i2 This includes the target's height, weight, and the shooting angle.

[0062] Furthermore, this embodiment also employs a gait feature extraction network to extract features from the target object x. i Extracting gait features f from gait sequence diagrams i3 .

[0063] Among them, the face feature extraction network, body feature extraction network and gait feature extraction network are all constructed using deep convolutional neural networks and trained with corresponding training data.

[0064] Gait features f i3 It includes features such as stride length, stride frequency, and gait cycle.

[0065] S5: Weighted fusion of facial features, body features and gait features of the target object to obtain the multimodal fusion features of the target object.

[0066] Since the facial, body, and gait features generated by pedestrians during walking are subject to various interferences, this embodiment evaluates the quality of these three features before extracting them. The unsupervised quality assessment algorithm designed in this method, combined with corresponding facial, body, and gait feature extraction models, receives the optimal facial frame, optimal pedestrian frame, and gait sequence image to be evaluated, providing a correct quality score for the input image from a recognition perspective, rather than from an intuitive visual quality perspective.

[0067] This embodiment first determines the quality score of each feature among facial features, body features, and gait features; then determines the weight of the corresponding feature based on the quality score of each feature, and performs weighted fusion of the three features.

[0068] Preferably, the quality score of each feature is mapped to a range of 0 to 1 to obtain the weight of each feature.

[0069] The process of determining the quality score for each feature is as follows:

[0070] like Figure 2 As shown, Figure 2 In this diagram, InputI is the input vector I, which is the vector input to the model after image preprocessing and other operations. Specifically, it includes the optimal face frame, the optimal pedestrian frame, and the gait sequence image. OutputO(I) is the output feature O(I) of the feature extraction network M. When the feature extraction network M is a face feature extraction network, it outputs face features; when the feature extraction network M is a human body feature extraction network, it outputs human body features; and when the feature extraction network M is a gait feature extraction network, it outputs gait features.

[0071] The feature extraction network M consists of multiple feature extraction subnetworks, M i This represents the feature extraction subnet of the feature extraction network M. i It is obtained by randomly dropping out some neurons from network M. i This represents the feature vector output by the corresponding subnet. For all feature vectors output by the subnets, calculate the Euclidean distance d(o) between each pair of features. i ,o jAfter calculating all the feature vectors, the quality is calculated using the following formula:

[0072]

[0073] In the formula, q(O(I)) is the quality score of the output feature O(I), σ is the sigmoid activation function, m is the number of subnets in the feature extraction network, and the sigmoid activation function is used to map q to the interval between 0 and 1.

[0074] Using the above method, the quality of facial features, body features and gait features are evaluated respectively, and the quality scores of facial features, body features and gait features are obtained. These scores are then mapped to the interval between 0 and 1 to obtain the weights of facial features, body features and gait features, which are q1, q2 and q3 respectively.

[0075] The facial features f obtained in S4 are weighted by weights q1, q2, and q3. i1 Human characteristics f i2 and gait features f i3 Perform weighted fusion to obtain the multimodal fusion features f of the target object. i .in:

[0076]

[0077] Among them, f i This indicates that the method applies to the target object x. i The extracted multimodal fusion features, q1, q2, and q3 represent the face features f, respectively. i1 Human characteristics f i2 and gait features f i3 The weight.

[0078] S6: Based on the multimodal fusion features of the target object, perform identity recognition on the target object to obtain the identity recognition result of the target object.

[0079] When using the multimodal fusion features of the target object obtained by S5 calculation for identity recognition, facial features, body features and gait features are comprehensively considered, thereby effectively improving the accuracy of identity recognition.

[0080] Example 2

[0081] In this embodiment, a gait recognition system based on multimodal feature fusion is disclosed, comprising:

[0082] The video acquisition module is used to acquire video streams containing the target object;

[0083] The gait sequence map acquisition module is used to extract the gait sequence map of the target object from the video stream;

[0084] The optimal frame determination module is used to select the optimal face frame and the optimal pedestrian frame from the gait sequence map of the target object.

[0085] The feature extraction module is used to extract facial features of the target object from the best face frame; extract human body features of the target object from the best pedestrian frame; and extract gait features of the target object from the gait sequence image.

[0086] The feature fusion module is used to weightedly fuse the facial features, body features, and gait features of the target object to obtain the multimodal fusion features of the target object;

[0087] The identity recognition module is used to identify the target object based on its multimodal fusion features and obtain the identity recognition result.

[0088] Example 3

[0089] In this embodiment, an electronic device is disclosed, including a memory and a processor, as well as computer instructions stored in the memory and running on the processor. When the processor executes the computer instructions, it completes the steps of the gait recognition method based on multimodal feature fusion disclosed in Embodiment 1.

[0090] Example 4

[0091] In this embodiment, a computer-readable storage medium is disclosed for storing computer instructions, which, when executed by a processor, complete the steps of the gait recognition method based on multimodal feature fusion disclosed in Embodiment 1.

[0092] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit it. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that modifications or equivalent substitutions can still be made to the specific implementation of the present invention. Any modifications or equivalent substitutions that do not depart from the spirit and scope of the present invention should be covered within the scope of protection of the claims of the present invention.

Claims

1. A gait recognition method based on multimodal feature fusion, characterized in that, include: Obtain the video stream containing the target object; Extracting gait sequence maps of target objects from video streams; Select the optimal face frame and the optimal pedestrian frame from the gait sequence map of the target object; Extract facial features of the target object from the best face frame; Extract human body features of the target object from the best pedestrian frame; Extract gait features of the target object from the gait sequence image; The facial features, body features, and gait features of the target object are weighted and fused to obtain the multimodal fusion features of the target object. Specifically, this involves: determining the quality score of each feature among the facial features, body features, and gait features; determining the weight of the corresponding feature based on the quality score of each feature; and performing weighted fusion of the three features. A feature extraction network is then used. M Includes multiple feature extraction subnetworks, M i Representation of feature extraction network M Feature extraction subnet, subnet M i It is the internet M This was obtained after randomly dropping out some neurons. o i This represents the feature vector output by the corresponding subnet; for all feature vectors output by the subnets, calculate the Euclidean distance between each pair of features. d ( o i , o j After calculating all the feature vectors, the quality is calculated using the following formula: In the formula, For output features The quality score is given by σ, where σ is the sigmoid activation function and m is the number of subnets in the feature extraction network. The target object is identified based on its multimodal fusion features, and the identification result of the target object is obtained.

2. The gait recognition method based on multimodal feature fusion as described in claim 1, characterized in that, The target object in the video stream is detected and tracked to obtain the image sequence of the target object; based on the image sequence of the target object, the gait sequence map of the target object is obtained.

3. The gait recognition method based on multimodal feature fusion as described in claim 1, characterized in that, Determine the face region of the target object in the optimal face frame and the body region of the target object in the optimal pedestrian frame; Extract facial features of the target object from its facial region; Extract human features of the target object from the human body region of the target object.

4. The gait recognition method based on multimodal feature fusion as described in claim 1, characterized in that, The maximum motion stability method is used to select the optimal face frame and the optimal pedestrian frame from the gait sequence map of the target object.

5. The gait recognition method based on multimodal feature fusion as described in claim 1, characterized in that, The quality score of each feature is mapped to a range of 0 to 1 to obtain the weight of each feature.

6. The gait recognition method based on multimodal feature fusion as described in claim 1, characterized in that, Gait characteristics include stride length, cadence, and gait cycle; Facial features include the target's gender, age, and ethnicity; Human characteristics include the height, weight, and shooting angle of the target subject.

7. A gait recognition system based on multimodal feature fusion, characterized in that, include: The video acquisition module is used to acquire video streams containing the target object; The gait sequence map acquisition module is used to extract the gait sequence map of the target object from the video stream; The optimal frame determination module is used to select the optimal face frame and the optimal pedestrian frame from the gait sequence map of the target object. The feature extraction module is used to extract facial features of the target object from the best face frame; extract human body features of the target object from the best pedestrian frame; and extract gait features of the target object from the gait sequence image. The feature fusion module is used to weightedly fuse facial features, body features, and gait features of the target object to obtain multimodal fused features of the target object. Specifically, it determines the quality score of each feature among facial features, body features, and gait features; determines the weight of the corresponding feature based on the quality score of each feature; and performs weighted fusion of the three features. The feature extraction network... M Includes multiple feature extraction subnetworks, M i Representation of feature extraction network M Feature extraction subnet, subnet M i It is the internet M This was obtained after randomly dropping out some neurons. o i This represents the feature vector output by the corresponding subnet; for all feature vectors output by the subnets, calculate the Euclidean distance between each pair of features. d ( o i , o j After calculating all the feature vectors, the quality is calculated using the following formula: In the formula, For output features The quality score is given by σ, where σ is the sigmoid activation function and m is the number of subnets in the feature extraction network. The identity recognition module is used to identify the target object based on its multimodal fusion features and obtain the identity recognition result.

8. An electronic device, characterized in that, It includes a memory and a processor, as well as computer instructions stored in the memory and running on the processor, which, when executed by the processor, perform the steps of the gait recognition method based on multimodal feature fusion as described in any one of claims 1-6.

9. A computer-readable storage medium, characterized in that, Used to store computer instructions, which, when executed by a processor, complete the steps of the gait recognition method based on multimodal feature fusion as described in any one of claims 1-6.

Images