Gait recognition method based on multi-modal feature fusion in low-light scene

Abstract

The application discloses a gait recognition method based on multi-modal feature fusion in a low-light scene and belongs to the gait recognition field. The method first obtains an initial detection set from a batch of pictures collected from a video, inputs the initial detection set into a person detection extraction network to extract a gait contour graph in the pictures and carries out noise reduction processing on the gait contour graph to improve the definition of the gait contour graph in the low-light scene, then inputs the initial detection set into a person skeleton feature extraction network to generate a skeleton model of a pedestrian in the pictures, and then inputs the processed gait contour graph and the skeleton model into a multi-modal feature fusion network, fuses two logical outputs in the fusion network to obtain a final pedestrian recognition result. The application can fully utilize various information existing in the pictures in the low-light condition, obtain a more comprehensive feature representation, and effectively improve the accuracy, robustness and practicability of the gait recognition in the low-light scene.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of gait recognition, and particularly relates to a gait recognition method based on multi-modal feature fusion in a low-light scene. BACKGROUND

[0002] Gait recognition is a biometric application that aims to identify a person by their walking pattern. In the field of deep learning, gait recognition is considered as a visual-based person retrieval method, which identifies moving subjects from a given sequence of gait captured by a visual camera. Compared with other forms of biometric technology (such as face recognition, fingerprint recognition, etc.), gait recognition has many significant advantages. The outstanding advantage of gait as a biometric feature is that it can identify people at a distance. In other words, gait can be used at low resolution. It usually occupies more pixels than other biometric features (such as face, fingerprint, iris), and gait information has a unique advantage in gait recognition as an inherent biometric feature. By extracting key information such as gait contour maps and gait skeleton features from a person's gait sequence, we can accurately identify the identity of the person, thereby achieving the identification of a person at a distance through gait.

[0003] Although gait recognition has made significant progress in the past few years, there are still many challenges in practical applications. The main challenge is to identify a scene with too low light. In a low-light situation, it is difficult to extract a complete human contour from a surveillance video obtained from real life. Low-quality contours have a great impact on template making, which will make it extremely difficult to identify a person in a low-light scene, thereby greatly reducing the recognition accuracy.

[0004] Current gait recognition techniques are mainly divided into two types: model-based and template-based. Gait energy images, which are the average of consecutive contours within a complete gait cycle, are the most commonly used template-based method, which requires image preprocessing from gait images or videos to distinguish human contours. The advantage of this method is that it can achieve reliable recognition results in a controllable environment, because these template features provide a large amount of discriminative information for gait recognition. However, it is difficult to obtain a complete human contour in real life due to occlusion and light changes. In the second type of model-based method, a pre-defined human model is needed to describe the dynamic and static features of the person. The main disadvantage of this category is that most methods only use sparse key point information, and there are still great challenges in extracting the basic model from gait images and videos. These methods lack flexibility due to the fixed recognition method and feature aggregation method, and have obvious shortcomings in extracting gait contours and skeleton features in low-light conditions, making it difficult to apply to the identification of people in low-light scenes. SUMMARY

[0005] This invention provides a gait recognition method based on multimodal feature fusion for low-light scenarios. This method can fully utilize various information present in images under low-light conditions to obtain a more comprehensive feature representation, effectively improving the accuracy, robustness, and practicality of gait recognition in low-light environments.

[0006] This invention provides a gait recognition method based on multimodal feature fusion in low-light scenes, comprising the following steps:

[0007] Step 1: Obtain gait contour sequence from a single person walking video in a low-light scene. Remodel the contour sequence based on a portion of the contour sequence and reduce noise. Input the denoised gait contour into the person detection and extraction network to obtain the gait contour features of the pedestrian.

[0008] Step 2: Input the gait contour sequence into the human skeleton feature extraction network to extract the skeleton features of pedestrians in the image;

[0009] Step 3: Input the pedestrian's gait contour features and skeleton features into a multimodal feature fusion network to perform feature fusion, obtain the fused pedestrian gait features, and perform gait recognition based on the fused pedestrian gait features.

[0010] Optionally, in one embodiment of the present invention, step 1 further includes:

[0011] Step 11: Process the CASIA-B dataset to simulate pedestrian gait in low-light scenes and obtain single-person walking videos in low-light scenes;

[0012] Step 12: Use a pedestrian detection and segmentation algorithm to extract and segment the single-person walking images in the single-person walking video into a gait contour map sequence;

[0013] Step 13: By synthesizing gait from partial sequence frames, the gait profile sequence is modeled as a Gaussian distribution and the gait cycle features are expressed as a continuous function, thus generating a complete cyclic gait.

[0014] Step 14: Denoise the reconstructed gait contour map and input the denoised gait contour map into the person detection and extraction network to obtain the gait contour features of the pedestrian.

[0015] Optionally, in one embodiment of the present invention, step 13 further includes:

[0016] The dimensionality of the gait contour data is reduced, and features are extracted using the following formula:

[0017] ξλξ T =(SS) T )×1 / N

[0018] Where ξ represents the common feature vector basis generated from the gait dataset, S represents the shape vector matrix, and N represents the total number of contours in the gait dataset;

[0019] shape vector u i Project the vectors onto the common feature basis vectors ξ and obtain the corresponding feature vectors a. i The formula is as follows:

[0020] a i =ξ T u i

[0021] Where, ξ T u represents the transpose of the common feature vector basis generated from the gait dataset. i Represents a shape vector;

[0022] The feature vector a i The transformation is converted into a continuous function, and the resulting eigenvectors are aligned and enhanced to form a zero-mean eigenma matrix F, as shown in the following formula:

[0023] F≡[a1(t),…,a M (t)] m×∞

[0024] Among them, a i (t) represents the eigenvector a i The transformed eigenvectors, where m represents the number of eigenvectors and ∞ indicates that F has infinite dimensions;

[0025] The infinite-dimensional F matrix can be transformed into a finite-dimensional eigencovariance matrix, as shown in the following formula:

[0026] K M×M =F T F

[0027] Among them, K M×M Let F represent a finite-dimensional eigenvector matrix, where F is the eigenma matrix with zero mean.

[0028] Calculate the components of K using calculus, redescribe K using these components, and then calculate K. M×M The eigenvector φ is given by the following formula:

[0029] K i×j =∫ T a i (t)a j (t)dt=vηv T

[0030] φ=Fv

[0031] By the average of the eigenvectors And the coefficients of the eigenvectors are used to calculate the new eigenvectors. The new feature vectors are then transformed into vectorized gait contour periods through equations, and the final gait contour sequence is generated through modeling, as shown in the following formula:

[0032]

[0033] Where ψ represents the eigenvector basis. Let f represent the average gait profile vector, and let f denote the modeling of the gait vector. This represents the newly generated gait profile sequence diagram.

[0034] Optionally, in one embodiment of the present invention, in step 14, denoising the reconstructed gait contour map includes:

[0035] The gait profile M1 is obtained using a classic morphological processing algorithm, as shown in the following formula:

[0036] M1 = erode(M)

[0037] Gait profile M2 is obtained by performing Mask and Contour operations on the obtained gait profile M1, as shown in the following formula:

[0038] M2 = Contours(Mask(M1))

[0039] Mask means creating a mask image of the same size as the original image, and Contours means contour extraction.

[0040] Gait profile M2 is digitized to obtain gait profile M3, and then Gaussian-processed to obtain the final denoised gait profile M4. The formula is as follows:

[0041] M3 = dilate(M2)

[0042] M4 = Gaussian(M3)

[0043] Here, dilate represents the dilation operation on the gait profile map after removing ghosting, and Gaussian represents the edge smoothing process of the gait profile map using Gaussian filtering.

[0044] Optionally, in one embodiment of the present invention, step 14, inputting the denoised gait contour map into the person detection and extraction network to obtain the pedestrian's gait contour features, includes:

[0045] The denoised gait contour is input into the person detection and extraction network to obtain multiple feature maps. The person detection and extraction network consists of multiple convolutional stages, and the convolutional result of each layer is passed to the next convolutional layer, as shown in the following formula:

[0046]

[0047] in, H represents the convolution operation. l-1 It is the output of the previous convolutional layer, W i Contains multiple filters, b l This indicates the bias shared by the normalization of features from different contour maps within the same sequence;

[0048] The normalized activity of each feature map at a certain spatial location is calculated using the following formula:

[0049]

[0050] Where a, β, and y represent adjustable configuration parameters;

[0051] The person detection and extraction network is trained using logistic regression. The predictor is constructed using two neural network layers containing trainable parameters, as shown in the following formula:

[0052] L(r)=ReLU(ξ+D a,β -D a,r )

[0053] Where a represents the true label, β represents the sample with the same label as the true label a, r represents the sample with a different label than the true label a, and ξ represents the boundary distance. The larger the boundary distance, the better the D. a,β The closer the expected distance, the better D a,r The greater the distance, the more the outer ReLU represents the activation function.

[0054] Optionally, in one embodiment of the present invention, step 2 further includes:

[0055] The pedestrian's posture information is estimated based on the gait contour sequence, and the human body joints are normalized based on the posture coordinate sequence in the posture information, as shown in the following formula:

[0056]

[0057] Among them, P i Let p′ be the coordinate of body joint i. i This is the normalized result, P neck H is the neck coordinate. hn This represents the height difference between the center of the neck and the center of the hip.

[0058] The gait skeleton features of the human skeleton are extracted using a human skeleton feature extraction network, as shown in the following formula:

[0059]

[0060] Among them, f in This represents a feature map with human joints, f out A represents the output gait skeleton features. k Represents the adjacency matrix of intra-frame key points. The degree matrix represents the distance between joints, with all diagonal elements being zero. k Let σ(·) represent the learnable weight matrix, and let σ(·) represent the activation function.

[0061] The predictor is constructed using cross-entropy loss to classify gait skeleton features, as shown in the following formula:

[0062]

[0063] Among them, y i It is the true label of the body joints, p i It is the predicted probability of body joint i.

[0064] Optionally, in one embodiment of the present invention, step 3, gait recognition based on the fused pedestrian gait features, includes:

[0065] Calculate the feature vector distance between the fused pedestrian gait features and the sample features, sort the calculated feature vector distances, and obtain the similarity result. The calculation formula is as follows:

[0066]

[0067] Among them, D m and D n D represents the minimum and maximum distances, respectively. sort D represents the distance between sorted feature vectors. n The similarity results are after normalization.

[0068] Based on the similarity results, the similarity results of the same sample label are summed and sorted. The results with the highest similarity are selected, and the probability distribution function is used to calculate the probability of the sample label to which the fused pedestrian gait features belong. The gait recognition result is determined based on the probability.

[0069] This invention discloses a gait recognition method based on multimodal feature fusion in low-light scenarios. First, an initial detection set is generated by acquiring a batch of images from a captured video. This initial detection set is then input into a person detection and extraction network to extract gait contour maps from the images. Noise reduction processing is applied to the gait contour maps to improve their clarity in low-light conditions. Next, the initial detection set is input into a person skeleton feature extraction network to generate a skeleton model of the pedestrian in the image. Finally, the processed gait contour map and skeleton model are input into a multimodal feature fusion network. The two logical outputs of the fusion network are then fused to obtain the final pedestrian recognition result. This invention can fully utilize various information present in images under low-light conditions to obtain a more comprehensive feature representation and effectively improve the accuracy, robustness, and practicality of gait recognition in low-light scenarios.

[0070] Additional aspects and advantages of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description

[0071] The above and / or additional aspects and advantages of the present invention will become apparent and readily understood from the following description of the embodiments taken in conjunction with the accompanying drawings, wherein:

[0072] Figure 1 A flowchart illustrating a gait recognition method based on multimodal feature fusion in low-light scenarios according to an embodiment of the present invention;

[0073] Figure 2 This is a flowchart of the gait recognition method of the present invention;

[0074] Figure 3 This is an architecture diagram of the gait feature extraction network based on multimodal feature fusion of the present invention. Detailed Implementation

[0075] Embodiments of the present invention are described in detail below, examples of which are illustrated in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain the present invention, and should not be construed as limiting the present invention.

[0076] Figure 1 This is a flowchart of a gait recognition method based on multimodal feature fusion in low-light scenes according to an embodiment of the present invention.

[0077] like Figure 1 As shown, the gait recognition method based on multimodal feature fusion in low-light scenarios includes the following steps:

[0078] Step 1: Obtain gait contour sequence from a single person walking video in a low-light scene. Remodel the contour sequence based on a portion of the contour sequence and perform noise reduction. Input the denoised gait contour into a person detection and extraction network to obtain the gait contour features of the pedestrian.

[0079] In an embodiment of the present invention, step 1 further includes:

[0080] Step 11: Process the CASIA-B dataset to simulate pedestrian gait in low-light scenes and obtain single-person walking videos in low-light scenes;

[0081] Step 12: Use a pedestrian detection and segmentation algorithm to extract and segment the single-person walking images in the single-person walking video into a gait contour map sequence;

[0082] Step 13: By synthesizing gait from partial sequence frames, the gait profile sequence is modeled as a Gaussian distribution and the gait cycle features are expressed as a continuous function, thus generating a complete cyclic gait.

[0083] Step 14: Denoise the reconstructed gait contour map and input the denoised gait contour map into the person detection and extraction network to obtain the gait contour features of the pedestrian.

[0084] In an embodiment of the present invention, step 13 further includes:

[0085] The dimensionality of the gait contour data is reduced, and features are extracted using the following formula:

[0086] ξλξ T =(SS) T )×1 / N

[0087] Where ξ represents the common feature vector basis generated from the gait dataset, S represents the shape vector matrix, and N represents the total number of contours in the gait dataset;

[0088] shape vector u i Project the vectors onto the common feature basis vectors ξ and obtain the corresponding feature vectors a. i The formula is as follows:

[0089] a i =ξ T u i

[0090] Where, ζ T u represents the transpose of the common feature vector basis generated from the gait dataset. i Represents a shape vector;

[0091] The feature vector a iThe transformation is converted into a continuous function, and the resulting eigenvectors are aligned and enhanced to form a zero-mean eigenma matrix F, as shown in the following formula:

[0092] F≡[a1(t),…,a M (t)] m×∞

[0093] Among them, a i (t) represents the eigenvector a i The transformed eigenvectors, where m represents the number of eigenvectors and ∞ indicates that F has infinite dimensions;

[0094] The infinite-dimensional F matrix can be transformed into a finite-dimensional eigencovariance matrix, as shown in the following formula:

[0095] K M×M =F T F

[0096] Among them, K M×M Let F represent a finite-dimensional eigenvector matrix, where F is the eigenma matrix with zero mean.

[0097] Calculate the components of K using calculus, redescribe K using these components, and then calculate K. M×M The eigenvector φ is given by the following formula:

[0098] K i×j =∫ T a i (t)a j (t)dt=vηv T

[0099] φ=Fv

[0100] By the average of the eigenvectors And the coefficients of the eigenvectors are used to calculate the new eigenvectors. The new feature vectors are then transformed into vectorized gait contour periods through equations, and the final gait contour sequence is generated through modeling, as shown in the following formula:

[0101]

[0102] Where ψ represents the eigenvector basis. Let f represent the average gait profile vector, and let f denote the modeling of the gait vector. This represents the newly generated gait profile sequence diagram.

[0103] In an embodiment of the present invention, step 14, denoising the reconstructed gait contour map, includes:

[0104] The gait profile M1 is obtained using a classic morphological processing algorithm, as shown in the following formula:

[0105] M1 = erode(M)

[0106] Gait profile M2 is obtained by performing Mask and Contour operations on the obtained gait profile M1, as shown in the following formula:

[0107] M2 = Contours(Mask(M1))

[0108] Mask means creating a mask image of the same size as the original image, and Contours means contour extraction.

[0109] Gait profile M2 is digitized to obtain gait profile M3, and then Gaussian-processed to obtain the final denoised gait profile M4. The formula is as follows:

[0110] M3 = dilate(M2)

[0111] M4 = Gaussian(M3)

[0112] Here, dilate represents the dilation operation on the gait profile map after removing ghosting, and Gaussian represents the edge smoothing process of the gait profile map using Gaussian filtering.

[0113] In an embodiment of the present invention, step 14, inputting the denoised gait contour map into the person detection and extraction network to obtain the pedestrian's gait contour features, includes:

[0114] The denoised gait contour is input into the person detection and extraction network to obtain multiple feature maps. The person detection and extraction network consists of multiple convolutional stages, and the convolutional result of each layer is passed to the next convolutional layer, as shown in the following formula:

[0115]

[0116] in, H represents the convolution operation. l-1 It is the output of the previous convolutional layer, W i Contains multiple filters, b l This indicates the bias shared by the normalization of features from different contour maps within the same sequence;

[0117] The normalized activity of each feature map at a certain spatial location is calculated using the following formula:

[0118]

[0119] Where a, β, and y represent adjustable configuration parameters;

[0120] The person detection and extraction network is trained using logistic regression. The predictor is constructed using two neural network layers containing trainable parameters, as shown in the following formula:

[0121] L(r)=Re Lu(ξ+D a,β -D a,r )

[0122] Where a represents the true label, β represents the sample with the same label as the true label a, r represents the sample with a different label than the true label a, and ξ represents the boundary distance. The larger the boundary distance, the better the D. a,β The closer the expected distance, the better D a,r The greater the distance, the more the outer ReLU represents the activation function.

[0123] Step 2: Input the gait contour sequence into the human skeleton feature extraction network to extract the skeletal features of pedestrians in the image.

[0124] In an embodiment of the present invention, step 2 further includes:

[0125] Pedestrian posture information is estimated based on gait contour sequences, and human joints are normalized based on the posture coordinate sequences in the posture information, as shown in the following formula:

[0126]

[0127] Among them, P i Let p′ be the coordinate of body joint i. i This is the normalized result, P neck H is the neck coordinate. hn This represents the height difference between the center of the neck and the center of the hip.

[0128] The gait skeleton features of the human skeleton are extracted using a human skeleton feature extraction network, as shown in the following formula:

[0129]

[0130] Among them, f in This represents a feature map with human joints, f out A represents the output gait skeleton features. k Represents the adjacency matrix of intra-frame key points. The degree matrix represents the distance between joints, with all diagonal elements being zero. k Let σ(·) represent the learnable weight matrix, and let σ(·) represent the activation function.

[0131] The predictor is constructed using cross-entropy loss to classify gait skeleton features, as shown in the following formula:

[0132]

[0133] Among them, y i It is the true label of the body joints, p i It is the predicted probability of body joint i.

[0134] Step 3: Input the pedestrian's gait contour features and skeleton features into a multimodal feature fusion network to perform feature fusion, obtain the fused pedestrian gait features, and perform gait recognition based on the fused pedestrian gait features.

[0135] In one embodiment of the present invention, step 3, gait recognition based on the fused pedestrian gait features, includes:

[0136] Calculate the feature vector distance between the fused pedestrian gait features and the sample features, sort the calculated feature vector distances, and obtain the similarity result. The calculation formula is as follows:

[0137]

[0138] Among them, D m and D n D represents the minimum and maximum distances, respectively. sort D represents the distance between sorted feature vectors. n The similarity results are after normalization.

[0139] Based on the similarity results, the similarity results of the same sample label are summed and sorted. The results with the highest similarity are selected, and the probability distribution function is used to calculate the probability of the sample label to which the fused pedestrian gait features belong. The gait recognition result is determined based on the probability.

[0140] The gait recognition method based on multimodal feature fusion in low-light scenes of this invention first extracts a gait contour sequence from a single-person walking video. Based on a portion of the contour sequence, the contour sequence is remodeled. Then, noise reduction is applied to the modeled contours, and gait features are extracted from the denoised gait contours. Simultaneously, the original image sequence is extracted from the walking video, and skeleton features are extracted from the original image sequence. The two feature extraction networks are fused to form a dual-branch neural network. Combining the skeleton and gait features, the final gait sequence features are further fused to improve the accuracy and robustness of gait recognition, addressing the challenges posed by practical applications in low-light scenes. Specifically, it includes:

[0141] 1. This invention utilizes the RVM algorithm to extract gait contour sequences from single-person videos as input representations of pedestrian gait. The extracted gait contour sequences are then remodeled to obtain more complete gait contour sequences, and the modeled gait contour sequences are subjected to noise reduction processing.

[0142] 2. This invention utilizes a 3DCNN network to extract spatiotemporal information and gait features from the denoised gait contour sequence, and improves the accuracy of feature extraction by the neural network through loss functions and other methods.

[0143] 3. This invention utilizes graph convolutional neural networks to extract the temporal and spatial features of the human skeleton, and uses cross-entropy loss to construct an effective predictor to classify gait skeleton features.

[0144] 4. This invention integrates the gait contour feature extraction network and the skeleton feature extraction network into a two-branch neural network. By fusing gait contour features and skeleton features, the final robust gait features are obtained. The final gait features are then compared with the features of the image library samples to calculate the similarity and output the top-k results, thereby achieving gait recognition of people in videos.

[0145] The following detailed description of the gait recognition method based on multimodal feature fusion in low-light scenarios according to the present invention will be provided through a specific embodiment.

[0146] Considering that pedestrian videos captured by ordinary cameras in low-light conditions often lack complete and clear information about the individuals, gait contour reconstruction can supplement pedestrian information and features. However, the reconstructed gait contour still contains noise. Therefore, a denoising algorithm is designed to remove ghosting in the gait contour to obtain a clearer image. Furthermore, considering that side gait contours may still be unusable after reconstruction and denoising, skeletal features of pedestrians can be used for more accurate identification at different angles. Therefore, considering all these factors, a two-branch neural network is designed to fuse gait contour and skeletal features to obtain the final gait features, enabling more accurate pedestrian identification using ordinary cameras in low-light scenarios.

[0147] like Figure 2 and Figure 3 As shown, the gait recognition method based on multimodal feature fusion in low-light scenes includes the following steps:

[0148] Step 1: Process the CASIA-B dataset to simulate pedestrian gait in low-light scenarios.

[0149] Step 2: Use the Robust Video Matting algorithm, a pedestrian detection and segmentation algorithm, to extract and segment the single-person walking images from the gait video processed in Step 1 into gait contour sequences.

[0150] Step 3: The gait contour sequence obtained in Step 2 is synthesized from a small number of sequence frames. The corresponding features of the gait cycle are modeled as Gaussian distributions and these features are expressed as continuous functions. Finally, a relatively complete periodic gait is generated through modeling.

[0151] Step 4: Extract the spatiotemporal information and features of the gait contour data obtained in Step 3 using a 3DCNN network, and improve the accuracy of feature extraction by the neural network using loss functions and other methods.

[0152] Step 5: Create a gait dataset in a real low-light scene, label the dataset, and repeat steps 2 and 3 to obtain the gait profile map of the self-made dataset.

[0153] Step 6: Design an algorithm to reduce noise in the gait profile obtained in Step 5, thereby improving the accuracy of gait recognition.

[0154] Step 7: Extract skeleton features from the self-made dataset in Step 6. Use networks such as GaitGraph to extract and train features of the skeleton outline of the person.

[0155] Step 8: Input the denoised gait contour map obtained in Step 6 and the skeleton contour map obtained in Step 7 into the pre-trained multimodal feature fusion network for feature fusion to achieve gait recognition of people in the self-made dataset under low light conditions.

[0156] Step 9: Calculate the feature similarity between the final gait features obtained in Step 8 and the features of multiple samples in the image library, and output the sample ID with the highest similarity to the target person.

[0157] Specifically, step 2, gait contour sequence extraction, includes the following steps:

[0158] Step 2-1: Crop the pedestrian walking video obtained in Step 1 and Step 5 into image sequences of different lengths;

[0159] Step 2-2: Normalize the image obtained in Step 2-1, and crop the image obtained in Step 2-1 to obtain an image with a length and width of 64 and 60 respectively.

[0160] Step 2-3: Extract the human figure outline from each frame of the image obtained in Step 2-2 using the RVM network;

[0161] Specifically, the modeling method for the gait contour sequence in step 3 includes the following steps:

[0162] Step 3-1: First, reduce the dimensionality of the gait contour data using PCA, then extract specific features, as shown in the following formula:

[0163] ξλξT =(SS) T )×1 / N

[0164] Where ξ represents the common feature vector basis generated from the entire gait dataset, S represents the shape vector matrix obtained through PCA, and N represents the total number of contours in the gait dataset;

[0165] Step 3-2: Convert the shape vector u i Project onto the common feature basis vector ξ and obtain the corresponding feature pattern a. i The formula is as follows:

[0166] a i =ξ T u i

[0167] Where, ξ T u represents the transpose of the common feature vector basis generated from the entire gait dataset. i Represents a shape vector;

[0168] Step 3-3: Transfer the feature vector a i The transformation is converted into a basic continuous function, and the transformed eigenvectors are aligned and enhanced into a zero-mean eigenma matrix F, as shown in the following formula:

[0169] F≡[a1(t),…,a M (t)] m×∞

[0170] Among them, a i (t) represents the eigenvector a i The transformed eigenvectors, where m represents the number of eigenvectors and ∞ indicates that F has infinite dimensions.

[0171] Step 3-4: Transform the infinite-dimensional F matrix obtained in Step 3-3 into a finite-dimensional feature covariance matrix, as shown in the following formula:

[0172] K M×M =F T F

[0173] Among them, K M×M Let F represent a finite-dimensional eigenvector matrix, where F is the eigenma matrix with zero mean.

[0174] Steps 3-5: Calculate the components of K using calculus, redescribe K using these components, and then calculate K. M×M The eigenvector φ is given by the following formula:

[0175] K i×j =∫ T a i (t)a j (t)dt=vηvT

[0176] φ=Fv

[0177] Steps 3-6: Using the average value of the feature vectors And the coefficients of the eigenvectors are used to calculate the new eigenvectors. These new feature vectors are then transformed into vectorized gait contour periods through equations, and the final gait contour sequence is generated through modeling, as shown in the following formula:

[0178]

[0179] Where ψ represents the eigenvector basis of PCA. Let f represent the average gait profile vector, and let f represent the modeling of the gait vector. This represents the newly generated gait profile sequence diagram.

[0180] Specifically, step 4, the method for extracting features from the gait contour map, includes the following steps:

[0181] Step 4-1: Obtain the denoised gait profile map through step 7;

[0182] Step 4-2: Input the gait contour map into the CNN network, which consists of multiple convolutional stages. The convolutional result of each layer is passed to the next convolutional layer, as shown in the following formula:

[0183]

[0184] in, H represents the convolution operation. l-1 It is the output of the previous convolutional layer, H0 is the original input data, and W is the output of the previous convolutional layer. i Contains multiple filters, b l This indicates the bias shared by the normalization of features from different contour maps within the same sequence.

[0185] Step 4-3: Calculate the normalized activity of each feature map at a certain spatial location, using the following formula:

[0186]

[0187] Where a, β, and y represent adjustable configuration parameters.

[0188] Step 4-4: Train the entire network using logistic regression. An effective predictor is constructed using two neural network layers containing trainable parameters, as shown in the following formula:

[0189] L(r)=ReLU(ξ+D a,β -D a,r )

[0190] Where a represents the true label, β represents the sample with the same label as the true label a, r represents the sample with a different label than the true label a, and ξ represents the boundary distance. The larger this value, the stronger the D. a,β The closer the expected distance, the better D a,r The greater the distance, the better. The outer ReLU represents the activation function.

[0191] Specifically, the denoising method for the gait contour map described in step 6 includes the following steps:

[0192] Step 6-1: Use the classic morphological processing algorithm Erode to obtain the contour M1, as shown in the following formula:

[0193] M1 = erode(M)

[0194] Step 6-2: Obtain M2 by performing Mask and Contour operations on M1 obtained in Step 6-1, as shown in the following formula:

[0195] M2 = Contours(Mask(M1))

[0196] Mask represents creating a mask image of the same size as the original image, and Contours represents contour extraction.

[0197] Step 6-3: Dilate the contour map M2 obtained in step 6-2 to obtain contour map M3, and then perform Gaussian processing on M3 to obtain the final denoised contour map M4. The formula is as follows:

[0198] M3 = dilate(M2)

[0199] M4 = Gaussian(M3)

[0200] Here, dilate represents the dilation operation on the contour map after removing ghosting, and Gaussian represents the edge smoothing process of the contour map using Gaussian filtering.

[0201] Specifically, step 7, the method for extracting skeleton features from the self-made dataset, includes the following steps:

[0202] Step 7-1: Estimate pose information based on the gait profile sequence obtained in Step 2;

[0203] Step 7-2: Extract the posture coordinate sequence from the posture information obtained in Step 7-1 and perform preprocessing to normalize the human joints, using the following formula:

[0204]

[0205] Among them, P iLet p′ be the coordinate of body joint i. i This is the normalized result, P neck H is the neck coordinate. n This represents the height difference between the center of the neck and the center of the hip.

[0206] Step 7-3: Extract the temporal and spatial features of the human skeleton using a graph convolutional neural network, as shown in the following formula:

[0207]

[0208] Among them, f in This represents a feature map with human joints, f out A represents the final skeleton feature output. k Represents the adjacency matrix of intra-frame key points. The degree matrix representing the relationship between joints has all zero elements on its diagonal. k σ represents the learnable weight matrix, and σ(·) represents the activation function.

[0209] Step 7-4: Use cross-entropy loss to construct an effective predictor to classify the gait skeleton features obtained in Step 7-3, as shown in the following formula:

[0210]

[0211] Among them, y i p is the true label of sample i. i It is the predicted probability of sample i.

[0212] Specifically, step 8, multimodal feature fusion, includes the following steps: designing a dual-branch neural network, merging the gait contour feature extraction network obtained in step 4 and the skeleton contour feature extraction network obtained in step 7-2 as different branches into a dual-branch network, and finally aggregating the two types of features into the final robust gait features.

[0213] Specifically, step 9, calculating the feature similarity between gait features and sample features in the sample space, includes the following steps:

[0214] Step 9-1: Calculate the distance between feature vectors based on the final gait features obtained in Step 8 and the sample features in the image library;

[0215] Step 9-2: Sort the feature vector distances calculated in Step 9-1 to obtain the similarity results, as shown in the following formula:

[0216]

[0217] Among them, D m and D nD represents the minimum and maximum distances, respectively. sort D represents the distance between sorted feature vectors. n This is the normalized similarity result;

[0218] Step 9-3: Based on the similarity obtained in Step 9-2, sum and sort the similarity results for the same sample label, and take the top-5 results with the highest similarity.

[0219] Step 9-4: Based on the top-5 results obtained in Step 9-3, use the probability distribution function to calculate the probability of the sample label to which the contour sequence belongs, and output the gait recognition result.

[0220] The gait recognition method based on multimodal feature fusion in low-light scenes proposed in this invention can achieve relatively accurate recognition of pedestrians using ordinary cameras in low-light scenes. It mainly achieves this by reconstructing the pedestrian gait contour map and denoising the modeled gait contour map to obtain a clearer gait contour map. It fully explores and utilizes the information of the pedestrian gait contour map, and uses skeleton feature extraction to make up for the fact that the gait contour reconstruction and denoising are still unusable. It flexibly fuses skeleton features and gait contour features to form the final robust gait features, effectively improving the robustness and accuracy of gait recognition in low-light scenes.

[0221] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., refer to specific features, structures, materials, or characteristics described in connection with that embodiment or example, which are included in at least one embodiment or example of the present invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0222] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this invention, "N" means at least two, such as two, three, etc., unless otherwise explicitly specified.

[0223] Any process or method description in the flowchart or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or N executable instructions for implementing custom logic functions or processes, and the scope of preferred embodiments of the invention includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the functions involved, as will be understood by those skilled in the art to which embodiments of the invention pertain.

Claims

1. A gait recognition method based on multimodal feature fusion in low-light scenes, characterized in that, Includes the following steps: Step 1: Obtain gait contour sequence from a single person walking video in a low-light scene. Remodel the contour sequence based on a portion of the contour sequence and reduce noise. Input the denoised gait contour into the person detection and extraction network to obtain the gait contour features of the pedestrian. Step 11: Process the CASIA-B dataset to simulate pedestrian gait in low-light scenes and obtain single-person walking videos in low-light scenes; Step 12: Use a pedestrian detection and segmentation algorithm to extract and segment the single-person walking images in the single-person walking video into a gait contour map sequence; Step 13: By synthesizing gait from partial sequence frames, the gait profile sequence is modeled as a Gaussian distribution and the gait cycle features are expressed as a continuous function, thus generating a complete cyclic gait. Step 14: Denoise the reconstructed gait contour map and input the denoised gait contour map into the person detection and extraction network to obtain the gait contour features of the pedestrian. Step 2: Input the gait contour sequence into the human skeleton feature extraction network to extract the skeleton features of pedestrians in the image; Step 3: Input the pedestrian's gait contour features and skeleton features into a multimodal feature fusion network to perform feature fusion, obtain the fused pedestrian gait features, and perform gait recognition based on the fused pedestrian gait features.

2. The method according to claim 1, characterized in that, Step 13 further includes: The dimensionality of the gait contour data is reduced using PCA, and features are extracted using the following formula: xxxx T =(SS T )×1 / N Where ξ represents the common feature vector basis generated from the gait dataset, S represents the shape vector matrix, and N represents the total number of contours in the gait dataset; shape vector u i Project onto the common eigenvector basis ξ, and obtain the corresponding eigenvector a. i The formula is as follows: a i =ξ T you i Where, ξ T u represents the transpose of the common feature vector basis generated from the gait dataset. i Represents a shape vector; The feature vector a i The transformation is converted into a continuous function, and the resulting eigenvectors are aligned and enhanced to form a zero-mean eigenma matrix F, as shown in the following formula: F≡[a1(t),…,a M (t)] m×∞ Among them, a i (t) represents the eigenvector a i The transformed eigenvectors, where m represents the number of eigenvectors and ∞ indicates that F has infinite dimensions; The infinite-dimensional F matrix can be transformed into a finite-dimensional eigencovariance matrix, as shown in the following formula: K M×M =F T F Among them, K M×M Let F represent a finite-dimensional eigenvector matrix, where F is the eigenma matrix with zero mean. Calculate the components of K using calculus, redescribe K using these components, and then calculate K. M×M The eigenvector φ is given by the following formula: K i×j =∫ T a i (t)a j (t)dt=vηv T Φ=Fv By the average of the eigenvectors And the coefficients of the eigenvectors are used to calculate the new eigenvectors. The new feature vectors are then transformed into vectorized gait contour periods through equations, and the final gait contour sequence is generated through modeling, as shown in the following formula: Where ψ represents the eigenvector basis. Let f represent the average gait profile vector, and let f denote the modeling of the gait vector. This represents the newly generated gait profile sequence diagram.

3. The method according to claim 1, characterized in that, In step 14, the noise reduction of the reconstructed gait profile includes: The gait profile M1 is obtained using a classic morphological processing algorithm, as shown in the following formula: M1 = erode(M) Gait profile M2 is obtained by performing Mask and Contour operations on the obtained gait profile M1, as shown in the following formula: M2 = Contours(Mask(M1)) Mask means creating a mask image of the same size as the original image, and Contours means contour extraction. Gait profile M2 is digitized to obtain gait profile M3, and then Gaussian-processed to obtain the final denoised gait profile M4. The formula is as follows: M3 = dilate(M2) M4 = Gaussian(M3) Here, dilate represents the dilation operation on the gait profile map after removing ghosting, and Gaussian represents the edge smoothing process of the gait profile map using Gaussian filtering.

4. The method according to claim 1, characterized in that, In step 14, the denoised gait contour map is input into the person detection and extraction network to obtain the pedestrian's gait contour features, including: The denoised gait contour is input into the person detection and extraction network to obtain multiple feature maps. The person detection and extraction network consists of multiple convolutional stages, and the convolutional result of each layer is passed to the next convolutional layer, as shown in the following formula: in, H represents the convolution operation. l-1 It is the output of the previous convolutional layer, W i Contains multiple filters, b l This indicates the bias shared by the normalization of features from different contour maps within the same sequence; The normalized activity of each feature map at a certain spatial location is calculated using the following formula: Where a, β, and y represent adjustable configuration parameters; The person detection and extraction network is trained using logistic regression. The predictor is constructed using two neural network layers containing trainable parameters, as shown in the following formula: L(r)=ReLu(ξ+D l,m -D l,r ) Where l represents the true label, m represents the sample with the same label as the true label l, r represents the sample with a different label than the true label l, and ξ represents the boundary distance. The larger the boundary distance, the greater the D. l,m The closer the expected distance, the better D l,r The greater the distance, the more the outer ReLU represents the activation function.

5. The method according to claim 1, characterized in that, Step 2 further includes: The pedestrian's posture information is estimated based on the gait contour sequence, and the human body joints are normalized based on the posture coordinate sequence in the posture information, as shown in the following formula: Among them, P i Let p′ be the coordinate of body joint i. i This is the normalized result, P neck H is the neck coordinate. hn This represents the height difference between the center of the neck and the center of the hip. The gait skeleton features of the human skeleton are extracted using a human skeleton feature extraction network, as shown in the following formula: Among them, f in This represents a feature map with human joints, f out A represents the output gait skeleton features. k Represents the adjacency matrix of intra-frame key points. The degree matrix represents the distance between joints, with all diagonal elements being zero. k Let σ(·) represent the learnable weight matrix, and let σ(·) represent the activation function. The predictor is constructed using cross-entropy loss to classify gait skeleton features, as shown in the following formula: Among them, y i It is the true label of the body joints, p i It is the predicted probability of body joint i.

6. The method according to claim 1, characterized in that, In step 3, gait recognition based on the fused pedestrian gait features includes: Calculate the feature vector distance between the fused pedestrian gait features and the sample features, sort the calculated feature vector distances, and obtain the similarity result. The calculation formula is as follows: Among them, D m and D n D represents the minimum and maximum distances, respectively. sort D represents the distance between sorted feature vectors. X The similarity results are after normalization. Based on the similarity results, the similarity results of the same sample label are summed and sorted. The results with the highest similarity are selected, and the probability distribution function is used to calculate the probability of the sample label to which the fused pedestrian gait features belong. The gait recognition result is determined based on the probability.

Images