Human body gait recognition method and system in complex scene

Abstract

The invention belongs to the technical field of gait recognition, and particularly relates to a human gait recognition method and system in a complex scene. And sequentially performing illumination correction, human body posture angle alignment and shielding processing on the extracted human body gait data, and performing human body gait recognition based on the gait data obtained after illumination correction, human body posture angle alignment and shielding processing. According to the occlusion processing, a comprehensive and reliable occlusion area judgment standard is set based on the connection relation between skeleton nodes, the human anatomy reference distance, kinematics constraint and the time continuity requirement, and then a missing human body area is reconstructed through optical flow compensation or bilinear interpolation based on information of visible pixels around the occlusion area. The accuracy of gait recognition in a complex scene is improved, the problem of adaptability of the complex scene is solved in a breakthrough mode, high-precision gait identity recognition is achieved, and the security and protection monitoring requirement of a real scene is met.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of gait recognition, and particularly relates to a human gait recognition method and system in a complex scene. BACKGROUND

[0002] Gait recognition technology can effectively improve public security monitoring capability, provide real-time and accurate identity recognition means for relevant departments, help to reduce crime rate and improve emergency response capability, provide new technical support for intelligent monitoring, traffic management and urban planning, thereby optimizing urban management efficiency and maintaining social safety and stability. For example, through gait recognition, real-time monitoring and analysis of urban flow are realized, data support is provided for traffic management and public facility configuration, and urban dynamic management level is improved. The application of human gait recognition technology is not limited to security monitoring, but can also be extended to medical health and intelligent transportation and other life fields. For example, in medical health monitoring, the action state of the elderly and special groups can be monitored in real time, and a safer living environment is provided for residents.

[0003] Gait recognition technology is to use the gait information of human body in video image for identity recognition. Therefore, obtaining a reliable human gait is the basis of recognition. However, as the application scene of gait recognition technology becomes more and more complex, there are many interference factors in the complex scene. For example, the video is easily blocked by other human bodies, and is also easily blocked by static objects such as vehicles and trees on the road surface, causing the loss of bone nodes; the contour extraction is distorted due to large light changes in the complex scene; the extracted gait features are distorted due to the change of view angle. These interference factors may act together to bring difficulties to the gait recognition task, so that the gait recognition accuracy is low, and the gait recognition is difficult to land to the real complex scene. SUMMARY

[0004] The purpose of the present application is to provide a human gait recognition method and system in a complex scene, so as to solve the problem of low gait recognition accuracy caused by many interference factors in the complex scene.

[0005] The application provides a human gait recognition method in a complex scene, which comprises: sequentially performing light correction, human posture angle alignment and occlusion processing on extracted human gait data, and performing human gait recognition based on the gait data obtained after the light correction, human posture angle alignment and occlusion processing; the occlusion processing comprises occlusion domain estimation and occlusion reconstruction; the occlusion domain estimation is used for judging whether the gait data has an occlusion area, and the judgment standard of the occlusion area is that the number of occlusion points is greater than or equal to a minimum occlusion point threshold value, the connected skeleton nodes of the occlusion points are visible, and the occlusion points are invisible for N consecutive frames, or the skeleton nodes are not in an expected position for N consecutive frames; the occlusion reconstruction is used for performing optical flow compensation on pixel information of the occlusion points by using an optical flow vector calculated by using visible skeleton nodes around the occlusion area in the light corrected video image, or performing interpolation completion on the occlusion area based on pixel information of at least four visible skeleton nodes around the occlusion area.

[0006] Further, the light correction comprises: adjusting the contrast of a local region histogram of each block in the video image by using a CLAHE method, correcting the image brightness by using Gamma, and binarizing each pixel and its neighborhood by using LBP; the human posture angle alignment is used for projecting the gait data to a unified visual angle by using a rotation matrix of a camera used when the original video image is shot.

[0007] Further, the video image used to extract the human gait data is determined based on the following process: globally light equalization is performed on the original video image, if the target detection effect of the video image after the global light equalization is greater than that of the original video image, then the global angle alignment is performed on the video image after the global light equalization, otherwise the global angle alignment is directly performed on the original video image; it is judged whether the occlusion domain range of the video image after the global angle alignment is greater than the occlusion domain threshold, if yes, the frame-level occlusion processing is performed, it is judged whether the target detection effect of the video image after the frame-level occlusion processing is greater than that of the video image before the frame-level occlusion processing, if yes, the human gait data is extracted based on the video image after the frame-level occlusion processing, otherwise the video image with the best target detection effect before the frame-level occlusion processing is selected to extract the human gait data; if the video image after the global angle alignment is less than or equal to the occlusion domain threshold, it is judged whether the target detection effect of the video image after the global angle alignment is greater than that of the video image before the global angle alignment, if yes, the human gait data is extracted based on the video image after the global angle alignment, otherwise the video image with the best target detection effect before the global angle alignment is selected to extract the human gait data; the global light equalization is to perform local region histogram processing and Gamma correction on the video image; the global angle alignment is to rotate the video image as a whole according to the optimal angle, the optimal angle refers to the angle after which the symmetry of the video image is best; the frame-level occlusion processing refers to updating the video image by using the exponential moving average method.

[0008] Further, the extraction process of the human gait data is: an improved DeepSORT algorithm is used to process the video image after the basic correction, and the trajectory of each target ID is output, and the human gait data of each target ID is extracted from the video based on the trajectory of each target ID; the improved DeepSORT algorithm is obtained by modifying the Mahalanobis distance formula for calculating the spatial feature similarity in the original DeepSORT algorithm by using a displacement field, and the displacement field is obtained by calculating the dense optical flow of adjacent frames; the modified Mahalanobis distance formula is:

[0009]

[0010] wherein, d (1) (i,j) is the spatial deviation of the prediction box and the detection box, a d is a weight coefficient, d j is the detection distribution of the jth detection box, y i is the prediction distribution of the ith tracking trajectory, S i -1 is the inverse of the covariance matrix between the two distributions, p i is the target center pixel coordinate, D flow (p i ) is pi a displacement field of the body part.

[0011] Further, the human gait data includes a video image, a human contour image, and a human skeleton image; the process of human gait recognition is: feature extraction is performed on the video image to obtain human global gait features, the human global gait features including color texture; human contour features and human skeleton features are extracted based on the human contour image and the human skeleton image extracted from the video image; the human global gait features, the human contour features, and the human skeleton features are fused, human identity matching is performed using the fusion features obtained after the fusion, the identity to which the human target belongs is determined, and a gait recognition result is obtained.

[0012] Further, the human contour features are obtained by inputting the human contour image into a multi-space domain refining module; the multi-space domain refining module includes a spatial feature self-division unit and a channel feature self-attention unit; the spatial feature self-division unit is used for dividing the input human contour image into regions and generating feature maps of the regions, and the feature maps of the regions are spliced to output a first feature map; the channel feature self-attention unit is used for giving different weights to feature channels with different identity expressiveness in the first feature map to obtain the human contour features.

[0013] Further, the process of dividing the input human contour image into regions and generating feature maps of the regions is: the human contour image is divided along the human height direction to obtain a first region; each first region is divided along the human height direction to obtain a second region; a smooth mask of the first region is generated based on the distance between the human contour image and the center position of the first region, and the first region feature map is obtained by element-wise multiplication of the smooth mask of the first region and the human contour image; a mask of the second region is calculated, and the second region feature map is obtained by element-wise multiplication of the mask of the second region and the first region feature map in which the second region is located, and the second region feature map is the feature map of each region.

[0014] Further, the human body skeleton features include human body static biological features and human body dynamic gait features; the human body static biological features are obtained by inputting the human body contour image and the human body skeleton image into a human body static biological feature extraction unit, the human body static biological feature extraction unit is used for calculating the height and the proportion of the height to each limb / trunk according to the pixel height of the human body limbs / trunk in the human body skeleton image and the pixel height of the human body in the human body contour image, and the swing of the limbs / trunk; the height, the proportion of the height to each limb / trunk and the swing of the limbs / trunk are taken as the original biological vector, and the human body static biological features are obtained after the original biological vector is normalized and nonlinearly mapped; the human body dynamic gait features are obtained by inputting the human body skeleton image into a human body dynamic gait feature refining unit, the human body dynamic gait feature refining unit is used for constructing a spatial graph according to the skeleton node coordinates in each video image and the spatial connection relationship of the skeleton nodes, then using a graph convolutional neural network to extract the spatial features in the spatial graph, and then performing convolution processing on the spatial features to obtain the space-time features, and the human body dynamic gait features are obtained based on the space-time features.

[0015] Further, the feature fusion process is that the human body static biological features and the human body dynamic gait features are fused to obtain dynamic biological fusion features; a multi-head attention mechanism is used to capture the correlation of the human body contour features and the human body global gait features in different time scales, and the contour original image fusion features are output; after the dynamic biological fusion features and the contour original image fusion features are spliced, the spliced features are input into a fully connected layer for feature extraction to obtain the fusion features.

[0016] To solve the above technical problems, the application further provides a human gait recognition system in a complex scene, comprising a processor, the processor is used for executing computer program instructions to realize the steps in the human gait recognition method in the complex scene.

[0017] The beneficial effects of the above technical solutions are: the present application is an improved invention, which corrects the gait data obtained under a complex scene including illumination changes, occlusions and changes in viewing angle, and the correction methods include illumination correction, human posture angle alignment and occlusion processing. Since the illumination correction is conducive to angle alignment, the illumination correction and angle alignment are prerequisites for occlusion area estimation and reconstruction, therefore, the correction strategy is to sequentially perform illumination correction, human posture angle alignment and occlusion processing on the extracted gait data, and human gait recognition is performed based on the gait data obtained after the illumination correction, human posture angle alignment and occlusion processing. In order to accurately reconstruct the key occlusion area of the human body in the video image, a comprehensive and reliable occlusion area determination standard is set based on the connection relationship between the bone nodes, the human anatomy reference distance, the kinematics constraint and the time continuity requirement, and then the missing human body area is reconstructed based on the information of the visible pixels around the occlusion area using optical flow compensation or bilinear interpolation, thereby deeply correcting the extracted gait data, improving the accuracy of gait recognition under complex scenes, breaking through the adaptability problem of complex scenes, realizing high-precision gait identity recognition, and meeting the requirements of real scene security monitoring. BRIEF DESCRIPTION OF DRAWINGS

[0018] Figure 1 is a human gait recognition flowchart under a complex scene of the method embodiment of the present application;

[0019] Figure 2 is a basic correction flowchart of the method embodiment of the present application;

[0020] Figure 3 is a video image flowchart for determining the extracted gait data under multi-element interference suppression of the method embodiment of the present application;

[0021] Figure 4 is a gait data extraction flowchart of the method embodiment of the present application;

[0022] Figure 5 is a deep correction flowchart of human gait data of the method embodiment of the present application;

[0023] Figure 6 is a human gait recognition flowchart based on three features of the method embodiment of the present application;

[0024] Figure 7 is a gait recognition model network architecture diagram of the method embodiment of the present application;

[0025] Figure 8 is a multi-space domain refining module network structure schematic diagram of the method embodiment of the present application. DETAILED DESCRIPTION

[0026] In order to make the objects, technical solutions and advantages of the present application clearer, the specific embodiments of the present application will be further described in combination with the drawings.

[0027] The gait recognition method of the present application can improve the accuracy of gait recognition in complex scenes.

[0028] Method embodiments

[0029] The gait recognition method of the present application can improve the accuracy of gait recognition in complex scenes. Figure 1 The gait recognition method of the present application can improve the accuracy of gait recognition in complex scenes.

[0030] 1. Obtain video data.

[0031] A high-definition camera is used to capture the video of a person walking in a complex scene, and a video V ori = [I1, I2,..., In] is obtained, where In represents the i-th original video image. T i

[0032] In one embodiment, in order to ensure the stability of the gait features of the pedestrians, the original video image I is subjected to basic correction, as shown in Figure 2 The process of basic correction includes:

[0033] 1) Global light equalization: basic processing for the factors affecting uneven illumination, global light equalization is a local region histogram processing and Gamma correction on the video image.

[0034] The local region histogram H x,y (k) of each block B x,y of the original video image is calculated:

[0035]

[0036] Where I(i,j) is the pixel point value of the original video image I, and k is the gray level (0-255).

[0037] Then Gamma correction is used to deal with the non-linear light response, so as to adjust the image brightness, make the dark details more prominent, and avoid overexposure in high brightness area, and obtain the pixel value P' of each pixel point after correction:

[0038]

[0039] Where γ is the Gamma coefficient, which is adjusted as needed (between 0.8 and 2.2, and here it is preset to 1.2).​​

[0040] 2) Global angle alignment: Basic processing for angle misalignment factors, rotate the video image as a whole according to the optimal angle, and the optimal angle refers to the best symmetry of the video image after rotation.

[0041] Specifically, the current video frame is detected by a target detector (such as a YOLOv8 model) to preliminarily obtain a human anchor box, and the symmetry score of the human anchor box is calculated by randomly selecting the human anchor box:

[0042]

[0043] where X is the anchor box image after rotation, Y is the anchor box image after horizontal flip, μ X ,μ Y is the image mean, σ X ,σ Y is the image standard deviation, σ XY is the covariance, and C1 and C2 are stable constants to avoid calculation errors.

[0044] Then search for the optimal angle θ' of the current frame in the range of [-30°, 30°] with a step of 10°:

[0045] θ' = argmax S(θ)

[0046] Calculate 5 times in succession, and take the average of the obtained θ' to obtain θ * ,θ * , which is the selected angle for angle alignment, and the video image is rotated as a whole by θ * , completing the basic processing of angle alignment.

[0047] 3) Frame-level occlusion processing: Basic processing for human occlusion factors. When the occlusion domain range in the video image is greater than the occlusion domain threshold, use the exponential moving average (EMA) method to update the video image. For example, when the i-th track in the t-th frame of the video image is occluded, use the exponential moving average (EMA) method to directly update the state of the i-th track in the t-th frame:

[0048]

[0049] where is the predicted state of the i-th track in the t-th frame calculated by EMA, i.e. the state of the i-th track in the t-th frame after frame-level occlusion reconstruction (the state refers to the appearance embedding state, i.e. the appearance feature vector), is the predicted state of the i-th track in the t-1-th frame calculated by EMA, f i t is the actual state of the i-th track in the t-th frame (i.e. the current matching detection embedding), and αt This represents the momentum term. The EMA occlusion domain update strategy is suitable for reconstructing the occlusion domain before extracting the contour and bone nodes, which can effectively reduce time consumption.

[0050] In another embodiment, the video image used for extracting human gait data is determined based on the following process: Global illumination equalization is performed on the original video image. If the target detection effect of the globally illuminated video image is greater than that of the original video image, global angle alignment is performed on the globally illuminated video image; otherwise, global angle alignment is directly performed based on the original video image. It is then determined whether the occlusion region of the globally angle-aligned video image is greater than an occlusion region threshold. If it is greater, frame-level occlusion processing is performed. Finally, it is determined whether the target detection effect of the frame-level occlusion processed video image is greater than that of the video image before frame-level occlusion processing. If it is greater, human gait data is extracted based on the frame-level occlusion processed video image; otherwise, frame-level occlusion processing is selected. Human gait data is extracted from the video image with the best object detection performance. If the object detection performance of the globally angle-aligned video image is less than or equal to the occlusion region threshold, it is determined whether the object detection performance of the globally angle-aligned video image is greater than that of the video image before global angle alignment. If it is greater, human gait data is extracted based on the globally angle-aligned video image; otherwise, the video image with the best object detection performance before global angle alignment is selected for gait data extraction. Global illumination equalization involves local region histogram processing and Gamma correction of the video image. Global angle alignment involves rotating the video image as a whole according to the optimal angle, where the optimal angle refers to the video image with the best symmetry after rotation. Frame-level occlusion processing refers to updating the video image using an exponential moving average method. Object detection performance can be evaluated based on confidence levels.

[0051] Specifically, such as Figure 3 As shown, the original video image is first subjected to YOLO object detection to extract the confidence boxes of the target human body. The average confidence of all human bodies in the image is 'a'. A confidence threshold α is set. When a > α, it indicates that the current detection effect is good, and the next step of human body tracking and subsequent recognition operations can be performed. When a ≤ α, it indicates that the current detection effect is poor, and a series of basic correction operations are required. At this time, the basic correction stage in complex environments begins. Since global illumination equalization and global angle alignment are prerequisites for occlusion processing, and illumination equalization helps with angle alignment, the original video image first enters the illumination equalization stage. The average confidence of the video image after object detection after global illumination equalization is set to 'b'. When b ≤ a, it indicates that global illumination equalization has no effect, and the original video image is still used to enter the angle alignment stage. When b > a, it indicates that the global illumination equalization operation has a positive effect, and the original video image enters the angle alignment stage.

[0052] Given the occlusion domain threshold β, the occlusion region range of the globally angle-aligned image is compared with β, if less than or equal to β, the globally angle-aligned video image is used for target detection to obtain the target detection confidence c1, and c1 is compared with the maximum confidence max(a, b) of the video image in the previous stage (i.e. before global angle alignment), if c1≤max(a, b), the original video image or the video image after global light equalization corresponding to the maximum confidence before global angle alignment is used for target detection and enters the subsequent processing of gait recognition. If c1>max(a, b), it indicates that the global angle alignment operation plays a positive role, and the globally angle-aligned video image is used for target detection and enters the subsequent processing of gait recognition. If the occlusion region range of the globally angle-aligned image is greater than β, frame-level occlusion processing is performed, and the average confidence after target detection after frame-level occlusion processing is c2, if c2≤max(a, b), it indicates that global angle alignment and frame-level occlusion processing do not play a positive role, and the video image corresponding to the maximum confidence in the video image (i.e. original video image, globally light equalized video image and globally angle-aligned video image) before frame-level occlusion processing is used for target detection and enters the subsequent processing of gait recognition. If c2>max(a, b), it indicates that global angle alignment and frame-level occlusion processing play a positive role, and the video image after frame-level occlusion processing is the basic correction of the original video image in the complex scene.

[0053] 2. Extract human gait data.

[0054] The human gait data includes video images, human contour images and human skeleton images. The human gait data extraction process is as shown in Figure 4

[0055] 1) Target tracking.

[0056] The target can be tracked based on the DeepSORT algorithm. Preferably, the improved DeepSORT algorithm is used to track the target, and the trajectory of each target ID is output.

[0057] The improved DeepSORT algorithm is obtained by modifying the Mahalanobis distance formula for calculating spatial feature similarity in the original DeepSORT algorithm using displacement field. The improved DeepSORT algorithm includes human target detection, Kalman filter prediction, similarity calculation, data association and trajectory generation, which will be introduced in detail below.

[0058] ① Human target detection.

[0059] ​Object detection is a crucial step in the data standardization phase of gait recognition tasks. Accurate and real-time detection of human targets is fundamental for subsequent gait feature extraction and identity recognition. The YOLOv8 model can be used to detect bounding boxes for all human figures in the current video image; the expression is:

[0060]

[0061] in, The bounding box coordinates and confidence scores of the human target predicted by the YOLOv8 model, with a confidence threshold of 0.5; F neck For the features output by the Neck module of YOLOv8, W head Features output by the head module of YOLOv8.

[0062] Alternatively, FASTER R-CNN or the latest YOLO model can be used to detect human bounding boxes in the current video image.

[0063] ② Kalman filter prediction.

[0064] In Kalman filter prediction, the Kalman filter is used to predict the state of the target in the current video image based on the state of the previous frame, and the prediction result is the prediction box.

[0065] ③ Similarity calculation.

[0066] Similarity includes the similarity of spatial features and the similarity of appearance features. Based on the similarity calculation, a posterior analysis is performed on the Kalman filter prediction results.

[0067] The similarity of spatial features is calculated using the Mahalanobis distance formula. This invention introduces optical flow displacement compensation to modify the Mahalanobis distance formula, based on the similarity calculation in the original DeepSORT algorithm. Specifically, the Farneback algorithm is used to calculate the dense optical flow of adjacent frames, obtaining the displacement field D. flow The formula for calculating the displacement field is:

[0068]

[0069] Among them, D flow (p i ) represents p i Displacement field at that location, p i The horizontal displacement components (x direction) of the preceding and following frames. p i The vertical displacement component (y direction) of the preceding and following frames, p i This represents the pixel coordinates of the target center.

[0070] The displacement field is fused with the predicted displacement in the prediction box of the Kalman filter of the original DeepSORT algorithm to obtain a revised Mahalanobis distance formula:

[0071]

[0072] where i represents the i-th pedestrian target being tracked, that is, the i-th existing tracking trajectory, j represents the j-th detection box in the current frame, d (1) is the Mahalanobis distance between the prediction box and the detection box, represents the spatial deviation between the prediction box and the detection box, a d is a weight coefficient, d j is the detection distribution of the j-th detection box, y i is the prediction distribution of the i-th tracking trajectory, is the inverse of the covariance matrix between the two distributions. The calculated Mahalanobis distance is the deviation between the prediction box and the detection box, and the uncertainty of the tracker state is estimated according to the deviation. a d It can be set to 0.7, or it can be customized according to actual conditions.

[0073] The appearance feature is obtained by extracting the prediction result through a feature extraction network model. The feature extraction network model uses the ResNet-50 model of the Fast-ReID library, and uses the ResNet-50 model (ImageNet pre-training) of the Fast-ReID library to generate a 128-dimensional feature vector as the appearance descriptor for appearance feature similarity calculation. ResNet-50 is used to extract 128-dimensional high-discriminative features, and the similarity is calculated by cosine distance, which overcomes the sensitivity problem of traditional CNN to clothing changes.

[0074] The similarity of the appearance feature is calculated using the cosine distance formula, and the calculation formula is:

[0075]

[0076] where d (2) (i,j) is the cosine distance between the prediction box and the detection box, representing the appearance deviation of the cosine distance between the prediction box and the detection box, R i represents the appearance of a plurality of objects related to the i-th trajectory, represents the appearance feature vector of the j-th detection box, represents the k-th feature vector in the appearance feature set of the i-th trajectory.

[0077] ④Data association and trajectory generation.

[0078] The Mahalanobis distance and the cosine distance correspond to a spatial gating function and an appearance gating function If the corresponding distance is less than a predefined threshold, then 1, otherwise 0. Only when both the spatial gating function and the appearance gating function are equal to 1, the gating matrix is 1, which is considered as a valid match, otherwise 0, which is considered as an invalid match. A joint cost matrix is constructed using the two distances:

[0079] c i,j = λd (1) (i,j) + (1 - λ)d (2) (i,j)

[0080] where c i,j is the joint cost matrix and λ is the Mahalanobis distance corresponding weight. In a new frame, the joint cost matrix and the gating matrix are used to associate new detections with existing tracks. Whenever a new detection is successfully associated with an existing track, the detection is deleted and added to the track, at which time the non-association degree of the track is zero.

[0081] In the tracking process, the confirmed tracker will store the corresponding detection feature mapping into the feature set list every time the feature matching is completed. The feature set list is updated every time the feature matching is completed. The feature set plays a role in ID association, and when the similarity of the features exceeds a certain threshold, it is considered to be a successful association. The association result greatly affects whether the target ID is stable, thereby affecting the tracking result. The exponential moving average method is used to directly update the state of the track in the next frame in the improved DeepSORT algorithm, and the EMA update strategy can reduce the time consumption.

[0082] 2) Human contour image extraction.

[0083] The human contour image is a binary contour image obtained by performing edge detection and morphological operation on the video frames corresponding to the track of the target ID in turn.

[0084] Edge detection can use the Canny edge detection algorithm, which uses methods such as Gaussian filtering, gradient calculation, and non-maximum suppression to obtain accurate and clear edges. First, a Gaussian filter is used to smooth the human image region cut out in the video frame corresponding to the track of the target ID, and the process is as follows:

[0085]

[0086] where x G and y G are the coordinates of a certain position of the template relative to the center of the template, G(x G , y G ) represents the Gaussian function, σ represents the standard deviation of the filter, and π is the circular constant. Gaussian filtering helps to blur the image and remove small fluctuations and noise in the image. The template refers to a mask, which here refers to the local region processed by the Gaussian filter.

[0087] Then the gradient of each pixel in the video image is calculated by Sobel operator to obtain the gradient image. The gradient includes the direction and intensity of the gradient, and the gradient reflects the change of the pixel value in the image. The size of the gradient represents the intensity of the edge. The gradient intensity J(x, y) of the pixel point located in the image (x, y) is:

[0088]

[0089] The direction of the gradient is the normal direction of the edge. The gradient direction θ(x, y) of the pixel point is:

[0090]

[0091] Wherein, G x (x, y) is the image gradient of the pixel point (x, y) in the horizontal direction (x-axis), and G y (x, y) is the image gradient of the pixel point (x, y) in the vertical direction (y-axis).

[0092] Then non-maximum suppression is performed on the gradient image. That is, the gradient intensity of each pixel is subjected to non-maximum suppression. First, all the pixel points are traversed. If the gradient intensity of the pixel point is greater than the first gradient intensity threshold and greater than or equal to the gradient intensity of the two pixel points along the gradient direction, the gradient intensity value of the pixel point is retained. On the contrary, if the gradient intensity of one of the two pixel points along the gradient direction of a certain pixel point is greater than the pixel point, the gradient of the pixel point is set to 0, so as to retain the local maximum value on the gradient image and suppress the response of the non-edge region. The approximation of the maximum value J'(x, y) is as follows:

[0093]

[0094] J Low is the first gradient intensity threshold, δ x and δ y are the offset values of the gradient direction.

[0095] A double threshold is used to mark the edge pixels, and the gradient image is divided into three regions of strong edge, weak edge and non-edge. The strong edge pixels and the weak edge pixels adjacent to the strong edge pixels are retained to obtain the preliminary human body contour image. The final value of the image (x, y) pixel point is:

[0096]

[0097] Wherein, J HighStrong, Weak, Non represent strong edge, weak edge and non-edge respectively. Finally, complete edges are formed by connecting strong edge pixels. Weak edge pixels are only retained when they are adjacent to strong edge pixels, otherwise they are considered as noise. J is determined by experiment Low High After the selection of J, a preliminary human contour image is obtained, and the contour texture map of the human image is displayed.

[0098] The preliminary human contour image obtained is post-processed by morphological operation: first, the preliminary human contour image foreground region obtained by the Canny edge detection is expanded by dilation operation to fill the gap of the contour edge, and then the closed operation operation of first dilation and then erosion is adopted to fill the small holes in the foreground human body, and the refined binary contour image is extracted.

[0099] 3) Human skeleton image extraction.

[0100] The human skeleton image is a skeleton image including each skeleton node and skeleton line generated by processing the binary contour image. The processing process is: after the binary contour image is normalized, it is input into the HRNet model, the model generates a heat map of the key skeleton node, then the heat map is analyzed to obtain the corresponding skeleton node coordinates, and then the skeleton line is generated according to the key skeleton node coordinates, and the skeleton image including each skeleton line is obtained.

[0101] Specifically, first, the binary contour image I ∈ R H×W After normalization, it is input into the HRNet model, and the features are extracted by the multi-resolution parallel branch of the HRNet model:

[0102] F (k) k = 1, 2,..., K

[0103] Where F (k) represents the feature map of the kth branch, K is the total number of branches, and Conv represents convolution. The high-resolution branch continuously performs convolution operation at each stage to maintain the detailed expression of the feature map.

[0104] The feature maps of different resolution branches are interactively integrated by the multi-resolution fusion module of the HRNet model:

[0105] F fused = Fuse(F (1) , F (2) ,..., F (K) )

[0106] Where F fused ​Fuse represents the feature fusion operation. The feature fusion operation includes upsampling (such as nearest neighbor interpolation) and downsampling (such as convolutional dimension reduction), ensuring that the feature maps of all branches can be aligned. Based on the fused features, key skeleton nodes are obtained. For each key skeleton node (such as human joint nodes: head, shoulders, and knees, etc.), a heat map H i :

[0107]

[0108] where x, y are the pixel coordinates in the heat map; x i , y i are the coordinates of the i-th key skeleton node; σ1 is the standard deviation of the Gaussian distribution, used to control the concentration of the heat map.

[0109] The maximum position of the heat map H i is analyzed to determine the key skeleton node coordinates:

[0110] (x i ,y i ) = argmax H i (x,y)

[0111] At the same time, the HRNet model uses mean square error (MSE) as the loss function, uses source video data as supervision, and measures the difference L between the predicted heat map and the real heat map H i :

[0112]

[0113] where N is the number of key skeleton nodes. According to the key skeleton node coordinates, the skeleton lines are generated, and the general order is: head to shoulder: nose-left shoulder-right shoulder. Upper limbs: left shoulder-left elbow-left wrist; right shoulder-right elbow-right wrist. Lower limbs: left hip-left knee-left ankle; right hip-right knee-right ankle. According to the above connection mode, the original skeleton graph of each frame is generated.

[0114] 3. Deep correction of extracted human gait data.

[0115] As shown in Figure 5 , the extracted human gait data is sequentially subjected to illumination correction, human posture angle alignment, and occlusion processing to achieve deep correction.

[0116] 1) Illumination correction: deep processing of factors affecting uneven illumination. It includes: using the limited contrast adaptive histogram equalization (CLAHE) method to adjust the contrast of the local region histogram of each block in the video image, and obtaining the contrast-adjusted local region histogram H'x,y (k) to limit the local contrast enhancement range to make the illumination distribution uniform:

[0117]

[0118] If no base correction is performed, a Gamma correction is also used to simulate the nonlinear response of the human eye to dark areas, adjust the image brightness, so that the dark details are more prominent, while avoiding overexposure in high brightness areas.

[0119] Then compare the pixels with their surrounding neighborhood pixels through the LBP texture descriptor to generate illumination-invariant features. LBP binarizes each pixel with its neighborhood:

[0120]

[0121] wherein, g p is the gray value of the neighborhood pixel, g c is the gray value of the center pixel.

[0122] 2) Human posture angle alignment: For the deep processing of angle misalignment influencing factors, a geometric transformation method is used to map gait data of different angles to a unified perspective, solving the distortion interference of oblique perspective. For example, use the rotation matrix of the camera used to shoot the original video image to project the gait data to a unified perspective, that is, use the rotation matrix to project the 3D joint nodes to the sagittal plane, eliminating the limb shortening effect caused by different perspectives.

[0123] Specifically, first determine the intrinsic matrix and extrinsic matrix of the camera used to shoot the original video image. Assume that the intrinsic matrix K and the extrinsic matrix [R|t] of the camera:

[0124]

[0125] wherein, f x and f y are focal length parameters, c x and c y are principal point coordinates, R is a rotation matrix describing the direction of the camera, and t is a translation vector describing the position of the camera.

[0126] The human body skeleton nodes estimated by the HRNet model are used to perform perspective transformation according to these human body skeleton nodes to unify the gait data to a standard perspective. Given a source image I and a target image I', let the coordinates of the human body skeleton nodes in the image I be (x, y), and through I'(x', y') = I(x, y) perspective transformation, its three-dimensional coordinates in the world coordinate system are P world =(X,Y,Z) T . Then there is a standard transformation basic equation:

[0127] [u,v,1] T = H [X,Y,Z,1] T

[0128] where (u,v) is the image coordinate in the standard view, H is the transformation matrix, H = K · [R | t], usually a 3x4 matrix, which is estimated by the known four points.

[0129] Then restore to the world coordinate (absolute coordinate):

[0130] [X',Y',Z'] T = R -1 · (P world -t)

[0131] Map the world coordinate to the standard view, get the target image I' = [u',v',1] T :

[0132] [u',v',1] T = K · [X',Y',Z'] T

[0133] 3) Human key occlusion reconstruction: Deep processing of human key part occlusion influencing factors, including occlusion domain estimation and occlusion reconstruction.

[0134] In the scene with low recognition speed requirement and high accuracy requirement (such as criminal investigation and surveillance monitoring video), high-fidelity skeleton node is needed to assist identity recognition, and occlusion domain estimation and reliable reconstruction are needed.

[0135] ① Occlusion domain estimation.

[0136] The judgment standard of the occlusion area is that the number of occlusion points is greater than or equal to the minimum occlusion point threshold, and the connected skeleton nodes of the occlusion points are visible, and the occlusion points are not visible for N consecutive frames, or the skeleton nodes are not in the expected position for N consecutive frames. Invisible means that the detection confidence of the skeleton node is less than the threshold.

[0137] Specifically, let the skeleton node set K g = {k1,k2,...,k n}, the detection confidence c i of each skeleton node k i ∈ [0,1]. Given threshold T conf (usually 0.5), so as to judge whether the skeleton node is reliably detected:

[0138]

[0139] where v i represents the i-th skeleton node k ithe visible state (1 = visible, 0 = occluded) of the skeleton nodes. Below a threshold T conf skeleton nodes are marked as "invisible".

[0140] The connection relationship between skeleton nodes (e.g. skeleton structure) is analyzed. If some connected skeleton nodes are not detected at the same time, it can be inferred that the part is likely to be occluded. Define the skeleton as a graph structure G = (K g , E), where E is the set of skeleton connection edges (e.g. elbow-wrist connection). The reliability function s ij of each edge e ij ∈ E is:

[0141]

[0142] where p i , p j are the positions of the connected skeleton node i and skeleton node j, d ref is the human anatomy reference distance (e.g. the length of the forearm of an adult ≈ 45 cm), and σ is the distance tolerance parameter (which can be 0.2).

[0143] If a certain skeleton node (e.g. wrist) is not detected, it is determined to be a suspected occlusion area O. For the suspected occlusion area O, the minimum number of occlusion points condition must be met:

[0144]

[0145] θ occ is the minimum number of occlusion points threshold (e.g. wrist + elbow ≥ 2), and the time continuity condition To must also be met:

[0146]

[0147] The time continuity condition means that the skeleton node connected to the occlusion point (e.g. elbow) is normally visible and the key node is not visible for N consecutive frames, N = Δt + 1.

[0148] Abnormal points are determined by checking the relative positions of the skeleton nodes in space, for example, the shoulder and head are not in the expected position. Define the expected position function (based on kinematic constraints):

[0149]

[0150] where represents the expected position of the i-th skeleton node in the t-th frame, Δp ji represents the joint relative displacement (e.g. the length direction displacement of the wrist relative to the elbow), represents the actual position of the j-th skeleton node in the t-th frame, represents the visible state of the j-th adjacent skeleton node in the t-th frame, N irepresents a set of adjacent bone nodes directly connected with the ith joint node.

[0151] The deviation degree of the ith bone node in the tth frame is expected position:

[0152]

[0153] Wherein, k represents the maximum tolerance distance threshold parameter, usually 1.5; L ref represents the length of the limb associated with the current bone node.

[0154] If the consecutive N frames are not in the expected position, the area is also determined as occlusion. Not in the expected position means T δ represents the deviation threshold.

[0155] Therefore, the total determination condition of the occlusion area is:

[0156]

[0157] ② Occlusion reconstruction.

[0158] After detecting the occlusion area, the occlusion area is reconstructed by using the optical flow method, which needs the video image after global light equalization and depends on the light optimization operation in the early stage.

[0159] That is, the pixel information of the occlusion point is compensated by the optical flow vector calculated by using the visible bone nodes (visible pixels) around the occlusion area in the video image after global light equalization:

[0160] I recover (x,y,t)≈I(x-u,y-v,t+t′)

[0161] Wherein, I recover (x,y,t) represents the pixel of the tth frame after occlusion reconstruction, x and y represent the pixel position of the occlusion point in the tth frame, (u,v) represents the optical flow vector, and t' represents the time increment.

[0162] Or based on the pixel information of at least 4 visible bone nodes around the occlusion area, the occlusion area is interpolated and completed, such as using the pixel information of 4 visible bone nodes around the occlusion area to complete the occlusion area by bilinear interpolation:

[0163]

[0164] Wherein ω i is the weight of the adjacent ith visible bone node, I(x i ,y i ,t′) represents the pixel in the reference frame (previous frame) with coordinates (x i ,y iThe pixel gray value of the image is obtained by bilinear interpolation. When the occlusion area is too large, a bilinear interpolation completion operation can obtain a smoother reconstruction effect.

[0165] 4. Perform human gait recognition based on the gait data obtained after illumination correction, human posture angle alignment, and occlusion processing.

[0166] When performing gait recognition, gait recognition can be performed based on human contour features and human skeleton features.

[0167] In another embodiment, feature extraction is performed on the video image to obtain human global gait features. The human global gait features include color texture, such as RGB clothing color blocks and contour RGB texture when the shoulders swing, etc. The human contour features, human skeleton features, and human global gait features extracted from the video image are subjected to multi-feature fusion, and the multi-fused features are used for gait recognition. As shown in Figure 6 , the steps include:

[0168] 1) Extract human global gait features.

[0169] The human global gait features are obtained by using a pruned Vision Transformer (Pruned ViT) model to extract features from the video image, and the pruned Vision Transformer model embeds the time position coding and the spatial position coding after linear projection.

[0170] Specifically, as shown in Figure 7 , the pruned Vision Transformer is a variant of the Transformer on visual images, which divides each frame of a video image with a size of HxWxC (heightxwidthxchannel number) into image blocks with a size of Pxp, each image block is expanded into a one-dimensional vector, and each frame is divided into N=H·W / P 2 image blocks, and the dimension of each image block is P 2 xC, let xij represent the jth image block feature of the ith frame, and the entire video is represented as:

[0171] X={x i,j |i∈[1,T],j∈[1,N]}

[0172] Each image block feature is mapped to a low-dimensional space with a dimension of d through linear projection:

[0173]

[0174] After projection, the feature representation matrix of each image block of each frame is obtained:

[0175]

[0176] wherein represents the spatial position encoding of the image block, represents the temporal position encoding of the frame in the sequence. The visual problem is converted into a Seq2Seq problem through the above processing.

[0177] The features of each frame of video image are input to the multi-head self-attention mechanism of the Vision Transformer model, and the input dimension is z e R (N+1)×d , the input is mapped to Query Q, Key K, Value V, and the dimension becomes h after N attention heads. The results after calculation of multiple attention heads are directly concatenated, and then the concatenated results are fused through a learnable parameter W O to obtain the final result output, which is also the input of the next Block. The calculation of the self-attention mechanism can be represented as:

[0178]

[0179] wherein Q i , K i , V i are the query vector, key vector and value vector of the attention head i respectively. Finally, the multi-head self-attention (MSA) and the multi-layer perceptron combination become the output of each Encoder layer, and the output is as follows:

[0180] Z l = Z l ′*W mlp + Z l ′

[0181] Z l is the output of the current Encoder, and is finally input into the next Transformer layer. Finally, the Transformer layer after pruning obtains the first token of the last Encoder, which is the human body gait feature Z ori .

[0182] 2) Extract human contour features.

[0183] The human contour feature is obtained by inputting the human contour image (Sils) into the multi-spatial domain refinement module (Multi-spatial Domain Refinement Module). The human contour image can also be preprocessed by Conv3 convolution before being input into the multi-spatial domain refinement module, and then input into the multi-spatial domain refinement module, so as to further improve the accuracy of human contour feature extraction. For example,Figure 8 As shown, the multi-space domain refining module includes a spatial feature self-partitioning unit (Spatial Feature Self-Partitioning Encoder) and a channel feature self-attention unit (Channel Feature Self-Attention Encoder).

[0184] The spatial feature self-partitioning unit is configured to divide the input human contour image into regions and generate feature maps of the regions, and output a first feature map after splicing the features of the regions. In an embodiment, the human contour image is divided along the height direction of the human body to obtain a first region; each first region is divided along the height direction of the human body to obtain a second region; a smooth mask of the first region is generated based on the distance between the human contour image and the center position of the first region, and the smooth mask of the first region is multiplied with the human contour image element by element to obtain a first region feature map; a mask of the second region is calculated, and the mask of the second region is multiplied with the first region feature map element by element to obtain a second region feature map, which is the feature map of each region.

[0185] Specifically, the spatial feature self-partitioning unit divides the input human contour image X in the spatial height dimension (H) into multiple regions using learnable partition parameters. Generally, two divisions are used, and m1-1 partition parameters are used to divide the input human contour image (i.e., first partitioning) to obtain m1 regions (denoted as first regions), such as dividing the upper body and the lower body, where m1 = 2; each first region is divided (i.e., second partitioning) to obtain m2 regions (denoted as second regions) in each first region, such as subdividing the front and rear halves of the limbs, where m2 = 4, so that the input human contour image is finally divided into m = m1 x m2 regions. These partition parameters can be adaptively adjusted during the training process. By dynamically adjusting the partition parameters, the body shape and posture changes of different targets are effectively adapted. The generation process of the feature map of each region is as follows:

[0186] S1: After initializing, sorting, boundary constraint, and normalization of the partition parameters, the center positions of the first regions are calculated.

[0187] Before the first partitioning, m1-1 partition parameters are first initialized. To avoid boundary overflow, the partition parameter p should satisfy the constraint:

[0188]

[0189] The first region positions are generated by sorting, boundary constraint, and normalization:

[0190] p norm= softmax(sort(clamp(p, 0.1, 0.9))) · cumsum(0)

[0191] where clamp means to limit each element of the division parameter p in the interval [0.1, 0.9], sort means to sort, softmax is used to normalize the sorted division parameter, and cumsum is used to calculate the cumulative sum to generate a monotonically increasing vector.

[0192] Then the center position of the first region is calculated:

[0193]

[0194] c i represents the i-th first region center position, and the position refers to the position in the height dimension.

[0195] S2: Generate a smooth mask of the first region based on the distance between the human contour image and the first region center position.

[0196] First, calculate the distance from each position g ∈ [0, 1] in the spatial height dimension (H) of the human contour image X to the first region center position:

[0197] d i (g) = |g - c i |

[0198] Then use the Sigmoid function to create a smooth mask of the first region to control the sharpness of the region boundary, and the expression of the smooth mask of the first region is:

[0199]

[0200] where d i (g) represents the distance from each position g in the spatial height dimension of the human contour image to the i-th first region center position; m i (g) represents the smooth mask of the first region; τ is the temperature coefficient, which is used to control the smoothness of the region boundary. When τ = 0.1, the boundary is sharp (suitable for laboratory standard environment); when τ = 0.3, the boundary is smooth (suitable for outdoor complex scenes). τ can be set to 0.2 to balance indoor and outdoor scenes. The sum of all smooth masks of the first region is 1.

[0201] S3: The smooth mask of the first region is multiplied element by element with the human contour image to obtain the first region feature map.

[0202] For an input image X ∈ R N×C×T×H×W , apply the corresponding smooth mask to multiply element by element to generate m1 coarse-grained region features, i.e. the first region feature map:

[0203] Xi = X i (g)

[0204] S4: According to the above manner, the mask m of the second region in the second division is calculated according to the distance between the first region feature map and the second region center position and the Sigmoid function j (g), for each first region feature map, the mask of the second region is multiplied with the first region feature map where it is located to obtain the second region feature map, and the second region feature map is the feature map of each region:

[0205] X ij = X i ⊙m j (g local )for j = 1 to m2

[0206] wherein X ij represents the jth second region feature map in the ith first region, X i represents the ith first region feature map, m j represents the mask of the jth second region, and g local is the local normalized height coordinate of the first region feature map. Finally, m second region feature maps are obtained.

[0207] S5: The feature maps of each region are spliced to output the first feature map X', or the feature maps of each region are respectively subjected to convolution (such as three-dimensional convolution) operation, and then the convolution results are spliced to output the first feature map X'.

[0208] The dimension of the first feature map is the same as the input, but the height information is refined.

[0209] The channel feature self-attention unit is used to give different weights to the feature channels with different identity expressiveness in the first feature map to obtain the human body contour feature. The channel feature self-attention unit includes a dimension reduction convolution, an attention generation convolution, a feature enhancement layer, at least two independent three-dimensional convolutions, and a feature fusion layer.

[0210] The first feature map input into the channel feature self-attention unit is sequentially subjected to dimension reduction convolution for dimension reduction and attention generation convolution for attention processing to generate an attention map A:

[0211] A = softmax(W2 LeakyRelu (W1 X'))

[0212] wherein W1 e R C×C / 4 represents the dimension reduction convolution, W2 e R C / 4×m represents the attention generation convolution, and LeakyReLU represents an activation function.

[0213] The feature enhancement layer adopts a channel broadcasting mechanism to enhance features, and outputs a weighted feature map after element-wise multiplication of the attention map and the first feature map

[0214]

[0215] wherein represents element-wise multiplication.

[0216] The weighted feature map is input into independent convolution for feature extraction, and each feature extraction result is input into a feature fusion layer for merging to output a human contour feature. In this embodiment, at least two independent three-dimensional convolutions (with a uniform kernel size of 3x3x3) are used for feature extraction, and finally the outputs of all three-dimensional convolutions are merged to obtain the human contour feature Z:

[0217]

[0218] wherein f represents the number of three-dimensional convolutions.

[0219] The channel feature self-attention module solves the problem that the traditional convolution cannot adaptively focus on important channels in gait recognition through a two-stage attention mechanism, simulates the human visual attention mechanism, further focuses on discriminative channel features on the basis of spatial division, adaptively selects features, and effectively suppresses background interference.

[0220] 3) Extract human skeleton features using a biometric feature unwinding module (Multi-spatial Domain Refinement Module).

[0221] The human skeleton features include human static biometric features and human dynamic gait features. The biometric feature unwinding module includes a human static biometric feature extraction unit and a human dynamic gait feature refinement unit, which are used to realize the decoupling of motion patterns and body structure through biomechanical constraints.

[0222] The human static biometric features are obtained by inputting the human contour image and the human skeleton image into the human static biometric feature extraction unit (Static Biometric Feature Extraction) for processing. The present application models the inherent properties of the human body through anatomical constraints, learns the basic linear relationship (height-proportion of each limb / trunk) and complex nonlinear relationship (amplitude of limb / trunk) of the human body. The human static biometric feature extraction unit is used to calculate the height and the proportion of each limb / trunk, as well as the amplitude of the limb / trunk, according to the pixel height occupied by the human body in the human skeleton image and the pixel height occupied by the human body in the human contour image.

[0223] Calculate the proportion of height and each limb

[0224]

[0225] where h represents the height (the height of the human body in the human body contour image) to determine the reference scale of the body, and l represents the length of the human body (the length of the human body in the human body contour image) to determine the reference scale of the body. i is the height of the human body (the Euclidean distance of the key points of the human body in the human body skeleton node image).

[0226] The static biological features are represented as original biological vectors. The height, the ratio of the height to each limb / trunk, and the swing of the limb / trunk are taken as the original biological vectors. For example, the original biological vector b raw is represented as:

[0227]

[0228] The original biological vectors are normalized to eliminate population differences:

[0229]

[0230] where μ bio and σ bio are the mean and standard deviation of the biological features of the training set, respectively.

[0231] Then, a two-layer fully connected network is used for nonlinear mapping to realize cross-population alignment, so that the biological features of different populations such as the elderly and children are comparable in the same space, and the static biological features of the human body are obtained:

[0232] u1=LeakyRelu(W1b norm +b1)

[0233] b embed =BatchNorm(W2u1+b2)

[0234] where b norm ∈R k+1 (k is the number of limbs), and b embed is obtained, which is the final embedding feature vector of the static biological features of the human body.

[0235] The human dynamic gait feature refining unit is used to construct a spatial graph according to the bone node coordinates in each video image and the spatial connection relationship of the bone nodes, then use a graph convolutional neural network to extract spatial features in the spatial graph, and then perform convolution processing on the spatial features to obtain spatiotemporal features, and obtain the human dynamic gait features based on the spatiotemporal features.

[0236] First, the human body bone gait data is represented as a time sequence of three-dimensional coordinate sequences, denoted as B={b1, b2,...,b T}, where b t ∈R V×Ccoordinates of V joints in the t-th frame (C=2 corresponds to XY axis, if it is a depth camera to extract depth image C=3), the overall input tensor dimension is T x V x C. Based on the human joint anatomy structure, an adjacency matrix A e {0, 1} is defined V×V , wherein A ij = 1 represents that there is a biomechanical connection between joints i and j (such as elbow-wrist joint), and A ij = 0 represents that it is a non-biomechanical connection point (such as neck-ankle joint), which is used to describe the spatial correlation between joints. The adjacency matrix is normalized to construct a spatial graph:

[0237]

[0238] , wherein D is a degree matrix.

[0239] A simplified ST-GCN graph convolution network model with hierarchical structure is used to fuse space-time features to process the spatial graph, extract the spatial dependence between joints, and the convolution kernel is a learnable weight matrix W t e R L×D (D is the number of output channels), to obtain spatial features H B :

[0240]

[0241] , wherein B is an input bone sequence, represents a spatio-temporal graph, and the local aggregation of spatial features is realized by sharing the adjacency matrix .

[0242] The spatial features are input into the causal convolution or dilated convolution to process the time dimension, the convolution kernel W t e R L×D , the bias term is b t , (L is the window length), the dynamic change pattern of the gait sequence is extracted, and the spatio-temporal features H t are obtained:

[0243] H t = relu(H B W t + b t )

[0244] In the simplified design, multiple layers of time convolution can be omitted, and only a single layer is retained to balance efficiency and feature expression ability.

[0245] The spatio-temporal features can be used as human dynamic gait features. Preferably, key spatio-temporal features related to gait balance are screened from the spatio-temporal features, and the key spatio-temporal features related to gait balance are used as human dynamic gait features.

[0246] A constraint mask matrix is defined:

[0247]

[0248] where M is a constraint mask matrix, is the left joint coordinate at time t, is the right joint coordinate at time t, d max is the maximum anatomical distance, which is specifically taken as 1.5 times the hip joint distance in human body image, and M ∈ [0, 1] D×K (K is the target feature dimension).

[0249] The key spatio-temporal features related to gait balance are obtained by inputting the features screened by a learnable weight matrix and the mask operation using a constraint mask matrix into a convolution layer for processing. This step can refine the features, enhance the key information, and suppress the redundancy.

[0250] F masked = H t W b ⊙M

[0251] b ref = sigmod(W O F masked + b O ) W O and b O are the convolution layer parameters, b ref is the human dynamic gait feature. The convolution layer is a fully connected layer or a lightweight convolution, which is used to map the screened features F masked to the target dimension F, and integrate the spatio-temporal information.

[0252] 4) The synesthesia integration module is used to fuse the human global gait features, human contour features, and human skeleton features.

[0253] The synesthesia integration module (Synesthesia Integration Module) includes a static-dynamic feature interaction mechanism, a multi-modal feature fusion, and a fully connected layer.

[0254] The static-dynamic feature interaction mechanism (Static-Dynamic Feature Interaction Decoder) is used to fuse the human static biological features and the human dynamic gait features to obtain dynamic biological fusion features.

[0255] The human static biological features b embed and the human dynamic gait features b ref are aligned to the same dimensional space to obtain the aligned human static biological features b′ embed ; and the aligned human static biological features b′ embed are input into the fully connected layer for processing.The human dynamic gait feature b ref The joint feature vector is obtained by concatenating the human static biometric feature b and the human dynamic gait feature b' in the dimension: ref embed .

[0256] The joint feature vector is input into the attention mechanism to obtain the attention weight a' corresponding to the aligned human static biometric feature and the human dynamic gait feature:

[0257] a' = sigmod(W ref [b ref ; b' embed ] + b ref_bias )

[0258] Wherein, W ref and b ref_bias represent the attention mechanism parameters, and the sigmod activation function limits the output attention weight to the interval [0, 1].

[0259] The weighted sum of the aligned human static biometric feature, the human dynamic gait feature, and the corresponding attention weight is the dynamic biometric fusion feature F dyn-bio :

[0260] F dyn-bio = a' · b ref + (1-a')b' embed

[0261] If the human biometric static feature (such as height) is strongly related to the current dynamic mode (such as large stride), then a' is increased to strengthen the static constraint; otherwise, a' is weakened. This mechanism enhances the discriminability of the dynamic mode through biometric constraints.

[0262] Multimodal feature fusion (Multimodal FeatureFusion Decoder).

[0263] The multi-head attention mechanism is used to capture the correlation between the human contour feature and the human global gait feature in different time scales, and the contour original image fusion feature F st is output.

[0264] The multi-head attention mechanism is used to capture the correlation between the (local contour) and the (global original image) in different time scales:

[0265] Q = F dyn-bio W q , K = [Z; Z ori ]W k , V = [Z; Z ori ]W v

[0266] ​

[0267] The dynamic biological fusion feature and the contour prototype image fusion feature are spliced and input to a full connection layer for feature extraction to obtain fusion feature D final .

[0268] D final = W f ·relu(cat(F st ,F dyn-bio ))+b f

[0269] The full connection layer can adopt multiple layers, for example, the features after splicing of the dynamic biological fusion feature and the contour prototype image fusion feature are processed by two full connection layers and ReLU activation functions to extract high-order discriminative features.

[0270] 5) The fusion feature obtained by feature fusion is used for human identity matching to determine the identity to which the human target belongs, and a gait recognition result is obtained.

[0271] In the identity recognition stage of gait recognition, a metric learning framework is used to construct an identity matching system, and a feature database is constructed:

[0272] D = {(f k ,ID k |k = 1, 2,..., K)}

[0273] Wherein, f k is the D final mean of the kth recognizable sample identity learned by the training sample, ID k is the corresponding identity label, and K is the total number of recognizable sample identities.

[0274] The similarity is calculated by using a similarity calculation function:

[0275]

[0276] Wherein, q is the fusion feature of the target to be identified, w t ∈[0, 1] is a time-varying weight (based on the gait cycle phase dynamic adjustment), and ε is a minimum value to avoid division by zero error.

[0277] According to the similarity, preliminary screening is performed (filtered by threshold):

[0278] C cand = {k | s(q, f k ) ≥ θ low}

[0279] Wherein, θ low is preset to 0.65, then the targets after preliminary screening are verified for space-time consistency to finely compare, and a comparison value score k is obtained.

[0280]

[0281] where T is the length of gait cycle, cossim is frame-level similarity function, here cosine similarity function, q t is the fusion feature of the t-th frame of the target to be identified, f k,t is the D of the t-th frame of the k-th identifiable sample identity learned by the training sample, and final is the mean value, and exp(-lambdaDeltaT) is a time decay function, and lambda is a time decay factor, which controls the decay rate. The greater the lambda, the faster the decay rate.

[0282] The final output identity decision ID result :

[0283]

[0284] where theta high is the final confidence threshold, which can be set by the user, for example, it can be 0.85.

[0285] System implementation

[0286] The human gait recognition system in a complex scene of the application comprises a processor, and the processor is used to execute computer program instructions to realize the human gait recognition method in a complex scene introduced in the method embodiment of the application, which will not be described in detail here. The processor can be a programmable logic device FPGA, a processing system or CPU of a computer, etc.

[0287] The application adopts the limited contrast adaptive histogram equalization method to limit the local contrast enhancement range, avoids overexposure, uses the Gamma correction to physically simulate the nonlinear response of the human eye to the dark area, adopts the perspective normalization method to solve the distortion interference of the oblique perspective, calculates the pitch angle through the hip-shoulder connecting line, projects the 3D joint to the sagittal plane through the rotation matrix, eliminates the limb shortening effect caused by the perspective, acquires the visible joint based on the HRNet, then reconstructs the missing human body area by using the physical continuity of the joint and the rigid structure of the trunk, solves the target occlusion interference, and realizes high-precision gait identity recognition in a complex scene with illumination change, occlusion and perspective change.

Claims

1. A method for human gait recognition in complex scenarios, characterized in that, include: The extracted human gait data are sequentially subjected to illumination correction, human posture angle alignment and occlusion processing, and human gait recognition is performed based on the gait data obtained after illumination correction, human posture angle alignment and occlusion processing. Occlusion processing includes occlusion domain estimation and occlusion reconstruction. Occlusion domain estimation is used to determine whether there are occluded regions in the gait data. The criteria for determining occlusion regions are: the number of occluded points is greater than or equal to the minimum threshold for the number of occluded points, and the skeletal nodes connected to the occluded points are visible and the occluded points are not visible for N consecutive frames, or the skeletal nodes are not in the expected position for N consecutive frames. Occlusion reconstruction is used to perform optical flow compensation on the pixel information of the occluded points using the optical flow vector calculated from the visible skeletal nodes around the occluded region in the illumination-corrected video image, or to interpolate and complete the occluded region based on the pixel information of at least 4 visible skeletal nodes around the occluded region.

2. The human gait recognition method in complex scenarios according to claim 1, characterized in that, The illumination correction includes: using the CLAHE method to adjust the contrast of the local region histogram of each block in the video image, using Gamma to correct the image brightness, and using LBP to binarize each pixel and its neighborhood; human pose angle alignment is used to project gait data to a unified viewpoint using the rotation matrix of the camera used when capturing the original video image.

3. The human gait recognition method in complex scenarios according to claim 1, characterized in that, The video images used to extract human gait data are determined based on the following process: global illumination equalization is performed on the original video images. If the target detection effect of the video images after global illumination equalization is greater than that of the original video images, global angle alignment is performed on the video images after global illumination equalization. Otherwise, global angle alignment is performed directly based on the original video images. The process involves several steps: First, determining if the occlusion region of the globally angle-aligned video image exceeds an occlusion threshold. If it does, frame-level occlusion processing is performed. Then, the target detection performance of the occluded video image is compared to that of the unoccluded image. If the latter is greater, human gait data is extracted from the occluded image; otherwise, the video image with the best target detection performance before occlusion processing is selected. If the globally angle-aligned video image is less than or equal to the occlusion threshold, the process continues. Global illumination equalization involves local region histogram processing and Gamma correction. Global angle alignment rotates the video image to the optimal angle, where the rotation results in the most symmetrical video image. Frame-level occlusion processing updates the video image using an exponential moving average method.

4. The human gait recognition method in complex scenarios according to claim 3, characterized in that, The process of extracting human gait data is as follows: the improved DeepSORT algorithm is used to process the video image after basic correction, output the trajectory of each target ID, and extract the human gait data of each target ID from the video based on the trajectory of each target ID. The improved DeepSORT algorithm is obtained by modifying the Mahalanobis distance formula for calculating spatial feature similarity in the original DeepSORT algorithm using a displacement field. This displacement field is obtained by calculating the dense optical flow of adjacent frames. The modified Mahalanobis distance formula is as follows: Where, d (1) (i,j) represents the spatial deviation between the predicted bounding box and the detected bounding box, α d d represents the weighting coefficient. j Let y be the detection distribution of the j-th detection box. i Let i be the predicted distribution of the i-th tracking trajectory. p is the inverse of the covariance matrix between the two distributions. i Let D be the center pixel coordinates of the target. flow (p i ) is p i The displacement field at that location.

5. The human gait recognition method in complex scenarios according to claim 1, characterized in that, The human gait data includes video images, human contour images, and human skeleton images. The human gait recognition process is as follows: feature extraction is performed on the video images to obtain global human gait features, which include color and texture; human contour features and human skeleton features are extracted based on the human contour images and human skeleton images extracted from the video images; the global human gait features, human contour features, and human skeleton features are fused; the fused features are used to perform human identity matching to determine the identity of the human target and obtain the gait recognition result.

6. The human gait recognition method in complex scenarios according to claim 5, characterized in that, The human body contour features are obtained by inputting the human body contour image into a multi-spatial domain refinement module; the multi-spatial domain refinement module includes a spatial feature self-division unit and a channel feature self-attention unit; The spatial feature self-division unit is used to divide the input human contour image into regions and generate feature maps for each region. The feature maps of each region are then stitched together to output the first feature map. The channel feature self-attention unit is used to assign different weights to feature channels with different identity representation in the first feature map to obtain human contour features.

7. The human gait recognition method in complex scenes according to claim 6, characterized in that, The process of dividing the input human contour image into regions and generating feature maps for each region is as follows: the human contour image is divided along the height direction of the human body to obtain the first region; Each first region is divided along the height of the human body to obtain a second region. A smooth mask for the first region is generated based on the distance between the human body contour image and the center of the first region. The smooth mask of the first region is multiplied element-wise with the human body contour image to obtain the feature map of the first region. The mask of the second region is calculated. The mask of the second region is multiplied element-wise with the feature map of the first region to obtain the feature map of the second region. The feature map of the second region is the feature map of each region.

8. The human gait recognition method in complex scenarios according to any one of claims 5-7, characterized in that, The human skeletal features include static biological features and dynamic gait features. Human static biometrics are obtained by inputting human contour images and human skeleton images into a human static biometrics extraction unit. The human static biometrics extraction unit is used to calculate the height to the ratio of each limb / trunk, as well as the limb / trunk amplitude, based on the pixel height occupied by human limbs / trunk in the human skeleton image and the pixel height occupied by human body in the human contour image. The height, the ratio of height to each limb / trunk, and the limb / trunk amplitude are used as the original biometric vectors. After normalization and nonlinear mapping of the original biometric vectors, the human static biometrics are obtained. Human dynamic gait features are obtained by inputting human skeleton images into the human dynamic gait feature refinement unit. The human dynamic gait feature refinement unit is used to construct a spatial graph based on the coordinates of the skeleton nodes in each video image and the spatial connection relationship of the skeleton nodes. Then, a graph convolutional neural network is used to extract the spatial features in the spatial graph. The spatial features are then convolved to obtain spatiotemporal features. Based on the spatiotemporal features, human dynamic gait features are obtained.

9. The human gait recognition method in complex scenes according to claim 8, characterized in that, The feature fusion process involves fusing static human biometrics with dynamic human gait features to obtain dynamic biometric fusion features; and using a multi-head attention mechanism to capture the correlation between human contour features and global human gait features at different time scales to output contour original image fusion features. The dynamic biological fusion features and the contour pre-image fusion features are concatenated and then input into a fully connected layer for feature extraction to obtain the fused features.

10. A human gait recognition system for complex scenarios, comprising a processor, characterized in that, The processor is used to execute computer program instructions to implement the steps in the human gait recognition method in complex scenarios as described in any one of claims 1-9.

Images