Multimodal gait recognition method, device, equipment and storage medium

Abstract

The application relates to the technical field of biometric recognition, and discloses a multi-modal gait recognition method based on uncertainty perception. The method comprises the following steps: acquiring multi-modal gait original data containing visual and inertial information, and calculating a preliminary uncertainty index; extracting space-time features and generating corresponding uncertainty feature maps; dynamically aligning the space-time features by using the uncertainty feature maps to obtain a standardized feature sequence; constructing an adaptive fusion weight based on the uncertainty feature maps, performing multi-modal fusion on the standardized feature sequence, and obtaining a fusion feature vector; inputting the fusion feature vector into an uncertainty propagation network to output an identity embedding vector with a confidence interval; and performing retrieval matching in a preset database in combination with the confidence interval to output a final recognition result. By quantifying uncertainty, dynamic feature processing is realized, noise interference is effectively inhibited, and the recognition accuracy and system reliability in a complex environment are improved.

Description

Technical Field

[0001] This invention relates to the fields of computer vision and biometric recognition technology, and particularly to a multimodal gait recognition method, apparatus, device, and storage medium. Background Technology With the continuous improvement of public safety monitoring systems and the widespread adoption of intelligent sensing devices, the demand for biometric identification technology in fields such as security monitoring, access control systems, and forensic identification has experienced explosive growth. Among the many biometric identification technologies, gait recognition has unique advantages such as non-contact capability, long-distance data collection, and difficulty in spoofing, and is considered a highly promising means of identity verification.

[0002] Traditional gait recognition methods primarily rely on appearance models or model-driven feature extraction. Appearance models typically utilize human contours, silhouettes, or optical flow information to construct gait energy images, which are then classified using template matching or machine learning algorithms. However, these methods are highly sensitive to image quality and are easily affected by factors such as lighting variations, cluttered backgrounds, clothing occlusion, and carried items. Model-driven methods attempt to analyze the motion trajectory of joints by building human skeletal models. While this overcomes the influence of appearance variations to some extent, the localization accuracy of skeletal nodes drops significantly in low-resolution videos or under severe occlusion, leading to a sharp deterioration in recognition performance.

[0003] In recent years, deep learning technology has made significant progress in the field of gait recognition. Convolutional neural networks and recurrent neural networks are widely used to automatically extract spatiotemporal features of gait. Although existing deep learning methods perform well on standard datasets, most existing technologies ignore the data uncertainties that exist in real-world application scenarios. These uncertainties mainly stem from two aspects: first, accidental uncertainty, caused by sensor noise, image compression distortion, environmental interference, etc.; second, cognitive uncertainty, caused by insufficient model training data, out-of-distribution samples, or the randomness of the model parameters themselves.

[0004] Existing technologies typically assume that the input gait data is deterministic and clear, directly inputting the raw or fused features into the classifier for judgment. When the input data is noisy or ambiguous, existing feature fusion strategies often employ fixed weight allocations, failing to dynamically adjust the contribution of each modality feature based on data quality. For example, when the scissor modality is severely affected by occlusion while the skeletal modality is relatively clear, a fixed-weight fusion network may assign excessively high weights to the severely damaged scissor modality, leading to incorrect recognition results. Furthermore, existing gait recognition systems lack confidence assessment of prediction results; the system cannot know the reliability of its own judgments, which is unacceptable in high-risk application scenarios.

[0005] Therefore, existing technologies lack a gait recognition method that can effectively perceive and quantify the uncertainties present in multimodal data, and dynamically adjust feature alignment, fusion, and decision-making mechanisms based on these uncertainties. Developing a multimodal gait recognition method with uncertainty perception capabilities has significant theoretical and practical value for improving the robustness, accuracy, and reliability of gait recognition in complex environments. Summary of the Invention

[0006] The purpose of this invention is to provide a multimodal gait recognition method, apparatus, device, and storage medium, which aims to solve the problems of decreased recognition rate and lack of confidence assessment caused by data quality fluctuations and fixed fusion strategies in complex environments in existing technologies.

[0007] To achieve the above objectives, the first aspect of the present invention provides a multimodal gait recognition method, comprising: Obtain raw gait data containing multimodal information, and perform denoising processing on the raw gait data to obtain preprocessed data, while calculating the preliminary uncertainty index of the preprocessed data; Extract the spatiotemporal features of the preprocessed data, and generate an uncertainty feature map that corresponds one-to-one with the spatiotemporal features based on the preliminary uncertainty index; The spatiotemporal features are dynamically aligned using the uncertainty feature map to obtain a standardized feature sequence; An adaptive fusion weight is constructed based on the uncertainty feature map, and the standardized feature sequence is fused using the adaptive fusion weight to obtain a fused feature vector. The fused feature vector is input into the uncertainty propagation network, which outputs an identity embedding vector with a confidence interval. The identity embedding vector is used to search and match in a preset database, and the final gait identity recognition result is output.

[0008] A second aspect of the present invention provides a multimodal gait recognition device, comprising: The data acquisition module is used to acquire raw gait data containing multimodal information, and to perform noise reduction processing on the raw gait data to obtain preprocessed data, while calculating the preliminary uncertainty index of the preprocessed data. The feature extraction module is used to extract the spatiotemporal features of the preprocessed data and generate an uncertainty feature map that corresponds one-to-one with the spatiotemporal features based on the preliminary uncertainty index. The dynamic alignment module is used to dynamically align the spatiotemporal features using the uncertainty feature map to obtain a standardized feature sequence; An adaptive fusion module is used to construct adaptive fusion weights based on the uncertainty feature map, and to perform multimodal fusion on the standardized feature sequence using the adaptive fusion weights to obtain a fused feature vector; The uncertainty propagation module is used to input the fused feature vector into the uncertainty propagation network and output an identity embedding vector with a confidence interval. The decision recognition module is used to perform a search and matching in a preset database based on the identity embedding vector, and output the final gait identity recognition result.

[0009] A third aspect of the present invention provides a computer device comprising: a memory and at least one processor, the memory storing instructions; the at least one processor invokes the instructions in the memory to cause the computer device to perform the multimodal gait recognition method as described in any of the preceding claims.

[0010] A fourth aspect of the present invention provides a computer-readable storage medium storing instructions that, when executed by a processor, implement the multimodal gait recognition method as described in any of the preceding claims.

[0011] Compared with the prior art, the present invention has the following significant advantages: (1) Enhanced environmental robustness: This invention calculates preliminary uncertainty indices and generates uncertainty feature maps, enabling the system to perceive external interferences such as changes in illumination, occlusion, and sensor noise in real time. During the feature alignment and fusion stages, the system can automatically suppress interference from high uncertainty features, thereby maintaining a high recognition accuracy even with low-quality data.

[0012] (2) Dynamic adaptive fusion mechanism: Existing technologies mostly use static weight fusion, which cannot adapt to dynamically changing data quality. This invention constructs adaptive fusion weights based on uncertain feature maps, realizing "on-demand allocation" based on data quality. When a certain modality (such as silhouette) is damaged, the system can automatically reduce its weight and increase the weight of other reliable modalities (such as skeleton), which significantly improves the efficiency of multimodal fusion.

[0013] (3) Explainable risk assessment: Traditional gait recognition systems only output a definite category label, and users cannot know the reliability of the result. This invention outputs an identity embedding vector with a confidence interval through an uncertainty propagation network, providing a quantitative confidence assessment for each recognition result. This is crucial in fields with extremely high security requirements, such as security and finance, and can effectively assist in manual review or trigger secondary verification.

[0014] (4) Precise feature alignment: In the gait cycle detection and feature alignment stages, this invention utilizes uncertainty information to eliminate abnormal key points, avoiding incorrect alignment based on noise points. This ensures the consistency and accuracy of subsequent feature fusion, laying a solid foundation for high-precision identity recognition.

[0015] (5) End-to-end full-process optimization: This invention integrates uncertainty perception throughout the complete closed loop from data preprocessing, feature extraction, alignment, fusion to final decision-making. The logic between each step is rigorous, and the output of the previous step (such as uncertainty index) serves as the key input of the next step, forming an effective transmission of information flow, avoiding logical gaps, and optimizing the overall performance of the recognition system. Attached Figure Description

[0016] Figure 1 This is a schematic diagram of an embodiment of the multimodal gait recognition method of the present invention; Figure 2 This is a schematic diagram of one embodiment of the multimodal gait recognition device of the present invention; Figure 3 This is a schematic diagram of one embodiment of the computer device in this invention. Detailed Implementation

[0017] The terms "first," "second," "third," "fourth," etc. (if present) in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" or "having" and any variations thereof are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0018] Example 1 This embodiment details the complete execution flow of a multimodal gait recognition method based on uncertainty perception. This embodiment is designed for complex indoor-outdoor mixed monitoring scenarios and can handle video data containing occlusion, sudden changes in lighting, and multi-view interference.

[0019] like Figure 1 As shown, the multimodal gait recognition method in this embodiment includes the following steps: 101. Obtain raw gait data containing multimodal information, and perform denoising processing on the raw gait data to obtain preprocessed data, while calculating the preliminary uncertainty index of the preprocessed data; The system initiates a data acquisition process. This process connects to a network of high-resolution industrial cameras deployed in the monitoring area, as well as inertial measurement units (IMUs) laid on the ground or integrated into wearable devices. The industrial camera network captures video of pedestrians walking, with the frame rate set to thirty frames per second to ensure sufficient gait details are captured. The IMUs collect accelerometer and gyroscope data of the pedestrians during their movement; this data assists in the identification of video data under extremely poor lighting conditions.

[0020] After acquiring the raw gait data, the system cleans the data. For video data, the system uses an adaptive Gaussian mixture model to model the background, and separates the foreground target by calculating the difference between the current frame and the background model. Subsequently, the system performs morphological processing on the foreground target, including erosion and dilation operations, to remove isolated noise points and fill holes inside the target, ultimately obtaining a binarized sequence of human silhouette images. For inertial measurement unit data, the system uses a Kalman filter algorithm to smooth the raw signal and remove high-frequency jitter noise.

[0021] While cleaning the data, the system begins calculating preliminary uncertainty metrics. For each silhouette image frame, the system calculates the contrast between foreground and background pixels and the continuity of edge gradients. If the contrast is low or the edge gradients are chaotic, it indicates that the image frame may have motion blur or insufficient lighting, and the system will assign a higher uncertainty value to that frame. For inertial measurement unit (IMU) data, the system calculates the variance of its acceleration amplitude. Excessive variance usually means that the sensor is being disturbed by non-walking motion, and it will also be assigned a high uncertainty value. Finally, the system aligns the uncertainty values ​​of the video and sensors according to timestamps to form a preliminary time-series uncertainty metric.

[0022] 102. Extract the spatiotemporal features of the preprocessed data, and generate an uncertainty feature map that corresponds one-to-one with the spatiotemporal features based on the preliminary uncertainty index; After obtaining preprocessed data and preliminary uncertainty indicators, the system enters the feature extraction stage. The preprocessed data is input into a specially designed two-stream convolutional neural network. This network consists of an appearance stream and a motion stream. The appearance stream takes a binarized silhouette image as input and uses a 3D convolutional kernel to extract the texture features of the human body contour that change over time; the motion stream takes an optical flow map between consecutive frames as input and captures the motion trajectory information of the human joints. The outputs of the two streams are fused early in the feature layer to form a high-dimensional feature tensor containing rich spatiotemporal information.

[0023] Simultaneously, a preliminary uncertainty index is input into an uncertainty estimation module based on a Bayesian neural network. This module does not directly output a definite numerical value, but rather a probability distribution representing the degree of uncertainty in the current data quality. For example, for a blurry image, the module might output a distribution with a large mean and variance. The system uses this probability distribution to modulate the spatiotemporal features. Specifically, the system weights each channel of the high-dimensional feature tensor, with the weights determined by the uncertainty probability distribution. If the uncertainty of a frame is high, the corresponding feature weight is reduced, or even set to zero. After this modulation, the generated feature map not only contains gait identity information but also explicitly encodes the credibility of that feature, i.e., the uncertainty feature map. The output here is a multidimensional matrix containing the eigenvalues ​​and their corresponding uncertainty variances.

[0024] 103. The spatiotemporal features are dynamically aligned using the uncertainty feature map to obtain a standardized feature sequence; A key challenge in gait recognition lies in the variations in gait cycle and gait deformation under different viewpoints. This step utilizes the uncertainty feature map generated in the previous step to guide the feature alignment process.

[0025] First, the system searches for key points in the gait cycle within the uncertainty feature map. Because gait is periodic, two-legged and one-legged support phases alternate. Traditional methods rely solely on peak values ​​of eigenvalues ​​to detect key points, making them susceptible to noise. In this embodiment, the system combines eigenvalues ​​and uncertainty variance to detect key points. Only when an eigenvalue shows a peak and the corresponding uncertainty variance is below a certain threshold is the point considered a valid key point. All peaks with excessively high uncertainty are considered abnormal interference points and are discarded.

[0026] After identifying the effective keypoints, the system uses a dynamic time warping algorithm to calculate the optimal matching path between different modalities or samples. During the warping calculation, the system introduces a metric based on uncertainty distance. The distance between two feature points depends not only on their Euclidean distance in the feature space but also on their uncertainty variance. If one of the two feature points has extremely high uncertainty, the matching cost between them will be significantly increased, thus guiding the warping algorithm to find a more reliable and deterministic matching path.

[0027] Finally, the system interpolates and aligns the feature sequence along the calculated optimal matching path. For gaps caused by outlier removal, the system fills them with adjacent feature points of low uncertainty through linear interpolation. The resulting standardized feature sequence is highly consistent over time, and the features at each time step have undergone reliability verification, eliminating recognition errors caused by temporal misalignment.

[0028] 104. Construct adaptive fusion weights based on the uncertain feature map, and use the adaptive fusion weights to perform multimodal fusion on the standardized feature sequence to obtain a fused feature vector; Aligned, standardized feature sequences contain information from various sources (e.g., silhouette appearance features and skeletal motion features). However, in real-world scenarios, the reliability of different modalities varies in real time. For example, silhouette features may be very rich at a side view angle, but depth information is lost at a camera-facing angle, reducing the effectiveness of silhouette features. This step aims to address this issue.

[0029] The system constructs adaptive fusion weights based on uncertainty feature maps. First, the system calculates the reliability score for each mode at the current time step. This score is obtained by inversely normalizing the uncertainty variance corresponding to that mode; the smaller the variance, the higher the score. Using this reliability score, the system initializes the parameter matrix of a spatial attention mechanism.

[0030] Subsequently, the system calculates a cross-modal correlation matrix using an attention mechanism. This matrix reflects the complementarity between features from different modalities. For example, when the scissor modality becomes ambiguous (highly uncertain), the skeletal modality usually remains clear; in this case, the correlation matrix shows that the skeletal modality has a stronger guiding significance for the scissor modality. The system uses this correlation to dynamically update the adaptive fusion weights. The update process is iterative; the system continuously adjusts the weights so that the fused features retain high-reliability information to the greatest extent possible while suppressing low-reliability information.

[0031] Finally, the system uses the updated adaptive fusion weights to perform a weighted summation of the standardized feature sequences. To prevent information loss during the weighting process, the system also introduces residual connections, directly adding the original input features to the weighted fused features. The resulting fused feature vector not only integrates the advantages of multimodal processing but also self-optimizes according to environmental conditions, exhibiting strong anti-interference capabilities.

[0032] 105. Input the fused feature vector into the uncertainty propagation network and output the identity embedding vector with confidence interval; This step transforms the fused feature vector into the final identity embedding vector used for comparison, and in the process, it completes the propagation of uncertainty.

[0033] The system constructs a deep metric learning network with a layer of random noise. Traditional neural network weights are fixed, while in this embodiment, the network weights are affected by the random noise layer during each forward propagation. This design is inspired by Bayesian deep learning and aims to simulate the uncertainty of model parameters.

[0034] After the fused feature vector is input into the network, the system performs multiple forward propagations (e.g., fifty times). In each propagation, a random noise layer adds a small perturbation to the weights of the fully connected layers. This means that even with the same input features, the network's prediction will fluctuate within a certain range each time.

[0035] The system performs statistical analysis on these fifty prediction results. First, it calculates the average of all predictions, which represents the network's optimal estimate of identity, i.e., the subject of the identity embedding vector. Second, it calculates the variance of all predictions, which quantifies the uncertainty of the model's predictions. If the input features are of high quality and the model is very familiar with those features, the variance of multiple predictions will be small; conversely, if the input features are ambiguous, the variance will be large.

[0036] Ultimately, the system uses the output mean as the identity embedding vector and transforms the output variance into a confidence interval. For example, the system provides an upper and lower bound for each dimension of the embedding vector. This identity embedding vector with a confidence interval is the core basis for subsequent identification decisions.

[0037] 106. Based on the identity embedding vector, perform a search and matching in the preset database, and output the final gait identity recognition result.

[0038] The system uses identity embedding vectors with confidence intervals to search a pre-set database to determine the identity of pedestrians. The pre-set database stores the identity embedding vectors of all registered users. Since registration data is typically collected in a controlled environment, the vectors in the database have high determinism. When a new vector to be identified is received, the system calculates the distance between the vector to be identified and every vector in the database.

[0039] In calculating distance, the system innovatively incorporates confidence intervals. Traditional Euclidean distance calculation is point-to-point, while the distance calculation in this embodiment is interval-to-point. If the confidence interval of the vector to be identified is wide (high uncertainty), and this interval covers the position of a vector in the database, the system will consider it a match, but will lower the score for the closeness of the match. If the confidence interval is narrow and very close to a vector in the database, the score will be high.

[0040] The system sorts all database candidates based on the corrected distance and selects the five candidates with the smallest distances as the candidate set. Then, the system determines whether the minimum distance in the candidate set is less than a preset threshold. This threshold is dynamic and is fine-tuned based on the uncertainty of the vector to be identified. If the uncertainty is high, the threshold is appropriately relaxed to prevent false negatives; if the uncertainty is low, the threshold is strictly tightened to prevent false positives.

[0041] If the minimum distance meets the threshold condition, the system outputs the identity label corresponding to the candidate as the final recognition result. If none of the distances meet the condition, the system outputs a "recognition failed" signal and prompts the user that the sample quality is poor, suggesting manual review or re-collection. This mechanism greatly improves the system's security and usability.

[0042] This embodiment provides a multimodal gait recognition method based on uncertainty perception. By creatively introducing and quantifying the uncertainty of data and models, it fundamentally solves the problem of poor robustness of existing technologies in complex environments. From the initial uncertainty calculation in the data acquisition stage, to the generation of uncertainty feature maps in the feature extraction stage, to the dynamic weight adjustment in the alignment and fusion stage, and finally to the confidence assessment in the final decision stage, this embodiment integrates uncertainty perception throughout the entire recognition process.

[0043] The method in this embodiment not only utilizes visual and inertial sensors to acquire rich gait information, but also intelligently determines which information is reliable and which needs to be suppressed. Through advanced techniques such as dynamic time warping, adaptive fusion attention mechanisms, and Monte Carlo sampling, this embodiment achieves active immunity to noise and interference, enabling the recognition results to maintain high accuracy even under extreme conditions such as low light, occlusion, and changing viewing angles.

[0044] Meanwhile, the recognition results with confidence intervals output in this embodiment greatly expand the application potential of gait recognition technology in highly sensitive fields such as security and finance. The system is no longer a black-box discriminator, but an intelligent assistant capable of self-assessment and controllable risk. This not only improves the user experience but also provides a scientific basis for system operation and maintenance and anomaly handling.

[0045] Example 2 This embodiment is a further detailed description of step 101 in Embodiment 1. This embodiment focuses on how to acquire high-quality multimodal gait data from complex real-world environments, and how to accurately quantify initial uncertainties during the data preprocessing stage, providing a reliable data foundation for subsequent feature extraction and fusion.

[0046] In this embodiment, step 101 further includes the following sub-steps: 1011. Acquire walking video stream through a multi-view sensor array and simultaneously acquire inertial measurement unit data, and use the walking video stream and the inertial measurement unit data as the raw gait data; In specific application scenarios, such as a busy subway station security checkpoint or company lobby, the system deploys a specially designed multi-view sensor array. This array not only includes conventional visible light cameras but also integrates infrared thermal imaging sensors to cope with extreme environments where there is no light at all. The sensor layout is precisely calculated, typically using a wraparound or crisscross arrangement to ensure that no matter from which angle a pedestrian passes by, at least two cameras can capture a video sequence containing the complete human gait cycle.

[0047] The acquisition process begins with obtaining the video stream. A visible light camera records video at a high frame rate of 60 frames per second, with the resolution set to ultra-high definition to ensure clear capture of details of foot contact with the ground and the texture information of clothing movement. Simultaneously, an infrared thermal imaging sensor works concurrently to generate a contour video stream based on human body thermal radiation. To address the time synchronization issue of multi-view data, a high-precision time protocol is used for clock synchronization at the hardware level, or specific synchronization frame identifiers are used for alignment at the software level to ensure that video streams from different perspectives remain consistent within a millisecond-level time error range.

[0048] Besides visual sensors, data acquisition by the inertial measurement unit (IMU) is equally crucial. In this embodiment, the IMU is either embedded in a specially designed smart mat or integrated into a personal identification card / smart bracelet. When a pedestrian steps on the mat or walks while wearing the bracelet, the IMU collects triaxial acceleration and triaxial angular velocity data in real time. This data is recorded at extremely high frequencies (e.g., 1 kilohertz per second), reflecting the minute fluctuations in the center of gravity and the inertial characteristics of the limbs during walking.

[0049] The system packages the collected multi-view walking video streams (including visible light and infrared data) with data from the high-frequency inertial measurement unit, collectively referred to as raw gait data. This raw data not only contains gait identification characteristics but also implicitly includes environmental noise information, serving as the source for subsequent processing. For example, in a specific test, a pedestrian wearing a long trench coat walked through the monitored area. The sensor array successfully recorded the pedestrian's side and rear view videos while walking, while the ground mat sensors recorded the impact acceleration curves of each step. These data together constitute the raw material for subsequent analysis.

[0050] 1012. Perform background modeling and foreground segmentation on the walking video stream, extract the human silhouette image sequence, and perform smoothing filtering on the inertial measurement unit data to generate the preprocessed data; After acquiring the raw gait data, rigorous preprocessing is necessary to remove irrelevant information. For video stream data, the system employs an adaptive Gaussian mixture model for background modeling. This algorithm can establish multiple Gaussian distributions for each pixel to represent slow changes in the background, such as gradual changes in illumination or slight movements of objects in the background. When a new frame of video is input, the algorithm matches the current pixel value with the existing Gaussian model. If the current pixel value can be well explained by the existing background Gaussian model, it is determined to be a background point; otherwise, it is determined to be a foreground point.

[0051] After obtaining the foreground mask, the system further performs morphological operations. First, an erosion operation is performed to remove isolated noise points from the foreground mask, which are typically caused by thermal noise from the image sensor itself or by flying insects in the environment. Next, a dilation operation is performed to connect broken human body parts and fill in the voids inside the body, especially the gaps created during segmentation between clothing and the body due to similar colors. After this series of processing steps, the system transforms the person in the color video into a clean sequence of binarized human silhouette images.

[0052] For inertial measurement unit (IMU) data, the raw data contains a large number of high-frequency glitches due to the electronic noise of the sensor itself and the random vibrations generated by walking on irregular ground. To eliminate these interferences, the system employs a Kalman filter algorithm. The Kalman filter establishes a system model that includes state equations and observation equations. The state variables include the actual acceleration and angular velocity, while the observation variables are the noisy values ​​directly read from the sensor. The filter continuously corrects its estimate of the true state through a cyclical iterative process of "prediction" and "update".

[0053] In the "prediction" step, the filter predicts the current state based on the state estimate from the previous moment and the physical motion model. In the "update" step, the filter adjusts the state estimate using the residual between the current sensor observation and the predicted value, thus offsetting the effects of noise. The acceleration curve after smoothing filtering becomes smooth and continuous, accurately reflecting the dynamic characteristics of human walking. The silhouette image sequence generated from the video stream and the processed inertial measurement unit data together constitute the preprocessed data. For example, if there is interference from a flag fluttering in the wind in the original video, the background modeling and foreground segmentation steps can effectively exclude the flag from the human silhouette, ensuring that the silhouette only contains the shape of the pedestrian; similarly, if a sudden spike signal appears in the sensor data (possibly due to a pedestrian tripping or an electrical signal impact on the sensor), the Kalman filter can smooth out the spike, restoring the true gait acceleration waveform.

[0054] 1013. Calculate the signal-to-noise ratio of the human silhouette image sequence and the acceleration variance of the inertial measurement unit data, and perform a weighted mapping between the signal-to-noise ratio and the acceleration variance to obtain the preliminary uncertainty index.

[0055] The generation of preprocessed data does not mean it can be used directly for identification. The system also needs to evaluate the reliability of this data, that is, to calculate the preliminary uncertainty index. This is a key step in quantifying data quality.

[0056] For a sequence of human silhouette images, the system calculates the signal-to-noise ratio (SNR) for each frame. The calculation first involves statistically analyzing the mean grayscale value of the foreground region (human pixels) and the mean grayscale value of the background region, then calculating the difference as the signal strength. Simultaneously, the standard deviation of the grayscale values ​​in the background region is calculated as the noise intensity. The SNR is the ratio of signal strength to noise intensity. When there is insufficient lighting or backlighting, the contrast between the foreground and background decreases, and the SNR drops significantly, indicating high uncertainty in that frame. Furthermore, the system also calculates the sharpness of the silhouette edges; if the edges are blurry, the SNR will also decrease accordingly.

[0057] For inertial measurement unit (IMU) data, the system calculates the acceleration variance within the sliding time window. Variance reflects the degree of data fluctuation. During a normal gait cycle, acceleration changes exhibit a specific regularity, and its variance remains within a relatively stable range. However, if a pedestrian falls, stops abruptly, or the sensor is subjected to strong external magnetic interference, the data waveform becomes chaotic, leading to a sharp increase in variance. High variance means the data deviates from the normal gait model and implies higher uncertainty.

[0058] After obtaining the signal-to-noise ratio (SNR) of the video mode and the acceleration variance of the inertial mode, the system needs to fuse these two metrics with different dimensions. Since a higher SNR value generally indicates better quality, while a lower variance value generally indicates better quality, the system first normalizes and inverts the variance to make it follow the same trend as the SNR (i.e., a higher value indicates better quality). Subsequently, the system uses a non-linear weighted mapping function, such as the sigmoid function or a piecewise linear function, to map the processed SNR and variance to the interval between zero and one. The parameters of the mapping function are obtained through training on a large amount of labeled data, enabling adaptive adjustment of the weights of the two modes in uncertainty assessment.

[0059] For example, in scenarios where vision is severely obstructed but inertial data is normal, the mapping function assigns a higher weight to the signal-to-noise ratio, making the final preliminary uncertainty index primarily reflect the low quality of the visual data. The output preliminary uncertainty index is a one-dimensional vector aligned with the original data time series, where each value corresponds to the degree of uncertainty of the data at a given moment. Values ​​close to zero indicate highly reliable data, while values ​​close to one indicate unreliable data. Through this step, the system not only obtains cleaned data but also endows the data with "self-awareness," enabling it to inform subsequent modules which parts are reliable and which are questionable.

[0060] Example 3 This embodiment is a further detailed description of step 102 in Embodiment 1. The core of this embodiment lies in using deep learning technology to extract deep gait features and explicitly encoding the uncertainty information obtained in the preceding steps into the feature space to generate an uncertainty feature map.

[0061] In this embodiment, step 102 further includes the following sub-steps: 1021. Input the preprocessed data into a dual-stream convolutional neural network to extract appearance texture features and skeletal motion dynamic features as the spatiotemporal features.

[0062] In this embodiment, the preprocessed data includes aligned silhouette image sequences and filtered inertial data. To comprehensively capture gait information, the system constructs a two-stream convolutional neural network architecture.

[0063] The first network is called the "Appearance Stream." The Appearance Stream receives a sequence of binarized silhouette images as input. This network uses a 3D convolutional neural network as its backbone, such as an improved version of ResNet-3D or C3D. The 3D convolutional kernels can slide simultaneously along both the spatial and temporal dimensions (image width and height) to capture the evolution of the human silhouette over time. The early convolutional kernels focus on extracting low-level texture features such as edges and corners, the mid-level kernels identify the shape of body parts such as legs and arms, and the deep kernels aggregate global information to form high-level semantic features, such as walking posture amplitude and stride width. The Appearance Stream ultimately outputs a 3D feature tensor containing appearance texture information.

[0064] The second network is called "Motion Flow". Considering that silhouette images may lose shape information when occluded, Motion Flow focuses on capturing dynamic motion trends. This network first uses optical flow estimation algorithms (such as FlowNet 2.0) to calculate pixel-level motion fields between consecutive silhouette frames, or directly receives the sequence of joint coordinates extracted from the silhouette. If the input is inertial data, the system transforms the one-dimensional time-series data into a two-dimensional "pseudo-image" by rearranging it, preserving the structure of the time axis and sensor axis. Motion Flow also uses a three-dimensional convolutional neural network or a spatiotemporal graph convolutional network (ST-GCN) for processing. ST-GCN is particularly suitable for processing skeletal joint data; it treats human joints as nodes in a graph, and connections between joints as edges, using graph convolution to extract features from the topological structure. The feature tensor output by Motion Flow mainly reflects the velocity, acceleration, and coordinated motion patterns between joints in various parts of the human body.

[0065] The system concatenates the appearance texture features output from the appearance stream with the skeletal motion dynamic features output from the motion stream along the channel dimension to form a preliminary spatiotemporal feature. This spatiotemporal feature is a high-dimensional matrix containing rich gait representation information. For example, appearance features may capture the "outward-pointing" foot contour features of a pedestrian, while motion features capture the dynamic pattern of "small arm swing amplitude." The combination of the two constitutes the pedestrian's unique gait fingerprint.

[0066] 1022. Construct an uncertainty estimation module based on Bayesian inference, input the preliminary uncertainty index into the uncertainty estimation module, and learn the data noise distribution.

[0067] Simply extracting spatiotemporal features is insufficient; the system also needs to understand the reliability of these features. Therefore, this step embeds an uncertainty estimation module based on Bayesian inference into the feature extraction network.

[0068] Traditional neural network weights are fixed point values, while the uncertainty estimation module in this embodiment employs variational inference. The network weights in this module are no longer considered fixed values, but are assumed to follow a certain probability distribution (usually a Gaussian distribution). The mean of the distribution represents the central position of the weights, and the variance represents the uncertainty of the weights.

[0069] During the training phase, the system uses the preliminary uncertainty index calculated in step 101 as the guiding input for this module. The preliminary uncertainty index is spatially upsampled or expanded to match the size of the spatiotemporal features. The uncertainty estimation module receives the preliminary uncertainty index and spatiotemporal features as input, maps them through several fully connected or convolutional layers, and outputs two sets of parameters: a mean parameter map and a variance parameter map. These two sets of parameters define the latent noise distribution that each feature point in the feature space follows.

[0070] The core of the learning process is minimizing the variational lower bound loss function. This loss function comprises two parts: reconstruction loss and KL divergence. The reconstruction loss ensures that the distribution of the module's output can effectively explain the spatiotemporal features of the input, while the KL divergence constrains the learned distribution from deviating too far from the prior distribution, preventing overfitting. Through this training mechanism, the module learns to transform initial uncertainty indicators into uncertainties at the deep feature level. For example, when the initial uncertainty indicator shows motion blur in a certain video segment, the uncertainty estimation module will correspondingly increase the variance parameter corresponding to the features in that region, indicating that the features in that region may have significant noise interference.

[0071] This process is not merely a simple transmission, but rather a form of "learning" and "reasoning." The module can correct for uncertainties based on contextual information. For example, even if initial indicators show high uncertainty in a particular frame, if the features of that frame are highly consistent with those of the preceding and following frames and conform to human kinematics, the module may appropriately reduce its estimated uncertainty, and vice versa. This reasoning ability enables the system to obtain deeper feature uncertainties more accurately than the original sensor noise estimation.

[0072] 1023. Using the noise distribution parameters output by the uncertainty estimation module, perform probability mapping on each feature dimension of the spatiotemporal features to generate the uncertainty feature map.

[0073] After the uncertainty estimation module outputs the mean and variance parameters of the noise distribution, the system uses these parameters to modulate the original spatiotemporal features and finally generates an uncertainty feature map.

[0074] The specific operation involves performing probability mapping on each feature dimension of the spatiotemporal features. For each value in the spatiotemporal feature tensor, the system does not use the value directly, but treats it as an observation sampled from a probability distribution. The system uses the variance parameter at that location output by the uncertainty estimation module to construct a normal distribution with the original feature values ​​as the mean. Subsequently, the system performs reparameterized sampling within this normal distribution.

[0075] The reparameterization technique is key to this step, allowing gradients to backpropagate through a sampling process. Specifically, the sampling process can be represented as: Eigenvalue = Mean + Standard Deviation × Random Noise. Here, the random noise is sampled from a standard normal distribution. In this way, the system directly injects uncertainty (standard deviation) into the eigenvalues.

[0076] The final generated feature map is named the "uncertainty feature map." This feature map has a special structure: in the channel dimension, it not only contains traditional gait feature values ​​but also the corresponding uncertainty variance values, or it combines the two through some encoding method (such as concatenating the variance as an additional channel after the feature values). Visually, if the uncertainty variance is visualized, a heatmap similar to the gait contour is obtained. Areas with high brightness in the heatmap indicate that the gait features in that region are uncertain, possibly due to occlusion or noise; areas with low brightness indicate that the feature extraction is stable and reliable.

[0077] For example, for a pedestrian partially obscured by a lamppost, the silhouette of their torso is likely clearly visible, resulting in high confidence (low variance) eigenvalues ​​in the corresponding uncertainty feature map. However, for the obscured legs, the eigenvalues ​​rely primarily on extrapolation and guessing by the network, leading to a significantly increased variance in the corresponding uncertainty feature map. This feature map with uncertainty annotations provides invaluable guidance for subsequent dynamic alignment and fusion, enabling the network to determine "where to trust and where not to trust."

[0078] Example 4 This embodiment is a further detailed description of step 103 in Embodiment 1. This embodiment addresses common problems in gait recognition such as gait speed variations, frame rate differences, and local noise interference by utilizing uncertainty information to achieve high-precision dynamic feature alignment.

[0079] In this embodiment, step 103 further includes the following sub-steps: 1031. Detect key points of gait cycle in the spatiotemporal features, and remove abnormal points with uncertainty higher than a preset threshold from the key points according to the uncertainty feature map.

[0080] Gait exhibits a clear periodicity, with a complete gait cycle typically consisting of a "two-footed support phase" and a "one-footed support phase." Accurately detecting key points in the gait cycle (such as the moment of toe lift-off and heel strike) is a prerequisite for feature alignment. However, under noise interference, traditional periodicity detection algorithms are prone to false detections.

[0081] This embodiment first uses energy concentration or autocorrelation function to search for periodic peaks in the time dimension of spatiotemporal features. When an energy peak appears, it indicates a drastic attitude change, typically a gait keyframe. The system records the timestamps corresponding to these peaks as candidate keypoints.

[0082] Unlike existing technologies, this system does not blindly accept all candidate keypoints. Instead, it uses the uncertainty feature map generated in the previous step. For each candidate keypoint, the system extracts the uncertainty variance at that moment. If this variance exceeds a preset dynamic threshold, the system marks the keypoint as an "outlier" and removes it. The dynamic threshold is set based on the statistical distribution of the uncertainty across the entire sequence, for example, by adding two standard deviations to the mean.

[0083] This rejection mechanism is highly effective. For example, if a pedestrian's silhouette suddenly becomes extremely cluttered due to carrying a large suitcase, the energy detector might mistakenly identify this interfering frame as a keypoint. However, the variance of the uncertainty feature map at that moment will be extremely high, exceeding a threshold, thus triggering the rejection mechanism. The system will continue searching within that time window, or use interpolation methods to find a suboptimal peak with lower uncertainty as a replacement keypoint. Through this step, the system ensures that the keypoints used for subsequent alignment are extracted based on high-quality features.

[0084] 1032. Based on the retained effective key points, perform dynamic time warping to calculate the optimal matching path between different modal features.

[0085] After obtaining the cleaned and valid keypoints, the system needs to address the issue of inconsistent gait speeds among different samples. Even for the same person, walking speed may vary at different times, causing the feature sequence to be stretched or compressed along the time axis. To solve this problem, the system employs a dynamic time warping algorithm.

[0086] The Dynamic Time Warping (DTW) algorithm finds the optimal matching path between two time series by constructing a distance matrix, minimizing the cumulative distance. However, in this embodiment, the system improves the DTW algorithm by introducing an uncertainty-weighted distance metric.

[0087] The system first constructs a cross-modal or cross-sample distance matrix. Each element in the matrix represents the distance between two feature points. When calculating this distance, the system considers not only the Euclidean distance between feature vectors but also uses variance information from the uncertainty feature map to weight the distance. The specific weighting strategy is: if any feature point involved in the matching has high uncertainty (large variance), then the matching cost (distance) between them will be penalized and increased.

[0088] When searching for the optimal matching path, the DTW algorithm uses dynamic programming to search from the top left corner to the bottom right corner of the matrix. Due to the aforementioned weighting mechanism, the algorithm tends to avoid regions with high uncertainty and instead matches along "reliable paths" with low uncertainty. This means that if a segment of feature data is of poor quality, the system will not force a distorted match between this data and another segment of high-quality data, but will instead look for more reliable feature points in the surrounding area to correspond to.

[0089] For example, when comparing a slow-walking sample and a fast-walking sample, the standard DTW might forcibly map noisy frames in the fast-walking sample to normal frames in the slow-walking sample in order to align the time axis. The method in this embodiment, however, identifies the high uncertainty of blurred frames in the fast-walking sample, thus bypassing these frames when calculating the matching path. Instead, it interpolates and connects them using more reliable adjacent frames to calculate an optimal matching path that truly reflects the similarity of gait structures.

[0090] 1033. Perform interpolation and alignment operations on the spatiotemporal features along the optimal matching path to fill in the positions of the removed outliers and generate the standardized feature sequence.

[0091] Once the optimal matching path is determined, the system possesses a "map" that aligns feature sequences of different lengths and speeds onto the same time axis. The system then begins performing interpolation and alignment operations.

[0092] First, the system sets a standard time length, such as a standard gait cycle sequence containing forty frames. Then, based on the optimal matching path, the system maps the original spatiotemporal features onto this standard time axis. For each matching point pair on the path, the system retains the original feature values. If a jump occurs in the path (i.e., one frame in the fast walking sequence corresponds to multiple frames in the slow walking sequence), the system uses linear interpolation or cubic spline interpolation to generate new feature values ​​on the standard time axis to fill the gaps caused by compression.

[0093] Specifically, for the locations of rejected outlier keypoints, the system also utilizes their neighboring valid keypoints for feature reconstruction. Since the uncertainty feature map provides variance information, the system can employ probabilistic interpolation methods such as Kalman smoothing or Gaussian process regression during reconstruction. These methods can use the uncertainty of neighboring points as input to the covariance matrix, thus providing an optimal estimate that ensures the reconstructed features are statistically consistent with the surrounding reliable features.

[0094] After the above processing, all input gait feature sequences were transformed into standardized feature sequences of uniform length and strictly aligned time axes. In this sequence, noise, temporal distortions, and outliers in the original data were effectively corrected. Each time slice corresponds to a specific phase in the gait cycle (such as the initial stage of left foot support, the end of the swing phase, etc.), and the feature values ​​of this slice have been subjected to uncertainty cleaning and calibration.

[0095] The standardized feature sequence output in this step has extremely high comparability. For example, regardless of whether the pedestrian's walking speed in the original video is one meter per second or 1.5 meters per second, and regardless of whether there are several frames obscured by vehicles, this step will generate a uniform feature matrix. Each column of the matrix strictly corresponds to the same stage of the gait, and the reliability of the feature values ​​is guaranteed to the greatest extent.

[0096] Example 5 This embodiment is a further detailed description of step 104 in Embodiment 1. This embodiment aims to solve the problem of how to dynamically allocate weights based on data quality during the fusion process of multimodal information, overcoming the limitations of traditional static fusion methods when faced with missing modes or noise interference.

[0097] In this embodiment, step 104 further includes the following sub-steps: 1041. Calculate the reliability score of each modal feature based on the uncertainty feature map, and initialize the attention mechanism parameters using the reliability score.

[0098] After feature alignment, the system obtains a standardized spatiotemporal feature sequence and a corresponding uncertainty feature map. To achieve effective fusion, the reliability of each modality at the current time step needs to be evaluated first. The system calculates the reliability score of each modality feature based on the uncertainty feature map.

[0099] Each value in the uncertainty feature map represents the uncertainty variance of that location's feature. A larger variance means a higher probability that the feature is affected by noise, and a lower reliability. The system calculates the reliability score through the following steps: First, the uncertainty variance is normalized, mapping it to the interval between zero and one; then, an inverse proportional function (such as a reciprocal function or an exponential decay function) is used to convert the normalized variance into a reliability score. Specifically, if the variance at a location is close to zero, its reliability score will be close to a perfect score of one; if the variance is large, the reliability score will drop rapidly.

[0100] For example, at a certain moment, a sudden change in illumination might cause a sharp increase in the uncertainty variance of the visual modality (silhouette features), resulting in a reliability score of only 0.2; while the inertial modality (acceleration features) is less affected, and its reliability score could be as high as 0.9. These scores constitute the data quality report for the current moment.

[0101] Using these reliability scores, the system initializes the parameters of the attention mechanism. The attention mechanism is designed as a multi-layered neural network structure, containing three linear transformation layers: query, key, and value. The system uses the calculated reliability scores as prior knowledge, injecting them into the initial weights of the attention mechanism. Specifically, the reliability scores are added as a bias term to the initial bias parameters of the attention layer, or the magnitude of the initial weights is adjusted based on the score. The aim is to ensure that the attention mechanism, at the beginning of training or inference, tends to focus on modal features with high reliability, thereby accelerating model convergence and improving fusion accuracy.

[0102] 1042. Calculate the cross-modal correlation matrix through the attention mechanism, and update the adaptive fusion weights based on the correlation matrix.

[0103] After initialization, the system enters the core stage of dynamic weight adjustment. The system utilizes an attention mechanism to calculate the cross-modal correlation matrix. The correlation matrix is ​​a two-dimensional matrix where rows and columns represent feature vectors from different modalities. Each element in the matrix represents the correlation strength between features from two different modalities.

[0104] During the calculation, the system uses visual modal features as the query and inertial modal features as the key (or vice versa), calculating the matching degree between them through dot product operations. To prevent the value from being too large, the system scales the dot product result. Since a reliability score has been introduced, the system considers the quality of features when calculating the correlation. If the reliability score of the query feature itself is very low, the system will automatically reduce its participation in the correlation calculation to avoid low-quality features misleading the overall judgment.

[0105] The generated correlation matrix visually demonstrates the complementarity between different modalities. For example, when visual features struggle to distinguish walking details (such as when legs are together), the acceleration patterns of inertial features may provide crucial discriminative information, resulting in high values ​​in the correlation matrix.

[0106] The system updates adaptive fusion weights in real time based on the correlation matrix. The update process employs either gradient descent or direct mapping. A weight generation network is designed within the system. This network receives the correlation matrix and the current reliability score as input and outputs a new set of weight coefficients. These weight coefficients reflect not only the reliability of the modality itself but also the interaction relationships between modalities. If the visual modality is highly correlated with the inertial modality, and the inertial modality is more reliable, the weight generation network increases the weight of the inertial modality while maintaining the weight of the visual modality to preserve its unique texture information. This update mechanism is dynamic; as the input data changes, the fusion weights are constantly adjusted to ensure that the fusion result is always dominated by the most reliable information source.

[0107] 1043. The standardized feature sequence is weighted and summed using the updated adaptive fusion weights, and the original features are added to the fused features through residual connections to output the fused feature vector.

[0108] After obtaining the updated adaptive fusion weights, the system begins the final fusion operation. The system applies the adaptive fusion weights to the normalized feature sequences of different modalities. This means that each feature vector is multiplied by its corresponding weight coefficient. Modal features with high weights are numerically amplified, while modal features with low weights are suppressed.

[0109] Subsequently, the system sums all the weighted feature vectors to generate preliminary fused features. This weighted summation method is both simple and efficient, enabling the convergence of multi-source information into a unified feature space. However, to prevent the loss of original feature details during the weighting process, especially those minute features that, while generally unreliable, contain unique identity information, the system introduces a residual connection mechanism.

[0110] Residual connections are implemented by directly adding the original input features (i.e., the standardized features before weighting) to the initial fused features. This design borrows from the idea of ​​deep residual networks, enabling the network to learn the residual mapping between input and output. Physically, this means that the system retains the complete spectral information of the original features while using adaptive weights to emphasize the main features. Even if the adaptive weights are misjudged, residual connections can ensure that the original information is passed to subsequent layers, preventing information gaps.

[0111] The feature vector after residual connection is processed by a final activation function (such as ReLU or Tanh) to output the final fused feature vector. This vector not only fuses complementary information from the visual and inertial sensors, but also maximizes the integrity and robustness of the information through adaptive weights and residual connection mechanisms. For example, in a specific test case, when the visual sensor temporarily fails due to strong light interference, the adaptive weights automatically increase the weight of the inertial features to 0.9 and decrease the weight of the visual features to 0.1. The fused feature vector is mainly composed of inertial features. At the same time, the residual connection preserves the weak visual signal. When the strong light interference disappears and the visual features become clear again, the weights can be quickly adjusted back, and the system seamlessly switches back to the dual-modal high-precision fusion state. This intelligent fusion mechanism is the key guarantee for the high recognition rate of this invention.

[0112] Example 6 This embodiment is a further detailed description of step 105 in Embodiment 1. This embodiment aims to use probabilistic deep learning technology to propagate uncertainty at the feature level to the final decision level, providing a quantifiable confidence assessment for the identification results.

[0113] In this embodiment, step 105 further includes the following sub-steps: 1051. Construct a deep metric learning network containing a random noise layer, and use the fused feature vector as the input of the deep metric learning network.

[0114] Traditional neural networks use fixed weights during inference, making it impossible to output predictions with uncertainty. To address this issue, this embodiment constructs a deep metric learning network that includes a layer of random noise. The basic structure of this network consists of multiple fully connected layers used to map high-dimensional fused feature vectors to a low-dimensional identity embedding space.

[0115] The core innovation of this network lies in the introduction of a random noise layer between the fully connected layers. This random noise layer is not a simple Dropout layer, but a variable layer designed based on the principles of Bayesian neural networks. During training, the random noise layer learns the mean and variance of the weight distribution through reparameterization techniques. During inference (testing), this layer no longer outputs fixed weight values, but instead performs random sampling based on the learned distribution.

[0116] In practice, the weights of each layer in a deep metric learning network are represented as a Gaussian distribution. After the fused feature vectors are input into the network, the data undergoes a first fully connected transformation. Before moving to the next layer, a random noise layer generates a random perturbation matrix based on the mean and variance of the current weights. This perturbation matrix is ​​added to the network weights, causing a small, random change in the network parameters with each forward propagation.

[0117] This design transforms deep metric learning networks into probabilistic models. Given the same fused feature vector input to the network, the output embedding vector will differ each time due to the randomness of the network weights. This difference is not a system error, but rather a simulation of the model's cognitive uncertainty. If the model is highly certain about the input features (i.e., the features are located in dense regions of the training data distribution), the output will be very stable regardless of small fluctuations in the weights; conversely, if the input features are ambiguous or belong to unseen categories (i.e., located at the edge of the data distribution), small perturbations in the weights will cause significant jumps in the output. The system utilizes this output variability to quantify uncertainty.

[0118] 1052. In each forward propagation of the deep metric learning network, the network weights are Monte Carlo sampled using the random noise layer to simulate multiple prediction processes.

[0119] To capture the aforementioned output variability, the system employs Monte Carlo sampling to perform multiple predictions. Monte Carlo sampling is a statistical method that approximates numerical results through random sampling.

[0120] In this embodiment, after the fused feature vector is input into the network, the system does not run the forward propagation only once, but runs it multiple times consecutively (e.g., fifty or one hundred times). At the beginning of each forward propagation, the random noise layer independently samples from the posterior distribution of the weights to generate a set of temporary network weights. The system uses this set of temporary weights to calculate the input features and obtain a temporary identity embedding vector.

[0121] This process can be viewed as simulating a group of "experts" judging the same gait sample. Each "expert" (i.e., a sampled state of the network) has its unique perspective and slight cognitive biases. The system records the judgments of all "experts." For example, for a specific fused feature vector, the embedding vector coordinates obtained from the first sample are [0.5, -0.2, 0.8], the coordinates obtained from the second sample are [0.52, -0.19, 0.79], and the coordinates obtained from the thirtieth sample are [0.48, -0.21, 0.81]. Although these results are generally consistent, there are random fluctuations in the numerical details.

[0122] Through this extensive process of repeated sampling and prediction, the system gathers a wealth of statistical information. This simulation process not only calculates the final identity features but, more importantly, probes the "flatness" of the model at each feature point. If the output distribution is very sharp (small variance), it indicates that the model is very confident in its judgment at that point; if the output distribution is very flat (large variance), it indicates that the model is hesitant. This information cannot be obtained through a fixed-weight network.

[0123] 1053. Calculate the output mean and output variance of the multiple prediction processes, take the output mean as the main body of the identity embedding vector, convert the output variance into a confidence interval, and merge to output the identity embedding vector with the confidence interval.

[0124] After completing a preset number of Monte Carlo samplings, the system collects the identity embedding vectors of all predicted outputs and begins statistical analysis.

[0125] First, the system calculates the arithmetic mean of all sampled vectors. This mean represents the model's optimal estimate of the identity feature after considering various random weight perturbations. Mathematically, it can be proven that this expected value is the best prediction point that minimizes the mean squared error. The system uses this mean vector as the main part of the final output identity embedding vector. This vector will be used to retrieve the closest matching target from the database.

[0126] Next, the system calculates the variance of all sampled vectors in each dimension. Variance measures how much the sampling results deviate from the mean. For each dimension, the system constructs a confidence interval based on the variance value. The width of the confidence interval is proportional to the square root of the variance (standard deviation). For example, the system can set a 95% confidence interval, which is the mean plus or minus 1.96 times the standard deviation.

[0127] Finally, the system outputs a special identity embedding vector structure. This structure not only contains a mean vector but also includes a confidence interval vector of the same dimension, or an additional attribute that encodes the confidence information as a vector. For example, the output identity embedding vector can be represented as: {Subject: [0.5, -0.2, 0.8], Lower confidence limit: [0.45, -0.25, 0.75], Upper confidence limit: [0.55, -0.15, 0.85]}.

[0128] This identity embedding vector with a confidence interval provides rich evidence for subsequent matching decisions. If the confidence interval output by the system is very narrow, it indicates that the feature extraction and recognition process is very reliable; if the confidence interval is very wide, even exceeding the recognition threshold, it indicates that the system's recognition of the sample poses a significant risk. This method of explicitly quantifying uncertainty transforms the gait recognition system from a "black box" to a "white box," greatly enhancing the system's practicality and security.

[0129] Example 7 This embodiment is a further detailed description of step 106 in Embodiment 1. This embodiment is the final step of the entire method, using the identity embedding vector with confidence intervals generated in the preceding steps to perform high-precision matching and decision-making in the database.

[0130] 1061. Calculate the Euclidean distance between the identity embedding vector and the registration vector in the preset database, and correct the Euclidean distance by combining the confidence interval.

[0131] The pre-built database stores gait feature templates for all authorized personnel. These templates are typically collected in a high-quality, controlled environment and averaged multiple times. Therefore, the registration vectors in the database are usually considered to be deterministic points (or have very little uncertainty). When the system receives an identity embedding vector with a confidence interval to be identified, it needs to calculate the similarity between this vector and each registration vector in the database.

[0132] First, the system calculates the Euclidean distance between the subject (mean) of the vector to be identified and the registered vector. Euclidean distance is a commonly used metric to measure the geometric distance between two vectors in the feature space; the smaller the distance, the more similar the two vectors are.

[0133] However, relying solely on the mean distance is insufficient because the vector to be identified carries confidence interval information. If the uncertainty of the vector to be identified is extremely high (the confidence interval is wide), even if its mean is very close to a registered vector, it cannot be concluded that the two are the same person, because the mean may just be the result of random fluctuations. Conversely, if the uncertainty of the vector to be identified is extremely low (the confidence interval is narrow) and the mean distance is very close, then the matching confidence is extremely high.

[0134] Therefore, the system modifies the Euclidean distance by incorporating confidence intervals. The core idea of ​​the correction formula is to introduce a "distance penalty term." The magnitude of the penalty term is proportional to the width of the confidence interval. Specifically, the system calculates the width of the confidence interval for the vector to be identified in each dimension (e.g., upper limit minus lower limit), and sums or weighted sums the widths of all dimensions to obtain a total uncertainty metric. Then, this uncertainty metric is multiplied by a penalty coefficient and added to the original Euclidean distance.

[0135] The corrected distance formula can be understood as: Corrected distance = Original Euclidean distance + λ × Confidence interval width. Here, λ is an adjustable parameter used to control the degree of influence of uncertainty on the matching results. Through this correction, the corrected distance of feature vectors that are close in mean but ambiguous will be increased, thus reducing the risk of them being mismatched; while the corrected distance of clear and stable feature vectors will remain small, ensuring the priority of correct matching.

[0136] 1062. Sort the candidates in the preset database according to the corrected distance, and select the top preset number of candidates with the smallest distance as the candidate set.

[0137] After calculating the corrected distance between the vector to be identified and all registered vectors in the database, the system begins filtering and sorting. The database may contain tens of thousands of registered vectors; after comparing them one by one, the system obtains a list of distances.

[0138] The system sorts all candidates in the database according to the corrected distance from smallest to largest. The candidates at the top of the list are the registered individuals whose vectors are closest to and most similar to the vector to be identified in the feature space.

[0139] To improve the system's fault tolerance and retrieval efficiency, the system does not directly select the top-ranked result as the final answer. Instead, it selects a predetermined number of candidate options with the smallest distance as a candidate set. This predetermined number is usually set to 5 or 10. This candidate set retains the most likely options, providing a buffer for subsequent fine-tuning.

[0140] For example, in one identification process, the top three individuals with the corrected vector distance to be identified are "Person A" (distance 0.5), "Person B" (distance 0.52), and "Person C" (distance 0.8). The system constructs a candidate set from these three. This mechanism is very useful in practice, especially when the characteristics of the person to be identified are very similar to those of people in the database (such as twins), or when the quality of the sample to be identified is generally not high enough to completely distinguish the top two. Providing a candidate set facilitates human-computer interaction for verification or secondary validation.

[0141] 1063. Determine whether the minimum distance in the candidate set is less than the judgment threshold. If yes, output the corresponding identity tag; otherwise, output the recognition failure signal.

[0142] The system has set a threshold for judgment. This threshold is derived from a large amount of experimental data and serves as the dividing line between "the same person" and "not the same person".

[0143] The system reads the corrected distance (i.e., minimum distance) of the candidate ranked first in the candidate set. The system compares this minimum distance with the decision threshold. The comparison logic is as follows: (1) If the minimum distance is less than the judgment threshold, the system considers that the similarity between the vector to be identified and the registered vector has reached the standard of "the same person" and its confidence interval also meets the reliability requirements. At this time, the system outputs the identity label (such as "Employee ID: 12345") corresponding to the candidate as the final identification result, and allows the person to pass through the access control or trigger relevant records.

[0144] (2) If the minimum distance is greater than or equal to the judgment threshold, the system considers that even the most similar candidate has too large a difference from the vector to be identified, or the confidence interval of the vector to be identified is too wide, leading to distance inflation. In this case, the system determines that the current gait data is insufficient to support reliable identity recognition. For security reasons, the system refuses to output any identity label and instead outputs a "recognition failed" or "access denied" signal. In practical applications, this can trigger an auxiliary verification process, such as prompting "please face the camera and try again" or requiring fingerprint / card swipe auxiliary verification.

[0145] This threshold-based discrimination mechanism effectively controls the false recognition rate (i.e., correctly identifying the wrong person). By introducing distance based on uncertainty correction and a strict judgment threshold, the method in this embodiment can exhibit extremely high vigilance when facing spoofing attacks, severe obstruction, or unregistered personnel intrusion, ensuring the security and reliability of the gait recognition system.

[0146] Please see Figure 2 One embodiment of the multimodal gait recognition device of the present invention includes: The data acquisition module 10 is used to acquire raw gait data containing multimodal information, and to perform noise reduction processing on the raw gait data to obtain preprocessed data, while calculating the preliminary uncertainty index of the preprocessed data. The feature extraction module 20 is used to extract the spatiotemporal features of the preprocessed data and generate an uncertainty feature map that corresponds one-to-one with the spatiotemporal features based on the preliminary uncertainty index. The dynamic alignment module 30 is used to dynamically align the spatiotemporal features using the uncertainty feature map to obtain a standardized feature sequence; Adaptive fusion module 40 is used to construct adaptive fusion weights based on the uncertainty feature map, and use the adaptive fusion weights to perform multimodal fusion on the standardized feature sequence to obtain a fused feature vector; Uncertainty propagation module 50 is used to input the fused feature vector into the uncertainty propagation network and output an identity embedding vector with a confidence interval. The decision recognition module 60 is used to perform a search and matching in a preset database based on the identity embedding vector and output the final gait identity recognition result.

[0147] In one embodiment, the data acquisition module 10 is used to: A walking video stream is acquired by a multi-view sensor array, and inertial measurement unit data is acquired simultaneously. The walking video stream and the inertial measurement unit data are used as the raw gait data. Background modeling and foreground segmentation are performed on the walking video stream to extract the human silhouette image sequence, and the inertial measurement unit data is smoothed and filtered to generate the preprocessed data; The signal-to-noise ratio of the human silhouette image sequence and the acceleration variance of the inertial measurement unit data are calculated. The signal-to-noise ratio and the acceleration variance are weighted and mapped to obtain the preliminary uncertainty index.

[0148] In one embodiment, the feature extraction module 20 is used to: The preprocessed data is input into a dual-stream convolutional neural network to extract appearance texture features and skeletal motion dynamic features as the spatiotemporal features. An uncertainty estimation module based on Bayesian inference is constructed, and the preliminary uncertainty index is input into the uncertainty estimation module to learn the data noise distribution; Using the noise distribution parameters output by the uncertainty estimation module, probability mapping is performed on each feature dimension of the spatiotemporal features to generate the uncertainty feature map.

[0149] In one embodiment, the dynamic alignment module 30 is used for: Detect key points of gait cycles in the spatiotemporal features, and remove outliers with uncertainty higher than a preset threshold from the key points based on the uncertainty feature map; Dynamic time warping is performed based on the retained valid key points to calculate the optimal matching path between different modal features; The spatiotemporal features are interpolated and aligned along the optimal matching path to fill in the positions of the removed outliers and generate the standardized feature sequence.

[0150] In one embodiment, the adaptive fusion module 40 is used to: The reliability score of each modality feature is calculated based on the uncertainty feature map, and the attention mechanism parameters are initialized using the reliability score. The cross-modal correlation matrix is ​​calculated using the attention mechanism parameters, and the adaptive fusion weights are updated based on the correlation matrix. The standardized feature sequence is weighted and summed using the updated adaptive fusion weights, and the original features are added to the fused features through residual connections to output a fused feature vector.

[0151] In one embodiment, the uncertainty propagation module 50 is used to: Construct a deep metric learning network that includes a random noise layer, and use the fused feature vector as the input to the deep metric learning network; During each forward propagation of the deep metric learning network, the network weights are Monte Carlo sampled using the random noise layer to simulate multiple prediction processes. The mean and variance of the outputs from the multiple prediction processes are statistically analyzed. The mean of the outputs is used as the main body of the identity embedding vector, and the variance of the outputs is transformed into a confidence interval. The identity embedding vectors with confidence intervals are then combined and output.

[0152] In one embodiment, the decision recognition module 60 is used for: Calculate the Euclidean distance between the identity embedding vector and the registration vector in the preset database, and correct the Euclidean distance by combining the confidence interval; The candidates in the preset database are sorted according to the corrected distance, and the top preset number of candidates with the smallest distance are selected as the candidate set. Determine whether the minimum distance in the candidate set is less than the judgment threshold. If yes, output the corresponding identity tag; otherwise, output a recognition failure signal.

[0153] Since the embodiments of the device part correspond to the embodiments of the above method, the description of the multimodal gait recognition device provided by the present invention should refer to the above method embodiments. The present invention will not be described again here, but it has the same beneficial effects as the above multimodal gait recognition method.

[0154] Figure 3 This is a schematic diagram of the structure of a computer device 500 provided in an embodiment of the present invention. The computer device 500 can vary significantly due to different configurations or performance characteristics. It may include one or more central processing units (CPUs) 510 (e.g., one or more processors) and a memory 520, and one or more storage media 530 (e.g., one or more mass storage devices) for storing application programs 533 or data 532. The memory 520 and storage media 530 can be temporary or persistent storage. The program stored in the storage media 530 may include one or more modules (not shown in the diagram), each module including a series of instruction operations on the computer device 500. Furthermore, the processor 510 may be configured to communicate with the storage media 530 and execute the series of instruction operations in the storage media 530 on the computer device 500.

[0155] Computer device 500 may also include one or more power supplies 540, one or more wired or wireless network interfaces 550, one or more input / output interfaces 560, and / or one or more operating systems 531, such as Windows Server, Mac OS X, Unix, Linux, FreeBSD, etc. Those skilled in the art will understand that... Figure 3 The computer device structure shown does not constitute a limitation on the computer device and may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0156] The present invention also provides a computer device, the computer device including a memory and a processor, wherein the memory stores computer-readable instructions, and when the computer-readable instructions are executed by the processor, the processor performs the steps of the multimodal gait recognition method described in the above embodiments. The present invention also provides a computer-readable storage medium, which can be a non-volatile computer-readable storage medium or a volatile computer-readable storage medium, wherein the computer-readable storage medium stores instructions, and when the instructions are executed on a computer, the computer performs the steps of the multimodal gait recognition method described in the above embodiments.

[0157] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0158] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0159] The above-described embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A multimodal gait recognition method, characterized in that, include: Obtain raw gait data containing multimodal information, and perform denoising processing on the raw gait data to obtain preprocessed data, while calculating the preliminary uncertainty index of the preprocessed data; Extract the spatiotemporal features of the preprocessed data, and generate an uncertainty feature map that corresponds one-to-one with the spatiotemporal features based on the preliminary uncertainty index; The spatiotemporal features are dynamically aligned using the uncertainty feature map to obtain a standardized feature sequence; An adaptive fusion weight is constructed based on the uncertainty feature map, and the standardized feature sequence is fused using the adaptive fusion weight to obtain a fused feature vector. The fused feature vector is input into the uncertainty propagation network, which outputs an identity embedding vector with a confidence interval. The identity embedding vector is used to search and match in a preset database, and the final gait identity recognition result is output.

2. The multimodal gait recognition method according to claim 1, characterized in that, The process of acquiring raw gait data containing multimodal information, denoising the raw gait data to obtain preprocessed data, and simultaneously calculating the preliminary uncertainty index of the preprocessed data includes: Walking video streams are acquired using a multi-view sensor array, and inertial measurement unit data is acquired simultaneously. The walking video streams and the inertial measurement unit data are used as the raw gait data. Background modeling and foreground segmentation are performed on the walking video stream to extract the human silhouette image sequence, and the inertial measurement unit data is smoothed and filtered to generate the preprocessed data; The signal-to-noise ratio of the human silhouette image sequence and the acceleration variance of the inertial measurement unit data are calculated. The signal-to-noise ratio and the acceleration variance are weighted and mapped to obtain the preliminary uncertainty index.

3. The multimodal gait recognition method according to claim 1, characterized in that, The step of extracting the spatiotemporal features of the preprocessed data and generating an uncertainty feature map corresponding one-to-one with the spatiotemporal features based on the preliminary uncertainty index includes: The preprocessed data is input into a dual-stream convolutional neural network to extract appearance texture features and skeletal motion dynamic features, which are used as the spatiotemporal features. An uncertainty estimation module based on Bayesian inference is constructed, and the preliminary uncertainty index is input into the uncertainty estimation module to learn the data noise distribution; Using the noise distribution parameters output by the uncertainty estimation module, probability mapping is performed on each feature dimension of the spatiotemporal features to generate the uncertainty feature map.

4. The multimodal gait recognition method according to claim 1, characterized in that, The step of dynamically aligning the spatiotemporal features using the uncertainty feature map to obtain a standardized feature sequence includes: Detect key points of gait cycles in the spatiotemporal features, and remove outliers with uncertainty higher than a preset threshold from the key points based on the uncertainty feature map; Dynamic time warping is performed based on the retained valid key points to calculate the optimal matching path between different modal features; The spatiotemporal features are interpolated and aligned along the optimal matching path to fill in the positions of the removed outliers and generate the standardized feature sequence.

5. The multimodal gait recognition method according to claim 1, characterized in that, The adaptive fusion weights constructed based on the uncertainty feature map, and the multimodal fusion of the standardized feature sequence using the adaptive fusion weights to obtain the fused feature vector, include: The reliability score of each modality feature is calculated based on the uncertainty feature map, and the attention mechanism parameters are initialized using the reliability score. The cross-modal correlation matrix is ​​calculated using the attention mechanism parameters, and the adaptive fusion weights are updated based on the correlation matrix. The standardized feature sequence is weighted and summed using the updated adaptive fusion weights, and the original features are added to the fused features through residual connections to output a fused feature vector.

6. The multimodal gait recognition method according to claim 1, characterized in that, The step of inputting the fused feature vector into the uncertainty propagation network and outputting an identity embedding vector with a confidence interval includes: Construct a deep metric learning network that includes a random noise layer, and use the fused feature vector as the input to the deep metric learning network; During each forward propagation of the deep metric learning network, the network weights are Monte Carlo sampled using the random noise layer to simulate multiple prediction processes. The mean and variance of the outputs from the multiple prediction processes are statistically analyzed. The mean of the outputs is used as the main body of the identity embedding vector, and the variance of the outputs is transformed into a confidence interval. The identity embedding vectors with confidence intervals are then combined and output.

7. The multimodal gait recognition method according to any one of claims 1-6, characterized in that, The step of searching and matching in a preset database based on the identity embedding vector to output the final gait identity recognition result includes: Calculate the Euclidean distance between the identity embedding vector and the registration vector in the preset database, and correct the Euclidean distance by combining the confidence interval; The candidates in the preset database are sorted according to the corrected distance, and the top preset number of candidates with the smallest distance are selected as the candidate set. Determine whether the minimum distance in the candidate set is less than the judgment threshold. If yes, output the corresponding identity label; otherwise, output a recognition failure signal.

8. A multimodal gait recognition device, characterized in that, include: The data acquisition module is used to acquire raw gait data containing multimodal information, and to perform noise reduction processing on the raw gait data to obtain preprocessed data, while calculating the preliminary uncertainty index of the preprocessed data. The feature extraction module is used to extract the spatiotemporal features of the preprocessed data and generate an uncertainty feature map that corresponds one-to-one with the spatiotemporal features based on the preliminary uncertainty index. The dynamic alignment module is used to dynamically align the spatiotemporal features using the uncertainty feature map to obtain a standardized feature sequence; An adaptive fusion module is used to construct adaptive fusion weights based on the uncertainty feature map, and to perform multimodal fusion on the standardized feature sequence using the adaptive fusion weights to obtain a fused feature vector; The uncertainty propagation module is used to input the fused feature vector into the uncertainty propagation network and output an identity embedding vector with a confidence interval. The decision recognition module is used to perform a search and matching in a preset database based on the identity embedding vector, and output the final gait identity recognition result.

9. A computer device, characterized in that, The computer device includes: a memory and at least one processor, the memory storing instructions; the at least one processor invokes the instructions in the memory to cause the computer device to perform the multimodal gait recognition method as described in any one of claims 1-7.

10. A computer-readable storage medium storing instructions thereon, characterized in that, When the instructions are executed by the processor, they implement the multimodal gait recognition method as described in any one of claims 1-7.

Images