Three-dimensional coordinate estimation and gait parameter measurement method based on multi-view time sequence model
By fusing video data collected by RGB cameras using a multi-view temporal model, the problem of high hardware cost and poor adaptability in existing technologies is solved, achieving high-precision and low-cost gait parameter measurement, which is suitable for various scenarios such as homes, hospitals, and sports venues.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-12-04
- Publication Date
- 2026-03-13
AI Technical Summary
Existing methods for measuring human gait parameters suffer from high hardware costs, poor adaptability, low comfort, and high data processing complexity. In particular, it is difficult to achieve high-precision gait parameter measurement in low-light or complex environments.
A multi-view temporal model-based approach is adopted, which uses multiple RGB cameras to collect video data, performs frame-level processing and temporal feature extraction, and eliminates depth ambiguity through multi-view data fusion, thereby achieving low-cost gait parameter measurement without special hardware.
It enables high-precision and low-cost gait parameter measurement in various scenarios, reduces hardware dependence, improves system adaptability and comfort, and simplifies operation and maintenance.
Smart Images
Figure CN121661709A_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of computer vision technology, and relates to human motion analysis technology and computer image recognition, specifically to a method for three-dimensional coordinate estimation and gait parameter measurement based on a multi-view temporal model. Background Technology
[0002] Existing methods for measuring human gait parameters include those based on inertial sensors, those relying on infrared generators and infrared sensors, and those based on video analysis.
[0003] Sensor-based solutions utilize devices equipped with various sensors, including accelerometers, gyroscopes, and pressure sensors, worn on key areas of the body such as the ankles, knees, and lower back. These devices capture real-time changes in physical quantities generated during human movement, and the data is analyzed to derive gait parameters such as stride length, cadence, gait speed, and joint angles. In some medical rehabilitation settings, such devices are used to monitor gait changes during patient rehabilitation to assess rehabilitation effectiveness.
[0004] The measurement scheme relying on an infrared generator and infrared capture devices works by using an infrared generator to emit infrared light, which is then irradiated onto markers placed on anatomical positions on the human body. Reflective paper covering the markers reflects the infrared light back to the infrared capture devices, which then receive the infrared signals. When two or more infrared capture devices simultaneously capture the infrared signal returned from a single marker, the three-dimensional coordinates of that marker can be determined. Furthermore, the kinematic parameters of the human body can be calculated using these coordinates, thus obtaining gait parameters. This type of high-precision motion capture system is commonly used in film and animation production, industrial robot motion simulation, and other fields to acquire accurate human motion data.
[0005] While gait measurement methods based on sensors and infrared sensors can provide relatively accurate results, they have several significant drawbacks. First, these methods typically require specialized hardware and complex installation processes, leading to high costs. Second, these systems are often limited by specific locations and environments; sensor performance can be affected by factors such as wearing position and body movement patterns, while infrared sensors are limited by field of view and lighting conditions, resulting in poor system adaptability. More importantly, sensors and sensors usually need to be in direct contact with the human body, potentially affecting user comfort and natural movement.
[0006] Video analytics-based solutions collect video data using cameras installed in specific locations, and then extract gait-related features using image processing techniques. Common feature extraction methods include contour-based methods, optical flow-based methods, and deep learning-based methods. Contour-based methods segment the silhouette of a pedestrian from video frames and further extract the geometric features of the silhouette; optical flow-based methods obtain the human body's motion state and gait features by calculating the motion information of pixels between adjacent frames in the video; deep learning-based methods have developed rapidly in recent years, training neural network models with large amounts of labeled data to automatically learn more abstract and representative gait features, enabling the identification of pedestrians or the measurement of gait parameters, and have wide applications in security monitoring, athlete training analysis, and other scenarios.
[0007] Compared to sensor- and infrared capture-based solutions, video analytics-based gait measurement methods offer several advantages. First, these methods eliminate the need for any worn hardware, allowing users to measure gait without physical limitations, thus improving system comfort and convenience. Furthermore, video analytics typically relies on existing camera equipment, reducing the cost of purchasing and maintaining expensive hardware. Second, video analytics methods are highly adaptable and can be applied in various environments, both indoors and outdoors, even complex ones, where video data acquisition is relatively easy. Additionally, video analytics enables remote monitoring, avoiding the limitations of sensors and capture devices requiring direct contact with the user's body.
[0008] However, the quality of video data is significantly affected by factors such as lighting, camera angle, and background complexity, especially in low-light or complex environments, which increases the difficulty of image processing and feature extraction. Secondly, video analytics algorithms typically require large amounts of training data; deep learning models require high-quality labeled data for training, and the training process may involve substantial computational resources, increasing the cost of system implementation and maintenance. Furthermore, video analytics methods have high requirements for real-time performance and processing capabilities; real-time processing of video data can introduce latency issues, affecting the system's response speed. Summary of the Invention
[0009] To address the shortcomings of existing technologies, this invention proposes a three-dimensional coordinate estimation and gait parameter measurement method based on a multi-view temporal model. Using multi-view video sequences captured by ordinary cameras as the core data source, the method first achieves accurate positioning and tracking of key human joints by performing frame-level processing and temporal feature extraction on the video sequences. Then, based on multi-view data fusion, the depth ambiguity of a single view is eliminated, ultimately achieving gait parameter measurement that requires no special hardware, is low-cost, and has strong universality.
[0010] The three-dimensional coordinate estimation method based on a multi-view temporal model includes the following steps:
[0011] Step 1. Multi-view video acquisition: Use multiple RGB cameras to simultaneously acquire gait video data from different angles to ensure comprehensive capture of all stages of the gait process and guarantee the synchronization and high quality of the video data.
[0012] Step 2. Video preprocessing: Perform frame synchronization calibration on the acquired multi-view videos, remove blurry frames and severely occluded frames, improve frame image quality, and lay the foundation for subsequent key point extraction.
[0013] Step 3. Camera calibration and coordinate transformation: Perform internal and external parameter calibration of the camera to ensure that multi-view video data is unified into a single coordinate system.
[0014] Step 4. Multi-view temporal model construction: Using multi-view image sequences and camera parameters as input, the image patches from different viewpoints are fused using the camera's intrinsic and extrinsic parameters. Then, spatial location coding, temporal location coding, and viewpoint location coding are introduced. After passing through a multi-layer Transformer structure, the effective fusion of multi-view spatiotemporal features is achieved through a self-attention mechanism.
[0015] Seventeen learnable query vectors are introduced, each corresponding to a 3D keypoint. Through a cross-attention mechanism using a continuous Transformer structure, each query vector adaptively extracts relevant information from the multi-view features output by the encoder, progressively extracting high-dimensional feature representations of the keypoints. Finally, a two-layer linear projection network with decreasing dimensionality maps the high-dimensional feature representation of each keypoint to 3D coordinates.
[0016] Step 5. Multi-view temporal model training: Train the model based on multi-view video data to optimize the gait parameter prediction capability.
[0017] A gait parameter measurement method based on a multi-view temporal model is proposed, which calculates gait parameters using the 3D keypoint temporal sequence output by the above method.
[0018] The present invention has the following beneficial effects:
[0019] Low hardware dependency and strong versatility: No special hardware such as infrared generators and infrared capture devices are required. Measurement can be achieved with just a regular RGB camera. Consumer-grade cameras can be used. The hardware cost is low. It can be deployed in various scenarios such as homes, hospitals, sports venues, and outdoors, solving the problem of limited scenarios for traditional infrared systems.
[0020] High data accuracy and reliability: It integrates multi-view data with advanced algorithms in the field of human pose estimation. The accuracy of key joint detection is better than that of traditional single-view video analysis methods. Furthermore, it eliminates depth ambiguity and improves the accuracy of gait parameter calculation by using temporal modeling and multi-view fusion.
[0021] Easy to operate and maintain: No professional personnel are required for complex hardware installation and debugging. The camera placement only requires simple fixing and angle calibration. Routine maintenance only requires periodic checks on the camera's power supply and storage status, reducing maintenance costs.
[0022] It has a wide range of applications: it can be directly applied to gait assessment for stroke patients, gait optimization for athletes, gait identity authentication and other fields. Compared with traditional methods, it can better meet the needs of low cost, high precision and easy deployment for gait parameter measurement in different scenarios. Attached Figure Description
[0023] Figure 1 This is a schematic diagram of the multi-view temporal model structure;
[0024] Figure 2 This is a schematic diagram of the encoding module;
[0025] Figure 3 This is a schematic diagram of the decoding module;
[0026] Figure 4 This is an example image for keypoint detection. Detailed Implementation
[0027] The present invention will be further explained below with reference to the accompanying drawings;
[0028] Step 1: Multi-view video acquisition
[0029] First, two RGB cameras with a resolution of 1280×720 and a frame rate of 60fps are installed on the front and side of the gait measurement area to ensure that the complete gait cycle can be captured. The gait measurement area is set as a standard walkway of 5-10 meters in length to ensure complete recording of the gait process.
[0030] To supplement and enrich the training data of the model and improve the generalization ability and accuracy of the system, data from the Human3.6M public dataset was used to supplement the collected data.
[0031] Step 2: Video Preprocessing
[0032] By employing a precise timestamp matching method, frame synchronization calibration is performed on video data from two cameras at different perspectives to ensure strict alignment of data across the timeline, eliminating time deviations caused by hardware differences or startup delays. The synchronized video data can be seamlessly fused, enabling precise matching of motion and scene information from each perspective. This avoids visual misalignment or data inconsistencies due to time differences, further improving the accuracy of subsequent analysis.
[0033] Step 3: Camera Calibration and Coordinate Transformation
[0034] First, each camera is calibrated using a checkerboard calibration method to obtain the camera's intrinsic and extrinsic parameter matrices, which are used for subsequent image coordinate transformation.
[0035]
[0036] Where K represents the camera intrinsic parameter matrix, and It's the focal length. and These are the coordinates of the center point of the image.
[0037] Image coordinates of each camera are obtained using camera calibration data. Convert to world coordinate system coordinates :
[0038]
[0039] in, It is the homography matrix obtained through camera calibration.
[0040] Step 4: Construction of multi-view time series model
[0041] Build a Figure 1 The view-time model based on the multi-view spatiotemporal Transformer shown takes multi-view image sequences and camera parameters as input, and directly outputs 3D keypoint coordinates through hierarchical feature extraction and spatiotemporal fusion, realizing end-to-end mapping from the original image to 3D pose:
[0042] s4.1 Image Patch Embedding Processing
[0043] For consecutive video frames from two different viewpoints, each frame is first segmented into 16×16 pixel non-overlapping image blocks. Each image block is then encoded into a 768-dimensional feature vector using linear projection, converting the original image into a feature sequence while preserving spatial structure information. For a 224×224 resolution image, 196 image blocks are generated per frame, resulting in a total of 2×T×196 feature vectors from the T frames from both viewpoints.
[0044] s4.2 Multi-view spatiotemporal feature coding
[0045] like Figure 2 As shown, image patches from different viewpoints are fused using the camera's intrinsic and extrinsic parameters. Then, a triple positional encoding is introduced: spatial positional encoding preserves the spatial relationships between image patches, temporal positional encoding captures inter-frame temporal dependencies, and viewpoint positional encoding distinguishes between different camera sources. A 12-layer Transformer structure is used as the encoder, with each layer containing a 12-head self-attention mechanism. Through the self-attention mechanism, the model simultaneously learns the temporal relationships within the same viewpoint and the spatial constraints between different viewpoints, achieving effective fusion of multi-view spatiotemporal features.
[0046] S4.3, 3D Keypoint Decoding
[0047] like Figure 3 As shown, 17 learnable query vectors are introduced, each corresponding to a 3D keypoint. An 8-layer Transformer structure is used as the decoder. Through a cross-attention mechanism, each query vector adaptively extracts relevant information from the multi-view features output by the encoder, progressively extracting high-dimensional feature representations of the keypoints. The decoding process fully utilizes the geometric consistency constraints between multiple views, effectively solving the depth blur problem under a single view.
[0048] s4.4, Coordinate Regression Output
[0049] Finally, through a two-layer linear projection network and a dimensionality-reducing design, the high-dimensional feature representation of each keypoint is mapped to 3D coordinates, such as... Figure 4 As shown.
[0050] The end-to-end architecture avoids the error accumulation of traditional two-stage methods. Multi-view spatiotemporal joint attention enables effective fusion of information between viewpoints, and a query-based decoding mechanism allows the model to adaptively learn keypoint features. Compared to traditional methods, significant improvements are achieved in accuracy, robustness, and efficiency, providing a reliable foundation for subsequent 3D pose analysis.
[0051] Step 5: Training the multi-view temporal model
[0052] The multi-view temporal model was trained using the multi-view videos acquired in step 1, combined with the average joint position error. and bone length constraint loss Set the composite loss function :
[0053]
[0054] The weighting coefficient λ is used to balance the keypoint accuracy and the skeleton proportion constraint, and in this embodiment it is set to 0.5.
[0055]
[0056]
[0057] in, For the predicted key point 3D coordinates, For the actual coordinates, For batch size, The length of the time series. Number of key points. For a predefined set of skeletal connections, The standardized bone length between key points i and j maintains the reasonable proportion of the human skeleton.
[0058] The AdamW optimizer was used for parameter optimization, with an initial learning rate of 1e-4 and a weight decay coefficient of 0.05 to prevent overfitting. Cosine annealing was used to dynamically adjust the learning rate, and the entire training process lasted for 300 epochs. The batch size was set to 16 samples, and the batch size was effectively increased by 4-step gradient accumulation. A gradient clipping strategy with a maximum gradient norm of 1.0 was used to ensure training stability, and FP16 mixed precision training was used to accelerate the computation process while maintaining numerical accuracy.
[0059] Finally, AP, AP50, and AP75 were used as metrics to evaluate the performance of this method compared with other methods. The results are shown in Table 1.
[0060] Table 1
[0061]
[0062] Experimental results show that the proposed method outperforms the comparison method in three metrics: AP, AP50, and AP75, demonstrating the effectiveness and superiority of the proposed multi-view temporal model in 3D keypoint detection tasks.
[0063] The gait parameter measurement method based on the multi-view temporal model calculates gait parameters using the 3D keypoint temporal sequence output by the above method. Through gait cycle segmentation and step event detection, the step length and step frequency within each gait cycle are calculated to obtain the individual's motion characteristics for subsequent analysis.
Claims
1. A three-dimensional coordinate estimation method based on a multi-view temporal model, which synchronizes gait video data collected from multiple RGB cameras from different angles, inputs it into a deep learning model, and estimates the three-dimensional coordinates of key points of the human body, characterized by: A multi-view temporal model is constructed, using gait video data collected from different angles after synchronization and the intrinsic and extrinsic parameters of the calibrated camera as model inputs. The intrinsic and extrinsic parameters are used to fuse image frames from different viewpoints. Then, spatial position coding, temporal position coding, and viewpoint position coding are introduced. Finally, a multi-layer Transformer structure is used to achieve effective fusion of multi-view spatiotemporal features and complete the coding. Seventeen learnable query vectors are introduced, each corresponding to a 3D keypoint. Through the cross-attention mechanism of the continuous Transformer structure, each query vector adaptively extracts relevant information from the multi-view features output by the encoder, and gradually extracts the high-dimensional feature representation of the keypoint to complete the decoding. By using a two-layer linear projection network with decreasing dimensionality, the high-dimensional feature representation of each key point is mapped to 3D coordinates, thus completing the 3D coordinate estimation of human body key points.
2. The three-dimensional coordinate estimation method based on a multi-view temporal model as described in claim 1, characterized in that: Frame synchronization calibration is performed on video data from different perspectives obtained from multiple cameras to eliminate time deviations caused by hardware differences or startup delays, and blurry frames and severely occluded frames are removed before being input into the deep learning model frame by frame.
3. The three-dimensional coordinate estimation method based on a multi-view temporal model as described in claim 2, characterized in that: For consecutive video frames from different perspectives, each frame image from each perspective is first divided into non-overlapping image blocks of 16×16 pixels. Each image block is encoded into a 768-dimensional feature vector through linear projection, converting the original image into a feature sequence while preserving spatial structure information.
4. The three-dimensional coordinate estimation method based on a multi-view temporal model as described in claim 1, characterized in that: The camera was calibrated using the checkerboard calibration method to obtain the intrinsic and extrinsic parameter matrices.
5. The three-dimensional coordinate estimation method based on a multi-view temporal model as described in claim 1, characterized in that: Set the composite loss function Training a multi-view time series model: The weighting coefficient λ is used to balance the accuracy of keypoints with the constraints of skeleton proportions. For the predicted key point 3D coordinates, For the actual coordinates, For batch size, The length of the time series. Number of key points; For a predefined set of skeletal connections, The standardized bone length between keypoints i and j.
6. The three-dimensional coordinate estimation method based on a multi-view temporal model as described in claim 5, characterized in that: The AdamW optimizer was used to optimize the model parameters. The initial learning rate was set to 1e-4 with a weight decay coefficient of 0.
05. At the same time, the learning rate was dynamically adjusted using a cosine annealing strategy. The batch size is set to 16 samples, and the batch size is effectively increased by 4-step gradient accumulation. A gradient pruning strategy with a maximum gradient norm of 1.0 is adopted to ensure training stability, and FP16 mixed precision training is used to accelerate the calculation process while maintaining numerical accuracy.
7. A computer-readable storage medium having a computer program stored thereon, which, when executed in a computer, causes the computer to perform the method of any one of claims 1 to 6.
8. A gait parameter measurement method based on a multi-view temporal model, characterized in that: By using the method described in any one of claims 1 to 6, the three-dimensional coordinate temporal sequence of human body key points is estimated from multi-view video images, and gait parameters are calculated.