A portable gait analysis system and method based on deep learning and fusion of multi-modal data

Abstract

The application discloses a kind of based on deep learning and the fusion portable gait analysis system and method of multi-modal data, it is related to medical rehabilitation technical field, including sequentially connected data acquisition layer, AI posture detection layer, post-processing layer, noise filtering layer, gait analysis layer and data output layer.System acquires RGB image, depth image and infrared image synchronously by depth camera, detects human key points and restores three-dimensional coordinates by using deep learning model after pre-processing, automatically divides gait cycle after combining marker point calibration, multi-stage filtering denoising, calculates joint angle, finally outputs motion data, visual video and stores record.The application can realize high-precision, stable and reliable gait analysis, and the system design is portable and easy to use, suitable for rehabilitation evaluation and motion monitoring scene.

Description

Technical Field

[0001] This invention relates to the field of medical rehabilitation technology, and more specifically to a portable gait analysis system and method based on the fusion of deep learning and multimodal data. Background Technology

[0002] In the fields of clinical rehabilitation medicine and sports medicine, accurate analysis of human gait is of great significance. Currently, traditional gait analysis methods all have significant shortcomings in practical applications, failing to meet the demands for low-cost, convenient, and high-precision gait analysis in clinical and sports settings. Specifically: (1) Traditional optical motion capture systems are extremely expensive, usually costing millions of yuan and requiring multiple professional operators. When using them, several specially designed reflective markers need to be pasted at multiple angles on each key point of the subject. They also rely on a dedicated large-area motion capture room, which is not suitable for ordinary hospitals and is difficult to popularize.

[0003] (2) The inertial sensor solution has the problem of large sensor size, which will interfere with the patient's natural gait; data acquisition requires dedicated synchronization equipment, and the equipment cost is still hundreds of thousands of yuan; at the same time, electromagnetic compatibility design to adapt to the medical environment is required, and there are currently no mature products on the market that meet the requirements, so the practicality is insufficient.

[0004] (3) The single-channel color camera video analysis scheme lacks depth information, cannot accurately recover the three-dimensional motion trajectory of the human body, and has low accuracy in calculating joint angles; moreover, the analysis results are easily affected by ambient light, shadows, patient clothing style and body shape, resulting in poor stability and accuracy. On this basis, the manual K-frame method is added, but because the video data volume is as high as thousands of frames, it is impossible to quickly complete the manual correction of key points, resulting in extremely low analysis efficiency.

[0005] (4) The multi-channel color camera video analysis solution also requires a dedicated dynamic compensation room, which is not suitable for ordinary hospitals. This solution will occupy a large amount of storage resources and huge computing power. Moreover, multi-channel video transmission requires the construction of a dedicated high-bandwidth, low-latency video transmission network and time synchronization network. The cost of use is high, ranging from hundreds of thousands to millions of yuan, making it difficult to promote.

[0006] In exoskeleton-assisted rehabilitation assessment scenarios, therapists need to evaluate patients' gait changes before, during, and after exoskeleton intervention, quantify the exoskeleton's gait-assisting effect, and objectively record rehabilitation progress to form a clinical report. This requires gait analysis systems to achieve: low-cost deployment, requiring only consumer-grade or basic industrial equipment for setup; ease of operation, eliminating the need for complex calibration and preparation procedures; objective and quantifiable assessment results, outputting repeatable numerical indicators; and a micro-contact design, requiring only simple calibration points and not affecting the patient's natural gait.

[0007] Based on the aforementioned business needs and scenario requirements, this field urgently needs a novel gait analysis solution to simultaneously address multiple technical challenges: First, how to accurately recover the 3D coordinates of key human body points using depth maps and camera intrinsic parameters, thus eliminating the sole reliance on 2D images; second, how to address the occasional missed key point detection issues in deep learning models and ensure data continuity; third, how to suppress key point trajectory noise while preserving the peak positions of gait features from smooth shifts; fourth, how to automatically detect key events such as heel strikes from complex motion data and achieve automatic gait cycle segmentation; and fifth, how to control overall costs and achieve low-cost operation and widespread application of the system. Summary of the Invention

[0008] In view of this, the present invention provides a portable gait analysis system and method based on the fusion of deep learning and multimodal data, which solves the problems existing in the background technology.

[0009] To achieve the above objectives, the present invention provides the following technical solution: A portable gait analysis system based on the fusion of deep learning and multimodal data includes a data acquisition layer, an AI posture detection layer, a post-processing layer, a noise filtering layer, a gait analysis layer, and a data output layer connected in sequence. The data acquisition layer is used to simultaneously acquire RGB images, depth images, and infrared images through a depth camera, and output multimodal image data to the AI ​​pose detection layer after preprocessing. The AI ​​pose detection layer is used to detect human pose based on the pre-processed RGB image using a visual deep learning model, extract the pixel coordinates of key points, and then restore the three-dimensional coordinates of key points by combining them with the pre-processed depth image before outputting them to the post-processing layer. The post-processing layer is used to calibrate the three-dimensional coordinates of key points by combining the markers worn by the patient, and outputs the calibrated key point coordinate sequence to the noise filtering layer. The noise filtering layer is used to reduce the noise of the key point coordinate sequence by using a two-stage cascaded method of median filtering and Savitzky-Golay filtering, and outputs the smoothed key point coordinate sequence to the gait analysis layer. The gait analysis layer is used to automatically divide the gait cycle and calculate joint angles based on the smoothed key point coordinate sequence, and output the gait analysis results to the data output layer. The data output layer is used to output JSON-formatted motion data based on gait analysis results, generate multi-view skeleton visualization videos, and store patient records in an SQLite database.

[0010] Optionally, the preprocessing operations of the data acquisition layer include: aligning the depth image to the RGB image viewpoint, and then performing transpose processing, flip processing, and white balance calibration processing on the frame-aligned RGB image, depth image, and infrared image to obtain preprocessed multimodal image data.

[0011] Optionally, the AI ​​pose detection layer includes a human key point detection module, a human tracking module, and a 3D coordinate recovery module connected in sequence. The human key point detection module is used to identify the human position and pose from the preprocessed RGB image, and output the human bounding box and the pixel coordinates and confidence scores of multiple key points; The human body tracking module is used to track detected human bodies across frames using the Byte Track algorithm, assigning a unique identifier to the same human body and selecting the human body bounding box with the largest area as the main analysis target. The 3D coordinate recovery module is used to convert the pixel coordinates of key points into three-dimensional spatial coordinates based on the intrinsic parameters of the depth camera and the depth values ​​corresponding to the pixels, using a pinhole camera model.

[0012] Optionally, the post-processing layer includes a marker detection module, a missing value imputation module, and a calibration offset module connected in sequence; The marker detection module is used to perform dual-range red detection in the HSV color space to locate the pixel position of the red spherical marker. The missing value imputation module is used to perform linear interpolation calculations on the effective key point data before and after the missing point in the time dimension to obtain reasonable replacement values ​​for imputation in order to maintain the continuity of the key point coordinate sequence. The calibration offset module is used to set depth offsets for key points corresponding to the shoulder, navel, hip, knee, and ankle in the interactive offset adjustment interface. The original depth value is added to the depth offset to obtain the calibrated depth value, so as to correct the measurement error of the depth camera.

[0013] Optionally, in the noise filtering layer, the impulse noise in the key point coordinate sequence is first removed by median filtering, and then the key point coordinate sequence is smoothed by Savitzky-Golay filtering to preserve its shape, so as to suppress high-frequency noise while keeping the peak position of gait features from shifting.

[0014] Optionally, the noise filtering layer also employs a one-dimensional Kalman filter to filter the key point coordinate sequence. The one-dimensional Kalman filter assumes that the system state follows a linear dynamic model and estimates the system state alternately through two steps: prediction and update.

[0015] Optionally, the gait analysis layer includes a gait cycle automatic segmentation module and a joint angle calculation module connected in sequence; The automatic gait cycle segmentation module is used to extract the ankle joint Y-axis coordinate sequence based on the three-dimensional trajectory of the ankle joint. After median filtering, Savitzky-Golay filtering and MAD normalization, the valley detection algorithm is used to identify heel strike events and the interval between adjacent heel strike times is divided into a gait cycle. The joint angle calculation module is used to calculate the angle between bone vectors based on the coordinates of key points, obtain the hip joint angle, knee joint angle and trunk tilt angle, and output the three-dimensional spatial angle as well as the projection angles of the XY plane, YZ plane and XZ plane.

[0016] A portable gait analysis method based on the fusion of deep learning and multimodal data, applied to any of the above-mentioned portable gait analysis systems based on the fusion of deep learning and multimodal data, includes the following steps: Multimodal image data is obtained by simultaneously acquiring RGB images, depth images, and infrared images using a depth camera and performing preprocessing. Human pose detection is performed based on the preprocessed RGB image using a visual deep learning model, key point pixel coordinates are extracted, and the three-dimensional coordinates of key points are recovered by combining the preprocessed depth image. The three-dimensional coordinates of key points are calibrated by combining the markers worn by the patient, resulting in a calibrated key point coordinate sequence; A two-stage cascaded method of median filtering and Savitzky-Golay filtering was used to denoise the key point coordinate sequence, resulting in a smoothed key point coordinate sequence. Gait cycles are automatically divided and joint angles are calculated based on the smoothed key point coordinate sequence to obtain gait analysis results; Based on the gait analysis results, output JSON format motion data, generate multi-view skeleton visualization videos, and store patient records in an SQLite database.

[0017] As can be seen from the above technical solution, compared with the prior art, the present invention discloses a portable gait analysis system and method based on the fusion of deep learning and multimodal data, which has the following beneficial effects: (1) By simultaneously acquiring RGB images, depth images and infrared images through a depth camera and performing precise preprocessing, combined with deep learning pose detection and ByteTrack tracking algorithm, stable recognition and tracking of human pose is achieved; by calibrating red marker points, linear interpolation of missing key points and depth offset compensation, the problems of inaccurate key point positioning and data loss caused by occlusion and detection errors are effectively solved, and the accuracy of three-dimensional coordinates is significantly improved.

[0018] (2) The median filter and Savitzky-Golay filter are used in series for noise reduction and shape-preserving smoothing. While removing noise, the peak values ​​of gait features are preserved, avoiding feature distortion and improving the smoothness and reliability of gait data.

[0019] (3) By detecting the valley value of the ankle joint trajectory, the gait cycle is automatically divided, and the three-dimensional angles and multi-plane projection angles of the hip joint, knee joint and trunk tilt angle are calculated simultaneously, realizing multi-dimensional and high-precision gait quantitative analysis.

[0020] (4) The overall system architecture is lightweight and portable, and can complete gait analysis without complex equipment, which reduces the cost of use and the limitations of the scenario. It is suitable for various scenarios such as clinical rehabilitation and daily exercise monitoring, and has the advantages of convenient deployment, stable detection and comprehensive analysis. Attached Figure Description

[0021] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0022] Figure 1 The architecture hierarchy diagram of the portable gait analysis system based on the fusion of deep learning and multimodal data provided by this invention; Figure 2 The camera initialization and configuration flowchart provided for this invention; Figure 3 The frame synchronization acquisition signaling diagram provided by this invention; Figure 4 The frame synchronization acquisition flowchart provided by this invention; Figure 5 The image preprocessing flowchart provided by this invention; Figure 6 Image preprocessing signaling diagram provided by the present invention; Figure 7 The flowchart for human key point detection provided by the present invention; Figure 8 This is a timing diagram of the AI ​​pose detection stage provided by the present invention; Figure 9 The 3D coordinate recovery flowchart provided by this invention; Figure 10 The flowchart for marker detection provided by this invention; Figure 11 Flowchart for linear interpolation to fill missing key points provided by the present invention; Figure 12 The timing diagram provided by the present invention for filling missing key points using linear interpolation; Figure 13 The flowchart for the application of depth calibration offset provided by this invention; Figure 14 The flowchart of the two-stage filtering algorithm provided by this invention; Figure 15 The flowchart of the Kalman filter algorithm provided by this invention; Figure 16 The timing diagram for the noise filtering stage provided by this invention; Figure 17 Here is a flowchart of the valley detection algorithm provided by this invention; Figure 18 The flowchart for calculating joint angles provided by this invention; Figure 19 The timing diagram for the gait analysis stage provided by this invention; Figure 20 The data output stage timing diagram provided for this invention; Figure 21 A schematic diagram of the calibration point sandwich structure provided by the present invention; Figure 22 This is a schematic diagram of the optical layer structure provided by the present invention. Detailed Implementation

[0023] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0024] To address the problems identified in the background technology, this invention proposes a technical approach of "multimodal data fusion + deep learning posture detection + automated calibration + signal processing" to achieve low-cost, portable gait acquisition and analysis. Specifically, this embodiment discloses a portable gait analysis system based on the fusion of deep learning and multimodal data, comprising a data acquisition layer, an AI posture detection layer, a post-processing layer, a noise filtering layer, a gait analysis layer, and a data output layer connected in sequence. The data acquisition layer is used to simultaneously acquire RGB images, depth images, and infrared images through a depth camera, and output multimodal image data to the AI ​​pose detection layer after preprocessing. Among them, the data streams of RGB images, depth images, and infrared images are strictly synchronized in time and calibrated in space by the factory, which effectively makes up for the lack of depth information in a monocular color camera. The AI ​​pose detection layer is used to detect human pose based on the pre-processed RGB image using a visual deep learning model, extract the pixel coordinates of key points, and then restore the three-dimensional coordinates of key points by combining them with the pre-processed depth image before outputting them to the post-processing layer. The post-processing layer is used to calibrate the three-dimensional coordinates of key points by combining the markers worn by the patient, and outputs the calibrated key point coordinate sequence to the noise filtering layer, which can help locate key medical points in the human body at a low cost. The noise filtering layer is used to reduce the noise of the key point coordinate sequence by using a two-stage cascaded method of median filtering and Savitzky-Golay filtering. The smoothed key point coordinate sequence is then output to the gait analysis layer, which can suppress noise while maintaining gait characteristics, thus resolving the contradiction between noise and shape preservation. The gait analysis layer is used to automatically divide the gait cycle and calculate joint angles based on the smoothed key point coordinate sequence, and output the gait analysis results to the data output layer. The data output layer is used to output JSON-formatted motion data based on gait analysis results, generate multi-view skeleton visualization videos, and store patient records in an SQLite database.

[0025] Next, the specific implementation scheme of the gait analysis system described in this invention will be detailed in detail, and will be explained layer by layer below. This system adopts a layered design, with each layer undertaking a specific function, and the layers are connected by data flow. See [link to documentation]. Figure 1 The diagram shows the hierarchical structure of the system architecture.

[0026] I. Data Acquisition Layer

[0027] In this embodiment, the preprocessing operation of the data acquisition layer includes: aligning the depth image to the RGB image viewpoint, and then performing transpose processing, flip processing, and white balance calibration processing on the frame-aligned RGB image, depth image, and infrared image to obtain preprocessed multimodal image data.

[0028] 1.1 Phase Overview

[0029] The data acquisition stage is the input source of the entire system, responsible for acquiring synchronized RGB images, depth images, and infrared images from the depth camera. The processing quality of this stage directly affects the accuracy of all subsequent analyses.

[0030] The following functions need to be implemented in this stage: 1) Initialization and configuration of the depth camera; 2) Synchronous acquisition of three data streams; 3) Frame alignment processing to ensure spatial correspondence between RGB images and depth images; 4) Image preprocessing, such as transpose, flip, and white balance calibration.

[0031] 1.2 Camera Initialization and Configuration

[0032] The depth camera is connected to the computing device via a USB interface. The system is configured to output three data streams simultaneously: RGB color stream for visual pose detection; depth stream for 3D coordinate recovery; and infrared stream for auxiliary analysis.

[0033] The configuration parameters are described in Table 1, and the camera initialization and configuration process is as follows: Figure 2 As shown, the frame synchronization acquisition signaling diagram is as follows: Figure 3 As shown.

[0034] Table 1 Configuration Parameters

[0035] 1.3 Frame Synchronization Acquisition

[0036] The depth camera internally implements a hardware-level frame synchronization mechanism to ensure that RGB, depth, and IR frames are acquired simultaneously. See also Figure 4 The process shown aligns the depth image to the RGB viewpoint by setting the alignment mode, ensuring that each RGB pixel can find its corresponding depth value.

[0037] 1.4 Image Preprocessing

[0038] The depth camera is mounted vertically, so the acquired images need to be rotated 90° and flipped horizontally to obtain the correct viewing angle. In addition, the system supports white balance calibration to adapt to different lighting environments.

[0039] The preprocessing steps are described in Table 2. The image preprocessing flow is as follows: Figure 5 As shown, the image preprocessing signaling diagram is as follows: Figure 6 As shown.

[0040] Table 2. Description of Preprocessing Steps

[0041] II. AI Pose Detection Layer

[0042] In this embodiment, the AI ​​pose detection layer includes a human key point detection module, a human tracking module, and a 3D coordinate recovery module connected in sequence. The human key point detection module is used to identify the human position and pose from the preprocessed RGB image, and output the human bounding box and the pixel coordinates and confidence scores of multiple key points; The human body tracking module is used to track detected human bodies across frames using the Byte Track algorithm, assigning a unique identifier to the same human body and selecting the human body bounding box with the largest area as the main analysis target. The 3D coordinate recovery module is used to convert the pixel coordinates of key points into three-dimensional spatial coordinates based on the intrinsic parameters of the depth camera and the depth values ​​corresponding to the pixels, using a pinhole camera model.

[0043] 2.1 Phase Overview

[0044] The AI ​​pose detection stage is the core analysis module of the system. It is responsible for detecting human pose from the preprocessed RGB image, extracting the pixel coordinates of key points, and combining the depth image to recover the three-dimensional spatial coordinates of the key points.

[0045] The following functions need to be implemented in this stage: 1) Human key point detection based on visual deep learning model; 2) Multi-target tracking based on Byte Track; 3) 3D coordinate recovery based on depth image.

[0046] 2.2 Human Key Point Detection

[0047] A visual deep learning model is a deep convolutional neural network model trained on a large amount of human pose data. It can identify the position and pose of a human body from an input RGB image and output the human body bounding box and the pixel coordinates and confidence scores of multiple key points.

[0048] The 20 key points are defined as shown in Table 3. The human body key point detection process is as follows: Figure 7 As shown, the timing diagram for the AI ​​pose detection stage is as follows: Figure 8 As shown.

[0049] Table 3 Key Point Definitions

[0050] 2.3 Coordinate Restoration

[0051] 3D coordinate restoration is one of the core technologies of this system, such as Figure 9 As shown, its principle is: based on the pinhole camera model, using the known intrinsic parameters of the camera (focal length) Main point The pixel coordinates can be obtained from the depth value of each pixel. Projected onto corresponding points in three-dimensional space .

[0052] Back projection formula (pixel coordinates → 3D coordinates):

[0053] III. Post-processing layer

[0054] In this embodiment, the post-processing layer includes a marker detection module, a missing value filling module, and a calibration offset module connected in sequence. The marker detection module is used to perform dual-range red detection in the HSV color space to locate the pixel position of the red spherical marker. The missing value imputation module is used to perform linear interpolation calculations on the effective key point data before and after the missing point in the time dimension to obtain reasonable replacement values ​​for imputation in order to maintain the continuity of the key point coordinate sequence. The calibration offset module is used to set depth offsets for key points corresponding to the shoulder, navel, hip, knee, and ankle in the interactive offset adjustment interface. The original depth value is added to the depth offset to obtain the calibrated depth value, so as to correct the measurement error of the depth camera.

[0055] 3.1 Phase Overview

[0056] The post-processing stage further processes the key point data output from the AI ​​detection stage, including operations such as red marker detection-assisted localization, missing key point interpolation and filling, and coordinate transformation.

[0057] The following functions need to be implemented in this stage: 1) Detection of red marker points, used to accurately locate key points such as the navel; 2) Linear interpolation, to fill in missing data in the key point sequence; 3) Coordinate system transformation (original_coord→flipped_coord).

[0058] 3.2 Design of Actively Illuminated Red Light Marker Points

[0059] Traditional optical motion capture systems use retro-reflective markers for joint positioning. This approach relies on the infrared spectrum, using a camera to capture the infrared light reflected from the markers to achieve 3D reconstruction. However, this passive optical design has significant drawbacks: firstly, the infrared light path is susceptible to interference from stray light in the environment, resulting in poor signal-to-noise ratio stability; secondly, it has strict requirements regarding the clothing of the captured object—dark clothing absorbs light, while light-colored clothing reflects light, both of which will lead to marker recognition failure. Therefore, traditional solutions must be implemented in a specialized motion capture room, and operators must wear customized motion capture suits and attach highly reflective infrared markers to specific locations, resulting in high deployment costs and operational complexity.

[0060] To address the aforementioned pain points, this embodiment proposes an active LED marker based on visible light, fundamentally eliminating the reliance on infrared reflection. This marker employs... Figure 21 The sandwich structure design shown: See the structure of the optical layer (outer layer). Figure 22It is composed of a semi-transparent polylactic acid (PLA) diffuser and a transparent copolyester (PETG) diffuser, which can convert point light sources into surface light sources and significantly improve the signal-to-noise ratio against complex backgrounds; Circuit layer (middle layer): integrates high-brightness red LEDs (wavelength 620-625nm, light intensity 2000-3000mcd, power consumption 0.06W, operating current 20mA, forward voltage 1.8-2.0V), independently powered by CR2032 button batteries; Fixed layer (bottom layer): Secured to the middle layer with screws, it holds flexible elastic straps to achieve self-adaptive wear and fit different body types.

[0061] Mechanical screws are used to connect the layers to ensure structural stability and maintainability.

[0062] 3.3 Inexpensive Red Marker Detection

[0063] Inexpensive red spherical markers worn on the human body exhibit a distinct red characteristic in the HSV color space. By performing dual-range red detection in the HSV color space, the pixel positions of these markers can be quickly and accurately located.

[0064] Explanation of the red HSV range: Since red spans both 0 degrees and 180 degrees in the HSV space, two detection ranges need to be defined: 1) Scope 1: Suitable for reds close to 0 degrees Celsius; 2) Range 2: Suitable for reds close to 180 degrees.

[0065] Among them, the first hue threshold and the second hue threshold are configurable parameters that can be adaptively adjusted according to the lighting environment, the material of the marker point, and the imaging characteristics of the camera to achieve stable detection of red marker points.

[0066] The HSV color space is explained in Table 4. The marker detection process is as follows: Figure 10 As shown.

[0067] Table 4 HSV Color Space Description

[0068] 3.4 Linear interpolation to fill missing key points

[0069] Due to occlusion, detection failure, or other reasons, keypoint sequences may contain missing data. See also Figure 11 , Figure 12 This embodiment uses a linear interpolation method to calculate a reasonable replacement value by utilizing the effective data before and after the missing point in the time dimension, thus maintaining the continuity of the data.

[0070] 3.5 Depth Calibration Offset Mechanism

[0071] Due to systematic measurement errors in depth cameras, depth values ​​for different body parts may exhibit fixed deviations. (Refer to...) Figure 13 In this embodiment, the depth measurement value can be calibrated and compensated by setting the offset of each part in the system UI.

[0072] The system provides an interactive offset adjustment interface, allowing users to set depth offset values ​​for the five locations shown in Table 5.

[0073] Table 5 Offset Configuration Instructions

[0074] IV. Noise Filtering Layer

[0075] In the noise filtering layer of this embodiment, the impulse noise in the key point coordinate sequence is first removed by median filtering, and then the key point coordinate sequence is smoothed by Savitzky-Golay filtering to preserve its shape, so as to suppress high-frequency noise while keeping the peak position of gait features from shifting.

[0076] In addition, the noise filtering layer also uses a one-dimensional Kalman filter to filter the key point coordinate sequence. The one-dimensional Kalman filter assumes that the system state follows a linear dynamic model and estimates the system state alternately through two steps: prediction and update.

[0077] 4.1 Phase Overview

[0078] The noise filtering stage performs temporal filtering on the key point coordinate sequence after post-processing to remove jitter caused by detection errors, depth noise, etc., while maintaining the integrity of gait features as much as possible.

[0079] The following functions need to be implemented in this stage: 1) Two-stage filtering schemes, namely median filtering and Savitzky-Golay filtering; 2) The alternative scheme is one-dimensional Kalman filtering.

[0080] 4.2 Two-stage filtering algorithm ( Figure 14 )

[0081] Level 1 - Median Filtering: Median filtering is a non-linear filtering method that replaces each data point with the median of its neighboring points. This method is very effective at suppressing impulse noise (such as salt-and-pepper noise).

[0082] Second-stage Savitzky-Golay filtering: Savitzky-Golay filtering is a filtering method based on local polynomial fitting. This method can remove high-frequency noise while preserving the peak characteristics of the signal (such as the heel strike time in gait).

[0083] 4.3 Kalman Filtering

[0084] Kalman filtering is a filtering method based on recursive optimal estimation. It assumes that the system state follows a linear dynamic model and estimates the system state alternately through two steps: prediction and update.

[0085] State transition equation:

[0086] Observation equation:

[0087] The Kalman filter algorithm flow is as follows: Figure 15 As shown, the timing diagram for the noise filtering stage is as follows: Figure 16 As shown.

[0088] V. Gait Analysis Layer

[0089] In this embodiment, the gait analysis layer includes an automatic gait cycle segmentation module and a joint angle calculation module connected in sequence. The automatic gait cycle segmentation module is used to extract the ankle joint Y-axis coordinate sequence based on the three-dimensional trajectory of the ankle joint. After median filtering, Savitzky-Golay filtering and MAD normalization, the valley detection algorithm is used to identify heel strike events and the interval between adjacent heel strike times is divided into a gait cycle. The joint angle calculation module is used to calculate the angle between bone vectors based on the coordinates of key points, obtain the hip joint angle, knee joint angle and trunk tilt angle, and output the three-dimensional spatial angle as well as the projection angles of the XY plane, YZ plane and XZ plane.

[0090] 5.1 Phase Overview

[0091] The gait analysis phase is the core output module of the system, responsible for extracting valuable medical parameters from the filtered key point data, including gait cycle division and joint angle calculation.

[0092] The following functions need to be implemented in this stage: 1) Automatic division of gait cycle based on ankle joint Y-axis trajectory; 2) Multi-plane projection joint angle calculation; 3) 3D point cloud computing.

[0093] 5.2 Automatic Gait Period Division

[0094] When a person walks, the ankle joint exhibits a periodic fluctuation pattern in its Y-axis coordinate within the camera coordinate system. When the heel strikes the ground, the ankle joint is at its lowest position (trough); when the foot lifts and swings, the ankle joint is at its highest position (peak). By detecting these trough moments, the gait cycle can be automatically defined, as detailed in the following process: Figure 17 As shown in Table 6, the key parameter configurations are as follows.

[0095] Table 6 Key Parameter Configuration

[0096] 5.3 Joint Angle Calculation

[0097] Joint angles are obtained by calculating the angle between two bone vectors. This system calculates the following three types of joint angles: 1) Hip joint angle: The angle between the spine vector and the thigh vector; 2) Knee joint angle: The angle between the thigh vector and the lower leg vector; 3) Trunk tilt angle: The angle between the spine vector and the vertical direction.

[0098] In addition to the 3D spatial angle, the system also calculates the projection angles of three orthogonal planes (XY, YZ, XZ), providing richer dimensions for motion analysis.

[0099] The joint angle calculation process is as follows: Figure 18 As shown, the time sequence diagram for the gait analysis phase is as follows: Figure 19 As shown.

[0100] VI. Data Output Layer

[0101] 6.1 Phase Overview

[0102] The data output phase outputs the analysis results in multiple formats, including structured JSON motion data, visualized videos, and patient database records.

[0103] The following outputs can be achieved in this stage: 1) complete motion data in JSON format; 2) multi-view skeleton visualization video; 3) patient records in SQLite database.

[0104] 6.2 Visual Video Generation

[0105] The multi-view video description is shown in Table 7, and the timing diagram of the data output stage is shown in Table 7. Figure 20 As shown.

[0106] Table 7 Multi-view video description

[0107] Furthermore, this embodiment also discloses a portable gait analysis method based on the fusion of deep learning and multimodal data, applied to the aforementioned portable gait analysis system based on the fusion of deep learning and multimodal data, including the following steps: Multimodal image data is obtained by simultaneously acquiring RGB images, depth images, and infrared images using a depth camera and performing preprocessing. Human pose detection is performed based on the preprocessed RGB image using a visual deep learning model, key point pixel coordinates are extracted, and the three-dimensional coordinates of key points are recovered by combining the preprocessed depth image. The three-dimensional coordinates of key points are calibrated by combining the markers worn by the patient, resulting in a calibrated key point coordinate sequence; A two-stage cascaded method of median filtering and Savitzky-Golay filtering was used to denoise the key point coordinate sequence, resulting in a smoothed key point coordinate sequence. Gait cycles are automatically divided and joint angles are calculated based on the smoothed key point coordinate sequence to obtain gait analysis results; Based on the gait analysis results, output JSON format motion data, generate multi-view skeleton visualization videos, and store patient records in an SQLite database.

[0108] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. The methods disclosed in the embodiments are described simply because they correspond to the systems disclosed in the embodiments; relevant details can be found in the method section.

[0109] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A portable gait analysis system based on the fusion of deep learning and multimodal data, characterized in that, It includes a data acquisition layer, an AI posture detection layer, a post-processing layer, a noise filtering layer, a gait analysis layer, and a data output layer, which are connected in sequence. The data acquisition layer is used to simultaneously acquire RGB images, depth images, and infrared images through a depth camera, and output multimodal image data to the AI ​​pose detection layer after preprocessing. The AI ​​pose detection layer is used to detect human pose based on the pre-processed RGB image using a visual deep learning model, extract the pixel coordinates of key points, and then restore the three-dimensional coordinates of key points by combining them with the pre-processed depth image before outputting them to the post-processing layer. The post-processing layer is used to calibrate the three-dimensional coordinates of key points by combining the markers worn by the patient, and outputs the calibrated key point coordinate sequence to the noise filtering layer. The noise filtering layer is used to reduce the noise of the key point coordinate sequence by using a two-stage cascaded method of median filtering and Savitzky-Golay filtering, and outputs the smoothed key point coordinate sequence to the gait analysis layer. The gait analysis layer is used to automatically divide the gait cycle and calculate joint angles based on the smoothed key point coordinate sequence, and output the gait analysis results to the data output layer. The data output layer is used to output JSON-formatted motion data based on gait analysis results, generate multi-view skeleton visualization videos, and store patient records in an SQLite database.

2. The portable gait analysis system based on the fusion of deep learning and multimodal data as described in claim 1, characterized in that, The preprocessing operations of the data acquisition layer include: aligning the depth image to the RGB image viewpoint, and then performing transpose processing, flip processing, and white balance calibration processing on the frame-aligned RGB image, depth image, and infrared image to obtain preprocessed multimodal image data.

3. The portable gait analysis system based on the fusion of deep learning and multimodal data according to claim 1, characterized in that, The AI ​​pose detection layer includes a human key point detection module, a human tracking module, and a 3D coordinate recovery module connected in sequence. The human key point detection module is used to identify the human position and pose from the preprocessed RGB image, and output the human bounding box and the pixel coordinates and confidence scores of multiple key points; The human body tracking module is used to track detected human bodies across frames using the Byte Track algorithm, assigning a unique identifier to the same human body and selecting the human body bounding box with the largest area as the main analysis target. The 3D coordinate recovery module is used to convert the pixel coordinates of key points into three-dimensional spatial coordinates based on the intrinsic parameters of the depth camera and the depth values ​​corresponding to the pixels, using a pinhole camera model.

4. The portable gait analysis system based on the fusion of deep learning and multimodal data according to claim 1, characterized in that, The post-processing layer includes a marker detection module, a missing value imputation module, and a calibration offset module connected in sequence. The marker detection module is used to perform dual-range red detection in the HSV color space to locate the pixel position of the red spherical marker. The missing value imputation module is used to perform linear interpolation calculations on the effective key point data before and after the missing point in the time dimension to obtain reasonable replacement values ​​for imputation in order to maintain the continuity of the key point coordinate sequence. The calibration offset module is used to set depth offsets for key points corresponding to the shoulder, navel, hip, knee, and ankle in the interactive offset adjustment interface. The original depth value is added to the depth offset to obtain the calibrated depth value, so as to correct the measurement error of the depth camera.

5. The portable gait analysis system based on the fusion of deep learning and multimodal data according to claim 1, characterized in that, In the noise filtering layer, the impulse noise in the key point coordinate sequence is first removed by median filtering, and then the key point coordinate sequence is smoothed by Savitzky-Golay filtering to preserve its shape, thus suppressing high-frequency noise while keeping the peak position of gait features from shifting.

6. The portable gait analysis system based on the fusion of deep learning and multimodal data according to claim 1, characterized in that, The noise filtering layer also uses a one-dimensional Kalman filter to filter the key point coordinate sequence. The one-dimensional Kalman filter assumes that the system state follows a linear dynamic model and estimates the system state alternately through two steps: prediction and update.

7. The portable gait analysis system based on the fusion of deep learning and multimodal data according to claim 1, characterized in that, The gait analysis layer includes a gait cycle automatic segmentation module and a joint angle calculation module connected in sequence; The automatic gait cycle segmentation module is used to extract the ankle joint Y-axis coordinate sequence based on the three-dimensional trajectory of the ankle joint. After median filtering, Savitzky-Golay filtering and MAD normalization, the valley detection algorithm is used to identify heel strike events and the interval between adjacent heel strike times is divided into a gait cycle. The joint angle calculation module is used to calculate the angle between bone vectors based on the coordinates of key points, obtain the hip joint angle, knee joint angle and trunk tilt angle, and output the three-dimensional spatial angle as well as the projection angles of the XY plane, YZ plane and XZ plane.

8. A portable gait analysis method based on the fusion of deep learning and multimodal data, characterized in that, The method applied to the portable gait analysis system based on the fusion of deep learning and multimodal data as described in any one of claims 1-7 includes the following steps: Multimodal image data is obtained by simultaneously acquiring RGB images, depth images, and infrared images using a depth camera and performing preprocessing. Human pose detection is performed based on the preprocessed RGB image using a visual deep learning model, key point pixel coordinates are extracted, and the three-dimensional coordinates of key points are recovered by combining the preprocessed depth image. The three-dimensional coordinates of key points are calibrated by combining the markers worn by the patient, resulting in a calibrated key point coordinate sequence; A two-stage cascaded method of median filtering and Savitzky-Golay filtering was used to denoise the key point coordinate sequence, resulting in a smoothed key point coordinate sequence. Gait cycles are automatically divided and joint angles are calculated based on the smoothed key point coordinate sequence to obtain gait analysis results; Based on the gait analysis results, output JSON format motion data, generate multi-view skeleton visualization videos, and store patient records in an SQLite database.

Images