Multi-modal sensing data automatic labeling method for gait evaluation

Abstract

The invention discloses a multi-modal sensing data automatic labeling method and system for gait assessment, and the system comprises a sensing collection module which comprises two paths of pressure sensors and a six-axis IMU sensor; the two paths of pressure sensors are used for acquiring plantar pressure data; the six-axis IMU sensor is used for acquiring lower limb movement data; the wireless transmission module is used for uploading the collected plantar pressure data and lower limb movement data to a cloud Internet of Things platform; the cloud data processing module is used for receiving and storing the plantar pressure data and the lower limb movement data; the data labeling module is used for automatically recognizing a gait cycle and carrying out multi-dimensional score labeling; and the user interaction module is used for interacting with the data annotation module and the cloud data processing module and displaying gait evaluation related data and results. According to the invention, a multi-mode sensing PCB is self-developed, a closed-loop system of collection, cloud loading, labeling and display is constructed, and accurate, continuous and remote evaluation of gaits of children with cerebral palsy is realized.

Description

Technical Field

[0001] This application relates to the field of medical and health monitoring and intelligent rehabilitation technology, and more specifically, to an automated annotation method for multimodal sensor data of gait assessment. Background Technology

[0002] Cerebral palsy (CP) is a motor dysfunction caused by abnormal brain development, often manifested as gait abnormalities, muscle tone disorders, and poor balance. Traditional gait assessment relies on clinical observation, video analysis, or expensive laboratory-grade equipment (such as force tables and 3D motion capture systems), which has limitations such as high cost, complex operation, and inability to achieve long-term home monitoring.

[0003] In recent years, with the development of wearable sensor technology, pressure sensors based on pressure sensor (polyvinylidene fluoride) films and MEMS inertial measurement units (IMUs) have been widely used in gait analysis. However, existing technologies still have the following problems: Data acquisition is fragmented: most systems only collect single-modal data (such as pressure only or IMU only), making it difficult to comprehensively reflect gait characteristics. For example, existing technology discloses a plantar pressure testing device and a method for using an all-fabric insole with the device. This method uses an electrospun pressure sensor piezoelectric fiber mesh to form an insole pressure sensing array, solving comfort / encapsulation issues, and is positioned for plantar pressure testing and display. The paper focuses on sensing and amplification / filtering, and upper-level acquisition and display; no cloud-based annotation training process or traceable scoring system is found, indicating a solution mainly based on a single pressure modality and local upper-level computer / display.

[0004] Low level of automation in data processing: There is a lack of effective data annotation and scoring mechanisms, relying on manual interpretation, which is inefficient and highly subjective. For example, existing technology discloses a gait analysis device and method based on an IMU inertial sensor. This method discloses a gait analysis device and method based on an IMU inertial sensor, using dual-shoe built-in IMUs, Bluetooth synchronization, and GSM uploading to a data server, and calculating stride length, stride period, etc. using complementary filtering. Although it has remote uploading capabilities, the overall process is based on a single IMU mode and parameter calculation, and only describes the signal processing and parameter calculation flow, without providing a quantifiable automated annotation process, an objective scoring system, or a trainable dataset construction / review mechanism.

[0005] Insufficient support for telemedicine: There is a lack of convenient data sharing and long-term trend analysis tools between parents and doctors. For example, existing technology discloses a smart insole for plantar pressure monitoring and gait analysis. This method discloses a smart insole for plantar pressure monitoring and gait analysis, including: an insole body, a pressure sensor, a heating module, a gait monitoring module, a wireless transmission module, and a control module. This invention includes 10 pressure points and an MPU6050, using NB-IoT for wireless transmission. Although it has wireless transmission capabilities, the patent mainly discloses the data acquisition and transmission channel, lacking a shared interface for family-doctor communication, a trend analysis module, and remote assessment and follow-up management processes.

[0006] Lack of standardized scoring systems: While some studies have proposed gait scoring, no quantifiable, traceable, and trainable labeled datasets have been developed. For example, existing technology discloses a fabric insole containing piezoelectric fibers with a core. This method connects multiple piezoelectric fibers with core pressure sensors in parallel on the bottom of the insole, and monitors the gait by amplifying the charge and transmitting the signal to a mobile phone / APP; it focuses on foot force and behavior monitoring. This invention mainly focuses on hardware and signal pathways, lacking explanations on standardized gait scoring metrics, label metadata, annotation processes, or how to construct a trainable dataset from the collected data.

[0007] Energy-saving strategy (event-triggered adaptive sampling): Existing integrated patent technologies typically describe the sensing / communication and algorithm flow, but rarely formulate the "foot contact event → high sampling wake-up → event end downsampling" process into a system-level energy-saving claim; a significant number of solutions default to normally open multi-point arrays. Our device, however, features a low-power front-end for long-term standby of the pressure sensor, event-triggered high sampling and a complete fusion unit; during the oscillation period, it returns to low power consumption, retaining only low-frequency IMU monitoring, significantly extending battery life, making it especially suitable for everyday wearables.

[0008] Therefore, there is an urgent need for an intelligent gait assessment system that integrates hardware acquisition, data annotation, cloud platform transfer, and multi-terminal visualization to achieve accurate, continuous, and remote assessment of the gait of children with cerebral palsy. Summary of the Invention

[0009] To address the aforementioned issues, this application provides an automated annotation method for multimodal sensor data in gait assessment. The method aims to achieve simultaneous acquisition of plantar pressure and lower leg movement data by integrating a self-developed PCB multimodal sensor; utilize the Alibaba Cloud IoT platform for data upload and storage; employ a data annotation program developed using QT to remotely connect to the cloud database and automate scoring and annotation; and utilize a mini-program to enable data visualization and trend analysis for both doctors and parents, ultimately constructing a closed-loop intelligent rehabilitation assessment system.

[0010] The first aspect of the present invention provides an automated annotation system for multimodal sensor data of gait assessment, comprising: a sensor acquisition module, a wireless transmission module, a cloud data processing module, and a data annotation module connected in sequence, and a user interaction module connected in communication with the cloud data processing module; The sensing and acquisition module includes two pressure sensors and a six-axis IMU sensor; the two pressure sensors are respectively set in the forefoot area and the heel area to collect plantar pressure data; the six-axis IMU sensor is fixed to the lower leg to collect lower limb motion data. The wireless transmission module is used to upload the collected plantar pressure data and lower limb movement data to the cloud-based Internet of Things platform; The cloud-based data processing module is used to receive and store plantar pressure data and lower limb movement data; The data annotation module includes: An automatic gait cycle segmentation unit is used to divide a single gait cycle based on plantar pressure data sequences; The COP scoring unit is used to calculate the absolute value of the difference between the number of positive and negative COP values ​​in a gait cycle based on the COP trajectory of the pressure center, and to map the absolute value of the difference into a COP score; The peak score unit is used to calculate the ratio of the peak forefoot pressure to the peak heel pressure, and to map the interval in which the ratio falls into the peak score. The support phase scoring unit is used to detect the heel landing time and toe take-off time based on lower limb motion data to determine the duration of the support phase and its proportion in the entire gait cycle, and to score based on the interval mapping at the proportion. The user interaction module is used to interact with the data annotation module and the cloud data processing module to display gait assessment-related data and results.

[0011] In one optional implementation, the sensing acquisition module includes a PCB board, on which two pressure sensor signal conditioning circuits and an MPU6050 IMU sensor are integrated, and multi-channel data is synchronously acquired through a main control chip.

[0012] In one optional implementation, the user interaction module includes a doctor's side and a parent's side, which filter data by time and patient ID, and display the rating trend in chart form.

[0013] A second aspect of this invention provides an automated annotation method for multimodal sensor data in gait assessment, comprising: The system synchronously collects the user's plantar pressure data and lower limb movement data using a multimodal sensor; wherein the multimodal sensor includes two pressure sensors located on the forefoot and heel, and a six-axis IMU sensor fixed to the calf. The collected plantar pressure data and lower limb movement data are uploaded to the cloud-based Internet of Things platform via a wireless communication module; The received plantar pressure data and lower limb movement data are analyzed and stored in the cloud database through the cloud IoT platform. Data is read from the cloud database, gait cycles are automatically identified, and each gait cycle is automatically labeled based on preset scoring rules to generate a scoring result including at least one of COP score, peak score and support phase score; The scoring results are stored in a cloud database and provided to doctors or parents through a visual interface, enabling data query and trend analysis.

[0014] In one alternative implementation, the method for automatically identifying gait cycles includes: Based on the heel pressure signal, the interval where the pressure peak is greater than a set threshold and the time interval between adjacent peaks is greater than a set duration is identified to determine a complete gait cycle.

[0015] In one optional implementation, the COP score is calculated as follows:

[0016]

[0017] in, The change in the center of pressure before and after, The left-right variation of the pressure center, x_front_pre is the forefoot pressure, x_back_pre is the rearfoot pressure, x_front is the forefoot coordinate, x_back is the rearfoot coordinate, and total_pre is the sum of the forefoot and rearfoot pressures.

[0018] In one optional implementation, the peak score calculation includes: Analyze the ratio between the peak pressure of the forefoot and the peak pressure of the heel. If the peak pressure of the forefoot exceeds the set ratio of the peak pressure of the heel, it is judged as an abnormal support mode, and a peak score is output accordingly.

[0019] In one optional implementation, the support phase score calculation includes: The support phase time period is extracted based on lower limb motion data. The acceleration variance and angular velocity integral within this time period are calculated. Combined with the proportion of the support phase duration to the entire gait cycle, a comprehensive support phase score is output.

[0020] A third aspect of this invention provides a wearable device for gait assessment, comprising: The flexible insole has at least two pressure sensors embedded inside, one in the forefoot area and the other in the heel area. The lower leg fixation device has a PCB board on it, on which an IMU sensor, a main control chip, a wireless communication module and a power management unit are integrated. The pressure sensor is connected to the PCB board via a signal line. The main control chip is used to synchronously collect plantar pressure data and lower limb movement data, and upload the plantar pressure data and lower limb movement data to the cloud IoT platform through a wireless communication module.

[0021] In this embodiment, the present invention utilizes a self-developed PCB-integrated multimodal sensor to achieve synchronous acquisition of plantar pressure and calf movement data. Data uploading and storage are completed using the Alibaba Cloud IoT platform. A data annotation program developed using QT enables remote connection to the cloud database and automated scoring and annotation. Furthermore, a mini-program is used to achieve data visualization and trend analysis for both doctors and parents, ultimately constructing a closed-loop intelligent rehabilitation assessment system. The present invention has the following beneficial effects: Multimodal fusion improves assessment accuracy: Simultaneous collection of plantar pressure and calf movement data overcomes the limitations of a single sensor and improves the gait anomaly recognition rate; Automated annotation improves efficiency: Scores are automatically generated using QT programs, forming a standardized labeled dataset that can be used for subsequent machine learning model training; Customized hardware ensures data quality: self-developed PCB optimizes signal acquisition paths, reduces noise interference, and ensures long-term monitoring reliability; Cloud-based collaboration enables remote management: Relying on the Alibaba Cloud IoT platform, data security is achieved in the cloud and cross-regional access is enabled, supporting remote diagnosis by doctors; The dual-platform mini-program promotes doctor-patient collaboration: doctors can have a comprehensive grasp of patient group data, and parents can intuitively understand the rehabilitation progress, thereby enhancing treatment compliance; Quantitative scoring supports personalized intervention: rehabilitation plans are dynamically adjusted based on scoring trends, promoting the implementation of precision medicine. Attached Figure Description

[0022] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the description of the embodiments of this application will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0023] Figure 1This is a framework diagram of an automated annotation system for multimodal sensor data for gait assessment proposed in one embodiment of this application; Figure 2 This is a schematic diagram of a PCB board sensor layout according to an embodiment of this application; Figure 3 This is a schematic diagram of the hardware acquisition module structure of a gait assessment system proposed in an embodiment of this application; Figure 4 This is a schematic diagram of the main control chip and USB charging interface circuit proposed in one embodiment of this application; Figure 5 This is a schematic diagram of a 3V3 level conversion circuit and a charging circuit proposed in an embodiment of this application; Figure 6 This is a schematic diagram of a SIM card interface circuit according to an embodiment of this application; Figure 7 This is a schematic diagram of a sampling circuit for a two-channel ADC according to an embodiment of this application; Figure 8 This is a circuit schematic diagram of an IMU sensor proposed in one embodiment of this application; Figure 9 This is a diagram of an intelligent orthotic shoe device according to an embodiment of this application; Figure 10 This is a timing diagram of plantar pressure, gait cycle detection, and acceleration angular velocity under normal gait, as proposed in an embodiment of this application. Figure 11 This is an analysis diagram of the coordination of lower leg and ankle movements under normal gait, as proposed in an embodiment of this application. Figure 12 This is a diagram illustrating the detection of plantar pressure and gait cycle, as well as the detection of acceleration angular velocity and support phase under severe gait abnormalities, as proposed in an embodiment of this application. Figure 13 Analysis of lower leg and ankle movement coordination in patients with severe gait abnormalities. Detailed Implementation

[0024] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0025] Please refer to Figure 1 , Figure 1 This is a framework diagram of an automated annotation system for multimodal sensor data for gait assessment, proposed in one embodiment of this application. Figure 1As shown, an automated annotation system for multimodal sensor data in gait assessment includes: a sensor acquisition module, a wireless transmission module, a cloud data processing module, and a data annotation module that are connected in sequence; and a user interaction module that is connected in communication with the cloud data processing module. The sensing and acquisition module includes two pressure sensors and a six-axis IMU sensor; the two pressure sensors are respectively set in the forefoot area and the heel area to collect plantar pressure data; the six-axis IMU sensor is fixed to the lower leg to collect lower limb motion data. The wireless transmission module is used to upload the collected plantar pressure data and lower limb movement data to the cloud-based Internet of Things platform; The cloud-based data processing module is used to receive and store plantar pressure data and lower limb movement data; The data annotation module includes: An automatic gait cycle segmentation unit is used to divide a single gait cycle based on plantar pressure data sequences; The COP scoring unit is used to calculate the absolute value of the difference between the number of positive and negative COP values ​​in a gait cycle based on the COP trajectory of the pressure center, and to map the absolute value of the difference into a COP score; The peak score unit is used to calculate the ratio of the peak forefoot pressure to the peak heel pressure, and to map the interval in which the ratio falls into the peak score. The support phase scoring unit is used to detect the heel landing time and toe take-off time based on lower limb motion data to determine the duration of the support phase and its proportion in the entire gait cycle, and to score based on the interval mapping at the proportion. The user interaction module is used to interact with the data annotation module and the cloud data processing module to display gait assessment-related data and results.

[0026] In this embodiment, Figure 1The following are demonstrated: (1) Data acquisition layer: The main control chip acquires the voltage signals of two pressure sensors (reflecting the plantar pressure distribution) and the acceleration and gyroscope data of MPU6050 (reflecting the motion posture) through multi-threaded parallel acquisition, and converts the data into string format and uploads it to the Alibaba Cloud platform. (2) Data storage layer: After receiving the data, the Alibaba Cloud platform automatically synchronizes it to the RDS cloud database to realize the persistent storage of the original data. (3) Data processing and annotation layer: The Qt annotation program calls the original data from the RDS database, calculates the COP (center of pressure) score, peak pressure score, and support phase duration score through algorithms, and determines whether the user has a forward lean state and gait abnormality level based on the above scores. (4) Data feedback layer: The annotation program saves the mean of each indicator back to the RDS database. The mini program calls the mean data through the interface. The doctor can view the data analysis results of all users, while the parent can only view the historical mean changes of the bound child and display the progress trend in a visual chart (such as a line graph).

[0027] Furthermore, the sensing acquisition module includes a PCB board, on which two pressure sensor signal conditioning circuits and an MPU6050 IMU sensor are integrated, and multi-channel data is synchronously acquired through the main control chip.

[0028] In this embodiment, the pressure sensor has a dual-channel independent signal conditioning circuit, which uses a resistor divider to reduce the voltage to a suitable voltage detection range for the main control chip. 6-axis IMU (MPU6050): I 2 C interface, 100Hz sampling rate, ±4g acceleration, ±500° / s gyroscope; Main control chip: Quectel EC800K-CNLC is responsible for multi-channel synchronous data acquisition and preprocessing; Power management: Powered by a 3.7V lithium battery, providing a battery life of ≥20 hours; Communication interface: It uses LTE to connect to the base station and connects to a third-party IoT platform; Packaging: Two pressure sensor films are embedded in the forefoot and heel, and the IMU module is fixed to the calf, ensuring wearing comfort and signal stability.

[0029] Please refer to Figure 2 , Figure 2 This is a schematic diagram of a PCB board sensor layout according to an embodiment of this application. Figure 2As shown, the overall layout of the PCB board and sensors at the lower leg is illustrated: the PCB board is fixed to the lower leg and integrates an IMU (Inertial Measurement Unit); two pressure sensor signal lines are led out from the PCB board and connected to the pressure sensors on the forefoot and heel respectively, for collecting foot pressure signals. It shows (1) PCB board: rectangular structure, fixed to the side of the lower leg part of the orthosis, as the hardware carrier for sensors and data processing. (2) IMU module: integrated on the surface of the PCB board, used to collect acceleration and angular velocity data of lower leg movement. (3) Pressure sensor: two pressure sensor thin film sensors are respectively pasted on the forefoot (metapoda head area) and heel (calcaneus area), and connected to the PCB board through two signal lines, for collecting pressure changes in different areas of the foot (converted into voltage signals). (4) Connection relationship: the signal lines of the pressure sensor are led out from the PCB board, and the length is adapted to the distance from the lower leg to the foot, ensuring that the sensor is stably attached to the foot when the user walks. This layout, through a distributed sensor design, enables the synchronous acquisition of limb movement postures and plantar pressure, and is compact and suitable for the daily rehabilitation scenarios of children with cerebral palsy.

[0030] Furthermore, the user interaction module includes a doctor's side and a parent's side, which can filter data by time and patient ID, and display the rating trend in chart form.

[0031] In this embodiment, the doctor's end supports viewing all bound patient data, filtering by name, ID, and time period, and viewing score trend charts and abnormal alarms; the parent's end only supports viewing their own child's data, displaying progress curves, and generating reports; the chart type is bar chart.

[0032] A second aspect of this invention provides an automated annotation method for multimodal sensor data in gait assessment, comprising: The system synchronously collects the user's plantar pressure data and lower limb movement data using a multimodal sensor; wherein the multimodal sensor includes two pressure sensors located on the forefoot and heel, and a six-axis IMU sensor fixed to the calf. The collected plantar pressure data and lower limb movement data are uploaded to the cloud-based Internet of Things platform via a wireless communication module; In this embodiment, the pressure data, acceleration data, and gyroscope data collected by the two piezoelectric films and the MPU6050 are directly uploaded to a third-party IoT platform via a base station; Alibaba Cloud IoT platform configuration: Device authentication (one device, one password); Topic routing: / device / data / upload → Rule Engine → RDS Cloud Database; Data format: string: device_id, timestamp, front_pressure, heel_pressure, acc_x / y / z, gyro_x / y / z, etc.

[0033] The received plantar pressure data and lower limb movement data are analyzed and stored in the cloud database through the cloud IoT platform. In this embodiment, a third-party IoT platform is integrated to achieve device authentication, data parsing, and automatic forwarding of data to the RDS cloud database by the rule engine. Database design: Raw table: stores raw sensor data; Labeling table: stores the scoring results after each labeling (including COP, peak value, support phase, comprehensive score, and anomaly level); Summary table: daily / weekly averages for use by the mini-program.

[0034] Data is read from the cloud database, gait cycles are automatically identified, and each gait cycle is automatically labeled based on preset scoring rules to generate a scoring result including at least one of COP score, peak score and support phase score; In this embodiment, a desktop application developed based on QT connects to an RDS cloud database to read raw data, segment gait cycles based on two selected heel pressure peaks greater than 500 and an interval greater than 15, and extract and automatically score and label these features. The QT annotation program features: support for multi-device data import; visual waveform display (pressure + IMU); automatic gait cycle detection (based on plantar pressure); and batch annotation and manual correction interfaces.

[0035] The scoring results are stored in a cloud database and provided to doctors or parents through a visual interface, enabling data query and trend analysis.

[0036] In this embodiment, the visualization interface can be a WeChat mini program, divided into a doctor's side and a parent's side, supporting data query, chart display, trend analysis and historical comparison.

[0037] Furthermore, methods for automatically identifying gait cycles include: Based on the heel pressure signal, the interval where the pressure peak is greater than a set threshold and the time interval between adjacent peaks is greater than a set duration is identified to determine a complete gait cycle.

[0038] Furthermore, the COP score is calculated as follows:

[0039]

[0040] in, The change in the center of pressure before and after, The left-right variation of the pressure center, x_front_pre is the forefoot pressure, x_back_pre is the rearfoot pressure, x_front is the forefoot coordinate, x_back is the rearfoot coordinate, and total_pre is the sum of the forefoot and rearfoot pressures.

[0041] In this embodiment, the annotation logic is as follows: the number of positive COP values ​​(forward tilt direction) and negative COP values ​​(backward tilt direction) within the statistical period is counted, and the absolute value of the difference is calculated; the smaller the absolute value of the difference, the higher the score: 0-7 scores 5 points; 8-14 scores 4 points; 15-20 scores 3 points; and the absolute value of the difference > 20 scores 2 points.

[0042] Output: Generate a COP score for each gait cycle, and count the number of positive COP values ​​(forward lean direction) and negative COP values ​​(backward lean direction) within the cycle. If the number of positive COP values ​​is greater than the number of negative COP values ​​within the cycle, it is marked as "forward lean"; if the number of positive COP values ​​is less than the number of negative COP values ​​within the cycle, it is marked as "backward lean".

[0043] Furthermore, the peak score calculation includes: Analyze the ratio between the peak pressure of the forefoot and the peak pressure of the heel. If the peak pressure of the forefoot exceeds the set ratio of the peak pressure of the heel, it is judged as an abnormal support mode, and a peak score is output accordingly.

[0044] In this embodiment, the annotation logic is as follows: During a normal gait, the peak heel strike pressure should be higher than the peak forefoot push-off pressure. If the peak forefoot strike pressure is abnormally high (e.g., >120% of the heel strike pressure), an abnormal support pattern is indicated. The ratio of the peak forefoot and heel strike pressure is mapped to a score of 0-5.

[0045] Output: A peak score is generated for each gait cycle. 2 peaks → 5 points; 3 peaks → 4 points; 4 peaks → 3 points; ≥5 peaks → 2 points; otherwise 0 points (normal gait usually has 2 key peaks in the forefoot).

[0046] Furthermore, the support phase score calculation includes: The support phase time period is extracted based on lower limb motion data. The acceleration variance and angular velocity integral within this time period are calculated. Combined with the proportion of the support phase duration to the entire gait cycle, a comprehensive support phase score is output.

[0047] In this embodiment, the support phase score is calculated by combining acceleration and angular velocity data collected by the MPU6050 sensor on the circuit board to analyze the degree of lower leg swaying during the support phase. The annotation logic is as follows: the support phase time period is extracted through the pressure signal, and the acceleration variance, angular velocity integral, and the proportion of the support phase time within the gait cycle are calculated for that time period. A larger integral value of the acceleration data and gyroscope data indicates worse stability. The specific formula is as follows:

[0048]

[0049] in, The raw ADC values ​​(e.g., Ax, Ay, Az) output by a certain axis (x / y / z) of the MPU6050 accelerometer. This is the accelerometer zero-drift calibration value (the average raw value collected under static conditions, used to eliminate zero-point error). This is the accelerometer sensitivity coefficient. It is the acceleration due to gravity. The raw ADC values ​​(e.g., Gx, Gy, Gz) output by a certain axis (x / y / z) of the MPU6050 gyroscope. This is the zero-drift calibration value for the gyroscope (the average raw value collected under static conditions, used to eliminate zero-point error). This is the gyroscope sensitivity coefficient. is the conversion factor from angle (° / s) to radian (rad / s).

[0050] Formula for calculating acceleration variance:

[0051]

[0052]

[0053] in, They are respectively The actual acceleration along the x / y / z axes at time t, where N is the total number of sampling points within a single gait cycle. The variance of acceleration is the largest variable. The larger the variance, the more violent the acceleration fluctuations within the gait cycle, and the worse the body stability.

[0054] Angular velocity integral formula:

[0055]

[0056] in, They are respectively The actual angular velocity along the x / y / z axes at time t, T is the total time of a single gait cycle, Δt is the sampling time interval, and N is the total number of sampling points within a single gait cycle. The integral of angular velocity indicates that the larger the integral of angular velocity, the greater the body rotation amplitude during the gait cycle, and the worse the stability.

[0057] The support phase is the time interval from heel strike (HS) to toe lift-off (TO). Two key moments need to be detected using z-axis acceleration characteristics: the heel strike moment (t). HS ): The moment of the first positive peak of z-axis acceleration during the gait cycle (when the lower leg generates an upward impact as the heel strikes the ground); Time of toe liftoff (t) TO ):t HS Then, at the moment of the first negative valley of z-axis acceleration (when the lower leg lifts off the ground as the toes lift, generating a downward weightlessness signal).

[0058] This is used to obtain the duration of the supporting phase:

[0059] Support ratio:

[0060] in, The total time of the current gait cycle (between two adjacent t) HS The difference, i.e., t HSnext - t HS ).

[0061] Output: Support ratio score. If the support ratio is 55-65%, then 5 points (normal range); 45-55% or 65-75% → 4 points; 40-45% or 75-80% → 3 points; otherwise → 2 points. Using a weighted method, calculate the acceleration variance and angular velocity integral to obtain a final result, and then label it in the database.

[0062] Please refer to Figure 9 , Figure 9 This is a diagram of an intelligent orthotic shoe device according to an embodiment of this application. Figure 9 As shown, a third aspect of the present invention provides a wearable device for gait assessment, comprising: The flexible insole has at least two pressure sensors embedded inside, one in the forefoot area and the other in the heel area. The lower leg fixation device has a PCB board on it, on which an IMU sensor, a main control chip, a wireless communication module and a power management unit are integrated. The pressure sensor is connected to the PCB board via a signal line. The main control chip is used to synchronously collect plantar pressure data and lower limb movement data, and upload the plantar pressure data and lower limb movement data to the cloud IoT platform through a wireless communication module.

[0063] Example: Gait monitoring procedure for a child with cerebral palsy The child wears orthotic shoes integrating pressure sensors and an IMU; pressing the power button on the orthotic shoe turns on the device, and the data is uploaded to the Alibaba Cloud IoT platform via a base station; after being parsed by the cloud platform, it is stored in the RDS cloud database; medical staff use a QT program to view the child's pressure data for one week; the program automatically segments the gait cycle, calculates the COP (center of pressure), peak value, and stance phase score, and generates daily averages; the comprehensive score shows that the child has "moderate gait abnormality," mainly manifested as a tendency to lean forward; the labeled results are stored in the database; parents can view the recent score trend through a mini-program and find that the stance phase score is gradually increasing, indicating an improvement in balance ability; the doctor adjusts the treatment plan accordingly.

[0064] Example 1, please refer to Figure 10 , Figure 10 This is a timing diagram of plantar pressure, gait cycle detection, and acceleration / angular velocity under normal gait, as proposed in an embodiment of this application. Figure 10 As shown, the effect demonstration section presents the evaluation results through specific analysis. For a child who underwent rehabilitation intervention, the analysis results showed that: a total of 43 effective gait cycles were analyzed, with an average cycle duration of 0.77 seconds, an average stance phase ratio of 55.3%, an average total score of 4.75 / 5.00, an average phase regularity of 4.81 / 5.00, and a diagnosis of normal gait. The analysis confidence level reached 80%. Detailed data for the first 10 cycles are shown in Table 1. Table 1

[0065] Example 2, please refer to Figure 12 , Figure 12 This is a diagram illustrating the detection of plantar pressure and gait cycle, as well as the detection of acceleration angular velocity and support phase under severe gait abnormalities, according to an embodiment of this application. Figure 12 As shown, the analysis results of another child who did not receive systematic intervention showed that: a total of 43 effective gait cycles were analyzed, with an average cycle duration of 0.67 seconds, an average stance phase ratio of 22.2%, an average total score of 2.58 / 5.00, and an average phase regularity of 2.77 / 5.00. The diagnosis was severe gait abnormality, with an analysis confidence level of 50%. Detailed data for the first 10 cycles are shown in Table 2: Table 2

[0066] In summary, for normal people, the extracted peak distribution before and after the foot is very regular in terms of time. First, the time of a walking cycle is regular with little deviation. In addition, in the walking cycle of two adjacent heel strikes, there is only one peak in the forefoot. This is normal for a step, which is a process of heel strike, transition to toe strike, and then the whole foot leaves the ground.

[0067] Observing the children's data, the cycles for each individual were irregular and exhibited unique characteristics. For example, one child on Saturday showed a clear pattern of more forefoot strikes and less heel strikes, primarily reflected in the pressure waveform as multiple toe strikes within a single walking cycle. The coefficient of variation (CV) primarily characterizes the repeatability of the data. The data shows that the CV for normal individuals is relatively small, indicating that the force exerted during walking is generally consistent and highly repeatable. However, the children's CVs for both forefoot and heel strikes are larger, suggesting that the body's center of gravity shifts less smoothly during walking, or that they exhibit "dragging" or "jumping" gait, leading to abnormal forefoot strikes.

[0068] Although preferred embodiments of the present application have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of the embodiments of the present application.

[0069] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other.

[0070] Although preferred embodiments of the present application have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of the embodiments of the present application.

[0071] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or terminal device that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or terminal device. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or terminal device that includes said element.

[0072] The above provides a detailed description of the automated annotation method for multimodal sensor data in gait assessment provided by this application. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the method and its core ideas. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. An automated annotation system for multimodal sensor data in gait assessment, characterized in that, It includes a sensor acquisition module, a wireless transmission module, a cloud data processing module, a data labeling module, and a user interaction module that are connected in sequence via communication; The sensing and acquisition module includes two pressure sensors and a six-axis IMU sensor; the two pressure sensors are respectively set in the forefoot area and the heel area to collect plantar pressure data; the six-axis IMU sensor is fixed to the lower leg to collect lower limb motion data. The wireless transmission module is used to upload the collected plantar pressure data and lower limb movement data to the cloud-based Internet of Things platform; The cloud-based data processing module is used to receive and store plantar pressure data and lower limb movement data; The data annotation module includes: An automatic gait cycle segmentation unit is used to divide a single gait cycle based on plantar pressure data sequences; The COP scoring unit is used to calculate the absolute value of the difference between the number of positive and negative COP values ​​in a gait cycle based on the COP trajectory of the pressure center, and to map the absolute value of the difference into a COP score; The peak score unit is used to calculate the ratio of the peak forefoot pressure to the peak heel pressure, and to map the interval in which the ratio falls into the peak score. The support phase scoring unit is used to detect the heel landing time and toe take-off time based on lower limb motion data to determine the duration of the support phase and its proportion in the entire gait cycle, and to score based on the interval mapping at the proportion. The user interaction module is used to interact with the data annotation module and the cloud data processing module to display gait assessment-related data and results.

2. The automated annotation system for multimodal sensor data for gait assessment according to claim 1, characterized in that, The sensing and acquisition module includes a PCB board, which integrates two pressure sensor signal conditioning circuits and an MPU6050 IMU sensor, and performs multi-channel data synchronous acquisition through the main control chip.

3. The automated annotation system for multimodal sensor data for gait assessment according to claim 1, characterized in that, The user interaction module includes a doctor's side and a parent's side, which can filter data by time and patient ID and display the rating trend in chart form.

4. The automated annotation method for multimodal sensor data for gait assessment according to any one of claims 1-3, characterized in that, include: Simultaneously collect user's plantar pressure data and lower limb movement data using multimodal sensors; The multimodal sensor includes two pressure sensors located on the forefoot and heel, and a six-axis IMU sensor fixed to the calf. The collected plantar pressure data and lower limb movement data are uploaded to the cloud-based Internet of Things platform via a wireless communication module; The received plantar pressure data and lower limb movement data are analyzed and stored in the cloud database through the cloud IoT platform. Data is read from the cloud database, gait cycles are automatically identified, and each gait cycle is automatically labeled based on preset scoring rules to generate a scoring result including at least one of COP score, peak score and support phase score; The scoring results are stored in a cloud database and provided to doctors or parents through a visual interface, enabling data query and trend analysis.

5. The automated annotation method for multimodal sensor data in gait assessment according to claim 4, characterized in that, Methods for automatically identifying gait cycles include: Based on the heel pressure signal, the interval where the pressure peak is greater than a set threshold and the time interval between adjacent peaks is greater than a set duration is identified to determine a complete gait cycle.

6. The automated annotation method for multimodal sensor data in gait assessment according to claim 4, characterized in that, The COP score is calculated as follows: in, The change in the center of pressure before and after, The left-right variation of the pressure center, x_front_pre is the forefoot pressure, x_back_pre is the rearfoot pressure, x_front is the forefoot coordinate, x_back is the rearfoot coordinate, and total_pre is the sum of the forefoot and rearfoot pressures.

7. The automated annotation method for multimodal sensor data in gait assessment according to claim 4, characterized in that, The peak score calculation includes: Analyze the ratio between the peak pressure of the forefoot and the peak pressure of the heel. If the peak pressure of the forefoot exceeds the set ratio of the peak pressure of the heel, it is judged as an abnormal support mode, and a peak score is output accordingly.

8. The automated annotation method for multimodal sensor data in gait assessment according to claim 4, characterized in that, The calculation of the support phase score includes: The support phase time period is extracted based on lower limb motion data. The acceleration variance and angular velocity integral within this time period are calculated. Combined with the proportion of the support phase duration to the entire gait cycle, a comprehensive support phase score is output.

9. A wearable device for gait assessment, characterized in that, include: The flexible insole has at least two pressure sensors embedded inside, one in the forefoot area and the other in the heel area. The lower leg fixation device has a PCB board on it, on which an IMU sensor, a main control chip, a wireless communication module and a power management unit are integrated. The pressure sensor is connected to the PCB board via a signal line. The main control chip is used to synchronously collect plantar pressure data and lower limb movement data, and upload the plantar pressure data and lower limb movement data to the cloud IoT platform through a wireless communication module.

Images