Quantitative evaluation method based on multi-modal data

Abstract

The invention discloses a quantitative evaluation method based on multi-modal data, and the method comprises the steps: collecting plantar pressure data through two paths of pressure sensors disposed on a forefoot and a heel, and collecting shank motion data through a 6-axis IMU sensor disposed on a shank part; performing data processing and feature extraction on the plantar pressure data and the shank movement data to obtain gait parameters; and evaluating the gait parameters to generate an evaluation result. According to the method, plantar pressure data and motion sensor data are fused, a cerebral palsy child gait quantitative evaluation method meeting clinical medicine requirements is combined, and doctors are directly assisted in completing gait abnormity diagnosis, rehabilitation scheme making and curative effect tracking.

Description

Technical Field

[0001] This application relates to the interdisciplinary field of medical health and sports assessment, and more specifically, to a quantitative assessment method based on multimodal data. Background Technology

[0002] Current methods for pediatric rehabilitation assessment have significant shortcomings in terms of sensor configuration, continuity, and clinical applicability, mainly in the following three aspects: (1) Sensor configuration: minimalist vs. multi-point array Many existing patents use multi-point pressure arrays (resistive / capacitive) + IMUs to collect gait events / parameters: for example, some patents use 8-point FSRs on the sole of the foot + IMUs in the lower leg to determine HS / TO using rule thresholds; others use multiple IMUs + pressure insoles for feature extraction and anomaly detection. These solutions have many hardware points and are costly and power-consuming.

[0003] PMAF: Only two PVDFs (forefoot / heel) + one lower leg IMU. The short-term energy of the PVDFs is used to construct the COP_proxy for event anchoring, while the IMU handles continuous phase propagation. This significantly reduces the number of hardware points while still recovering the contact trajectory and gait phase. Compared to the mainstream approach of "multi-point array COP calculation," the number of sensors is significantly reduced (the literature generally recommends 3–17 or more sensor points to ensure COP accuracy). However, the data source is singular, lacking multimodal data fusion processing. (2) Phase estimation mechanism: "phase perception" fusion of phase anchoring and propulsion A representative IMU patent uses an adaptive Hopf oscillator to estimate the gait phase from the thigh IMU swing angle. It does not rely on the plantar contact signal and has a certain real-time advantage during sudden stops / accelerations. However, it lacks strong anchoring of real contact events and is susceptible to drift / attitude estimation errors.

[0004] PMAF: It sets "phase" as the core latent variable; PVDF provides hard anchoring during strong observation periods (contact / energy peak), and IMU provides continuous propagation. Likelihood-weighted fusion is performed using PF / EKF + confidence factor γ(t) (PVDF is trusted more during contact periods, and IMU is trusted more during oscillation periods). This statistical fusion of the "phase-sensing observation model," unlike rule / threshold or single-source oscillator methods, balances continuity and anchoring, reducing cumulative drift. (Existing fusion patents mostly focus on "parallel sensor + rule / feature" levels; publicly available implementations using "phase" as a unified state for probabilistic fusion are rare.) (3) COP trajectory reconstruction: replace multi-array with dual PVDF + attitude projection Publicly available solutions generally estimate COP using multi-point pressure distribution or IMU-only regression: such as multi-pressure point insoles / patents and several studies (pressure-sensitive arrays, textile capacitance, simplified layout evaluation); there are also academic approaches that estimate COP using only IMU (which have high requirements for modeling / scenario).

[0005] PMA: Energy weighting formula using dual PVDF

[0006]

[0007] Get forward and backward Then, the contact trajectory is reconstructed by projecting the IMU attitude data onto world coordinates. Compared to "direct integration with multi-point arrays," this method requires fewer data points, is simpler to deploy, and has lower power consumption. Compared to IMU-only methods, it provides physical anchoring during the contact period, resulting in a more stable trajectory. Note: Existing PVDF insole / hardware patents mostly focus on structure and readout circuits, with fewer claims on algorithms that fuse PVDF with lower leg IMU geometry / attitude to reconstruct world coordinate system (COP). This invention, based on collaboration with hospitals and combined with doctors' clinical experience, constructs a closed-loop system of "data acquisition - quantitative evaluation - data display - clinical diagnosis mapping" to solve the above problems.

[0008] (4) Online individualization and adaptation Many patents emphasize factory / one-time calibration or offline template / feature library comparison (such as some patents using "abnormal database" comparison), while online adaptive / personalized template learning is not common.

[0009] PMAF: While estimating the phase, it resamples the PVDF waveform of each cycle to the φ space for online template / dictionary updates (EM / weighted average); template drift can also trigger alarms (sensor slippage / shoe replacement / disease progression), enhancing long-term robustness and interpretability.

[0010] (5) Energy saving strategy: event-triggered sampling adaptive Existing integrated patents typically describe the sensing / communication and algorithm process, but rarely form system-level energy-saving claims for "foot contact event → high sampling wake-up → event end downsampling"; a considerable number of solutions default to normally open multi-point arrays.

[0011] PMAF: With a low-power PVDF front end, it can standby for a long time, and the event-triggered high sampling and complete fusion are integrated. During the swing period, it returns to low power consumption and only retains low-frequency IMU monitoring, which significantly extends the battery life and is especially suitable for everyday wearables.

[0012] (6) Common approach: Feature extraction + comparison with database / experience threshold.

[0013] PMAF: Directly outputs physically interpretable indicators such as phase alignment reconstruction error, phase deviation (IMU propulsion vs PVDF correction), COP path offset / smoothness, and left-right symmetry. It can also be connected to a class of SVM / isolated forests for online "normal domain" learning, making the source of anomalies clearer (event anchoring anomaly / attitude anomaly / contact pattern change). Summary of the Invention

[0014] To address the aforementioned issues, this application provides a quantitative assessment method based on multimodal data, aiming to resolve the problems of existing assessment methods being highly subjective, having crude data processing, lacking historical data display, and being disconnected from clinical diagnosis.

[0015] The first aspect of this invention provides a quantitative evaluation method based on multimodal data, comprising: S100: Collects plantar pressure data through two pressure sensors located on the forefoot and heel, and collects calf movement data through a 6-axis IMU sensor located on the calf. S200. Perform data processing and feature extraction on plantar pressure data and lower leg movement data to obtain gait parameters, specifically including: S210. Preprocess the plantar pressure data and lower leg motion data to obtain short-time energy signals, acceleration variance and angular velocity integral. S220. Based on short-time energy signals and lower leg movement data, gait event points are identified through an adaptive threshold detection method, and gait cycles are divided according to gait event points; S230. Within the gait event point, obtain the phase observation value through template matching, and update the template based on the phase observation value; S240. Based on phase observations and lower leg movement data, the gait phase is estimated by fusing the extended Kalman filter (EKF) algorithm. S250, based on gait phase and short-time energy signals, the pressure center COP proxy value is obtained; S260. Extract gait parameters in each gait cycle. Gait parameters include acceleration variance, angular velocity integral, pressure center variation range, support phase duration, and support phase percentage. S300: Evaluate the gait parameters and generate evaluation results.

[0016] In one optional implementation, the preprocessing in S210 includes high-pass filtering and low-pass filtering of the user's plantar pressure data, and converting the calf movement data into acceleration and angular velocity in physical units; Based on the preprocessed acceleration and angular velocity, the acceleration variance and angular velocity integral are obtained; Based on the preprocessed plantar pressure data, the formula for calculating the short-time energy signal is as follows:

[0017] Where E[n] is the merged short-time energy signal, W is the half-width of the window used to calculate the short-time energy, and m is the summation index. The process iterates within the window... This is the forefoot pressure signal after high-pass and low-pass filtering. This is the hind foot pressure signal after high-pass and low-pass filtering.

[0018] In an optional implementation, step S220 employs an adaptive threshold detection method based on the median and the absolute deviation of the median, with the threshold calculation formula as follows: +

[0019] in, The adaptive threshold is calculated for the energy signal, and E[n] is the merged short-time energy signal. The median of the combined energy signal E[n] is given by k, where k is the threshold coefficient.

[0020] In one optional implementation, the specific steps of the S230 template matching are as follows: At the gait event point Extract a pressure signal segment of length L from the vicinity; The pressure signal segment and the template signal are cross-correlated and normalized to obtain a cross-correlation sequence. Find the lag value that maximizes the cross-correlation coefficient sequence and map the lag value to the phase observation value; Specifically, when the maximum cross-correlation sequence exceeds a preset confidence threshold, the template is also updated.

[0021] Where T is the template vector being maintained and updated, α is the learning rate for template updates, ρ is the confidence threshold for template updates, and S is the learning rate at event point. The current signal segment of length L is captured nearby.

[0022] In one optional implementation, the COP proxy value is calculated using the following formula:

[0023] in, Let n be the pressure center proxy value at time n. This is the installation location for the forefoot pressure sensor. This is the installation location for the hindfoot pressure sensor. It is a very small constant; The range of pressure center variation is obtained by taking the difference between the maximum and minimum COP values ​​within one gait cycle.

[0024] In an optional implementation, the extraction of the support phase duration and support phase ratio in S260 includes: Calculate the mean value of z-axis acceleration within one gait cycle. with standard deviation ; by ±κ The first local maximum value is used as the landing point for threshold detection. The last local minimum is taken as the distance from the location. ; The duration of the support phase is:

[0025] in, Let be the duration of the support phase in the i-th gait cycle; The proportion of support is:

[0026] in, Let be the total duration of the i-th gait cycle. The proportion of support phase in the i-th gait cycle.

[0027] In an optional implementation, S300 further includes determining the coordination of the ankle and knee joints based on plantar pressure data and calf movement data, respectively. Ankle coordination is assessed by calculating the ratio of forefoot pressure to heel pressure during the initial ground contact phase. If the ratio of forefoot pressure to heel pressure is greater than the statistical threshold for healthy individuals, it is considered an abnormality in ankle function. Knee joint coordination is assessed by calculating the root mean square value of the acceleration in the left and right directions of the lower leg during the gait cycle. If the root mean square value is greater than the statistical threshold for healthy individuals, it is judged as an abnormal knee joint stability.

[0028] A second aspect of the present invention provides a quantitative evaluation device based on multimodal data, the device comprising: The data acquisition module is used to collect plantar pressure data and calf movement data through pressure sensors and IMU sensors; The data processing module is used for data preprocessing, event detection, phase estimation, COP calculation, and support phase detection. The scoring and diagnosis module is used to perform periodic scoring and coordination assessment based on gait parameters, and generate evaluation results. The data storage and transmission module is used to upload data to the cloud platform and associate it with medical record information.

[0029] A third aspect of the present invention provides an electronic device, characterized in that it includes: a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement a quantitative evaluation method based on multimodal data.

[0030] A fourth aspect of the present invention provides a computer-readable storage medium, characterized in that a computer program is stored on the computer-readable storage medium, and the computer program, when executed by a processor, is a quantitative evaluation method based on multimodal data.

[0031] In the embodiments disclosed herein, this application has the following beneficial effects: Improving diagnostic consistency: Quantitative indicators enhance the consistency of physician assessments; Shorten assessment time: Single-case assessment greatly reduces the time doctors spend on visual inspection (including data collection and report generation); Enhanced rehabilitation follow-up: It allows for comparison of score changes at different time points (e.g., how many points the abnormal pattern score improves after 3 months of rehabilitation), providing a direct reflection of the treatment effect; High clinical adaptability: The scoring rules are directly related to cerebral palsy classification and intervention plans, without the need for doctors to interpret them a second time. Attached Figure Description

[0032] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the description of the embodiments of this application will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0033] Figure 1 This is a flowchart of a quantitative evaluation method based on multimodal data proposed in an embodiment of this application; Figure 2 This is a comparison diagram of normal gait multimodal signals proposed in an embodiment of this application; Figure 3 This is a timing diagram of normal gait events proposed in an embodiment of this application; Figure 4 This is a comparison diagram of multimodal signals of severe abnormal gait proposed in an embodiment of this application; Figure 5 This is a timing diagram of severe abnormal gait events proposed in an embodiment of this application; Figure 6 This is a structural diagram of a quantitative evaluation device based on multimodal data according to an embodiment of this application; Figure 7 This is a schematic diagram of an electronic device according to this application. Detailed Implementation

[0034] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0035] Please refer to Figure 1 , Figure 1 This is a flowchart illustrating a quantitative evaluation method based on multimodal data proposed in an embodiment of this application. Figure 1 As shown, a quantitative evaluation method based on multimodal data includes: S100: Collects plantar pressure data through two pressure sensors located on the forefoot and heel, and collects calf movement data through a 6-axis IMU sensor located on the calf. In this embodiment, the hardware configuration uses a custom PCB board that integrates a plantar pressure sensor (covering the forefoot and heel areas, with a sampling rate of 16.7Hz) and an accelerometer and gyroscope sensor (placed on the lower leg, with a sampling rate of 16.7Hz). Data collection scenario: Under the guidance of a doctor, data collection is completed in the hospital's rehabilitation assessment room, simultaneously recording the child's basic clinical information (such as cerebral palsy classification and previous rehabilitation history) and the doctor's preliminary visual assessment conclusions (such as "left foot inversion"). Data transmission: Foot pressure data collected by two piezoelectric thin film sensors placed on the forefoot and heel is uploaded to the Alibaba Cloud platform in real time, and then transferred to the cloud database (RDS) through the Alibaba Cloud platform for storage in association with hospital medical records.

[0036] S200. Perform data processing and feature extraction on plantar pressure data and lower leg movement data to obtain gait parameters, specifically including: S210. Preprocess the plantar pressure data and lower leg motion data to obtain short-time energy signals, acceleration variance and angular velocity integral. S220. Based on short-time energy signals and lower leg movement data, gait event points are identified through an adaptive threshold detection method, and gait cycles are divided according to gait event points; S230. Within the gait event point, obtain the phase observation value through template matching, and update the template based on the phase observation value; S240. Based on phase observations and lower leg movement data, the gait phase is estimated by fusing the extended Kalman filter (EKF) algorithm. In this embodiment, the state vector of the EKF includes phase and phase rate.

[0037]

[0038]

[0039]

[0040] Where x[n] is the state vector at time n, ϕ[n] is the estimated gait phase (radians) at time n, ω[n] is the estimated phase change rate (radians / second) sampling interval at time n, Δt=1 / fs, and g(·) is a function that maps the gyroscope amplitude W[n] to the phase rate, for example, a linear mapping g(W)= * W+ . , These are the coefficients that need to be calibrated using data.

[0041] EKF prediction steps:

[0042]

[0043] in, This is a priori estimate of the current state based on the prediction of the previous state. Here is the error covariance matrix for state estimation. Let f be the Jacobian matrix of the state transition function f. For the above linear phase prediction model, The process noise covariance matrix represents the uncertainty of the model.

[0044] EKF update steps (when phase observations are available) hour):

[0045]

[0046]

[0047]

[0048]

[0049]

[0050] in, To measure the residual, i.e., the difference between the observed and predicted values, h(·) is the measurement function, where h(x) = That is, directly observing the phase. To measure the Jacobian matrix of the function h, =

[10] , To measure the noise covariance, it is a scalar representing the phase observation. Uncertainty The residual covariance matrix is... The Kalman gain determines the weights of the predicted and observed values ​​in the final estimate. To integrate the predicted and observed state posterior estimates, It is an identity matrix.

[0051] S250, based on gait phase and short-time energy signals, the pressure center COP proxy value is obtained; S260. Extract gait parameters in each gait cycle. Gait parameters include acceleration variance, angular velocity integral, pressure center variation range, support phase duration, and support phase percentage. S300: Evaluate the gait parameters and generate evaluation results.

[0052] In this embodiment, the indicator calculation and scoring are specifically as follows: For each cycle Calculate the following indicators and map them to a score S∈[1,5] (5 points is optimal): Support ratio score :based on Normal walking typically accounts for about 60%.

[0053] Acceleration variance score Based on resultant acceleration The variance Vi over the period. A small variance indicates stable motion.

[0054] Angular velocity integral score Based on the integral of the absolute value of the resultant angular velocity over the period It reflects the amplitude and energy consumption of the swinging leg movement.

[0055] COP rating : Scope of variation based on COP proxy Or its symmetry. Reflects pressure distribution and the rolling process.

[0056] Phase regularity score : Based on the first-order difference standard deviation of the phase sequence ϕ[n] within the period A small standard deviation indicates a stable gait rhythm.

[0057] The total score for the cycle and the overall diagnosis are as follows:

[0058]

[0059]

[0060] in, The total score for the i-th gait cycle is... Let the weight of the j-th scoring indicator satisfy... (For example: ... = 0.15, w5 = 0.2, w6 = 0.2), Rewards are based on the duration of the period (e.g., a fixed score is given if the period is valid). The average of the total scores across all periods. The average score for the phase regularity of all cycles is given, where N is the total number of cycles.

[0061] Example of general diagnostic rules: gait is normal and regular. ≥4.0 and ≥4.0; mild gait abnormalities are ≤3.5 <4.0; moderate gait abnormality is 3.0≤ <3.5; severe gait abnormality is <3.0.

[0062] The system assists in the output of diagnostic reports, generating structured reports that conform to clinical practice, including: quantitative scores and abnormality levels (with deviations from the mean of normal children of the same age); automatically marked abnormal features (such as "abnormality level 3, forward gait"); and clinical decision recommendations (such as "severe intervention recommended").

[0063] Furthermore, the preprocessing in S210 includes high-pass filtering and low-pass filtering of the user's plantar pressure data, and converting the calf movement data into acceleration and angular velocity in physical units; Based on the preprocessed acceleration and angular velocity, the acceleration variance and angular velocity integral are obtained; Based on the preprocessed plantar pressure data, the formula for calculating the short-time energy signal is as follows:

[0064] Where E[n] is the merged short-time energy signal, W is the half-width of the window used to calculate the short-time energy, and m is the summation index. The process iterates within the window... This is the forefoot pressure signal after high-pass and low-pass filtering. This is the hind foot pressure signal after high-pass and low-pass filtering.

[0065] In this embodiment, Qt is used to connect to the cloud database. Sensor data transmitted from the cloud platform is extracted and refreshed according to time. 200 sets of pressure data, 200 sets of acceleration data, and 200 sets of gyroscope data are extracted at one time for data analysis.

[0066] Convert the lower leg motion data into acceleration and angular velocity in physical units:

[0067]

[0068] in, The raw ADC values ​​(e.g., Ax, Ay, Az) output by a certain axis (x / y / z) of the MPU6050 accelerometer. This is the accelerometer zero-drift calibration value (the average raw value collected under static conditions, used to eliminate zero-point error). This is the accelerometer sensitivity coefficient. It is the acceleration due to gravity. The raw ADC values ​​(e.g., Gx, Gy, Gz) output by a certain axis (x / y / z) of the MPU6050 gyroscope. This is the zero-drift calibration value for the gyroscope (the average raw value collected under static conditions, used to eliminate zero-point error). This is the gyroscope sensitivity coefficient. is the conversion factor from angle (° / s) to radian (rad / s).

[0069] Calculate the acceleration and angular velocity combined models:

[0070]

[0071] in, , , These are the raw acceleration data along the three axes (x, y, z) at sampling point n. , , These are the raw angular velocities of the three axes (x, y, z) at sampling point n.

[0072] Furthermore, step S220 employs an adaptive threshold detection method based on the median and the absolute deviation of the median, with the threshold calculation formula as follows: +

[0073] in, The adaptive threshold is calculated for the energy signal, and E[n] is the merged short-time energy signal. The median of the combined energy signal E[n] is given by k, where k is the threshold coefficient.

[0074] In this embodiment, peak detection and merging are performed to find peak points that meet the height and minimum interval conditions on the energy signal and gyroscope signal, respectively.

[0075]

[0076]

[0077]

[0078] in, P is the set of candidate event points (indices) detected on the energy signal E[n]. Q is the set of candidate event points (indices) detected on the gyroscope signal W[n], and t is the discrete-time index (sample point position) that satisfies the condition. This is the index of the last accepted candidate event point. The minimum sample interval between event points. = * fs] is used to prevent the detection of excessively dense peaks. It is usually set to 0.35–0.6 seconds. The final event sequence is the union of P and Q, after deduplication and sorting. ={e1, e2, ..., eM}.

[0079] Furthermore, the specific steps of the S230 template matching are as follows: At the gait event point Extract a pressure signal segment of length L from the vicinity; The pressure signal segment and the template signal are cross-correlated and normalized to obtain a cross-correlation sequence. In this embodiment, at the event time Extract a segment of the signal from the sum of the pressure signals before and after:

[0080]

[0081]

[0082] Where S is at the event point The current signal segment of length L is extracted from the vicinity, where L is the template length (number of sample points), typically corresponding to a duration of 0.6 seconds or 1.2 seconds, depending on the sampling frequency. , The current signal segment after normalization. The template signal is normalized. , Let S be the mean and standard deviation of the template signal S, respectively. , Let T be the mean and standard deviation of the template signal T, respectively. It is a very small constant (e.g., 1e-8) to prevent division by zero.

[0083] Find the lag value that maximizes the cross-correlation coefficient sequence and map the lag value to the phase observation value; In this embodiment, the normalized cross-correlation function between the current signal segment and the template is calculated to find the maximum correlation value and its corresponding hysteresis value:

[0084]

[0085]

[0086] Where r(k) is the normalized cross-correlation coefficient at lag k. To maximize the lag value of the cross-correlation coefficient r(k), The maximum cross-correlation coefficient is used to measure the degree of matching between the current signal segment and the template. The calculated phase anchor point observations range from [0, 2π) radians. These will be used as the EKF measurement.

[0087] Specifically, when the maximum cross-correlation sequence exceeds a preset confidence threshold, the template is also updated.

[0088] Where T is the template vector being maintained and updated, α is the learning rate for template updates, ρ is the confidence threshold for template updates, and S is the learning rate at event point. The current signal segment of length L is captured nearby.

[0089] In this embodiment, if the matching degree is high enough, the template is updated using the current signal segment in a moving average manner. α can be a fixed small value (e.g., 0.1) or an adaptive value α = 1 / (1 + count); ρ only changes when r... max The template will only be updated when the value is greater than or equal to ρ, in order to prevent abnormal waveforms from contaminating the template.

[0090] Furthermore, the formula for calculating the COP proxy value is as follows:

[0091] in, Let n be the pressure center proxy value at time n. This is the installation location for the forefoot pressure sensor. This is the installation location for the hindfoot pressure sensor. It is a very small constant; The range of pressure center variation is obtained by taking the difference between the maximum and minimum COP values ​​within one gait cycle.

[0092] In this embodiment, within a gait cycle Within this scope, the range of changes for the COP agent can be calculated:

[0093] in, The maximum range of COP surrogate value within the i-th gait cycle can be used as a surrogate index for stride length or gait stability.

[0094] Furthermore, the extraction of the S260 support phase duration and support phase proportion includes: Calculate the mean value of z-axis acceleration within one gait cycle. with standard deviation ; by ±κ The first local maximum value is used as the landing point for threshold detection. The last local minimum is taken as the distance from the location. ; The duration of the support phase is:

[0095] in, Let be the duration of the support phase in the i-th gait cycle; The proportion of support is:

[0096] in, Let be the total duration of the i-th gait cycle. The proportion of support phase in the i-th gait cycle.

[0097] In this embodiment, during the period Calculate the statistics of Z-axis acceleration within the range [n0, n1), and set a threshold for peak / valley detection:

[0098]

[0099] in, Z-axis acceleration during the i-th period The mean, Z-axis acceleration during the i-th period The standard deviation.

[0100] landing point It is usually the first local maximum that satisfies the following conditions:

[0101] Distance from location It is usually the last local minimum that satisfies the following conditions:

[0102] Wherein, κ is an empirical coefficient, typically ranging from 0.4 to 0.6, used to control the leniency of the detection threshold. This is the sample index corresponding to the detected landing time. This is the sample index corresponding to the detected time of departure from the ground.

[0103] Furthermore, S300 also includes determining the coordination of the ankle and knee joints based on plantar pressure data and calf movement data, respectively: Ankle coordination is assessed by calculating the ratio of forefoot pressure to heel pressure during the initial ground contact phase. If the ratio of forefoot pressure to heel pressure is greater than the statistical threshold for healthy individuals, it is considered an abnormality in ankle function. Knee joint coordination is assessed by calculating the root mean square value of the acceleration in the left and right directions of the lower leg during the gait cycle. If the root mean square value is greater than the statistical threshold for healthy individuals, it is judged as an abnormal knee joint stability.

[0104] In this embodiment, the principle for judging the coordination of the ankle and knee joints is as follows: The first step is timing alignment. Each step begins with a forward acceleration, which first increases and then decreases, finally decreasing to zero. When the acceleration decreases to zero, the heel lands. The second step is foot pressure timing detection, where the heel lands first, followed by the forefoot. If the forefoot lands first, or if the heel and forefoot land simultaneously, there is a problem with the ankle joint. The third step is acceleration detection. In a normal gait, there is only forward, backward, up, and down acceleration. If lateral acceleration is detected, there is a problem with the knee joint.

[0105] Gait event detection (heel contact point segmentation): First, each gait cycle needs to be accurately segmented from the signal, and the heel strike event is the key point for segmenting gait cycles.

[0106]

[0107] Where H is the set of indices of all detected heel-to-ground event points, and t is the discrete-time index (sample point position). This represents the heel pressure value at time index t. The trigger threshold for heel pressure is typically set to 110% - 120% of the static pressure value.

[0108] Ankle joint dysfunction detection: Within each gait cycle ( H) Detect whether the triggering sequence of plantar pressure is abnormal.

[0109]

[0110] in, The initial ground contact pressure ratio quantifies the ratio of forefoot pressure to heel pressure in the short period after heel strikes the ground. This is the starting point of the i-th gait cycle (heel strike event). To determine the analysis window length, the number of samples corresponding to 50ms after the heel touches the ground ( =0.05× , (sampling frequency) Let ϵ be the forefoot pressure value at time index k, where ϵ is a very small constant to prevent the denominator from being zero.

[0111]

[0112] in, This is a marker for ankle joint abnormalities during the i-th gait cycle, where 1 indicates abnormality and 0 indicates normality. The threshold is determined by statistical analysis of data from healthy individuals, for example... =0.5, A value >0.5 indicates excessive weight-bearing on the forefoot during initial contact with the ground, consistent with the abnormal "toe-first, heel-second" pattern.

[0113] Knee joint stability abnormality detection: Within the same gait cycle, analyze the acceleration of the lower leg in the left and right directions (inner and outer sides) to determine the stability of the knee joint.

[0114]

[0115] in, Let M be the root mean square (RMS) of the lateral acceleration in the i-th gait cycle, used to measure the lateral sway energy of the knee joint in that cycle, and M be the length of the current gait cycle (number of samples). - This represents the left-right acceleration value of the lower leg at time index n.

[0116]

[0117] in, This is a marker for knee joint abnormalities during the i-th gait cycle, where 1 indicates abnormality and 0 indicates normality. The threshold is determined by statistical analysis of data from healthy individuals, for example, by setting the threshold to "healthy population". The 95th percentile of the distribution The excessive swaying in the left and right directions is consistent with the abnormal pattern of "acceleration in the left and right directions".

[0118] Algorithm Output: For each gait cycle i, the algorithm outputs a diagnostic tuple:

[0119] By analyzing multiple consecutive cycles (e.g., walking for 1 minute) and By performing statistical analysis (such as calculating the proportion of abnormalities), an overall assessment of the subject's function can be obtained.

[0120] Please refer to Figure 6 , Figure 6 This is a structural diagram of a quantitative evaluation method based on multimodal data proposed in an embodiment of this application. Figure 6 As shown in the figure, this disclosure also provides a quantitative evaluation device based on multimodal data, the device including: a data acquisition module 601, a data processing module 602, a scoring and diagnosis module 603, and a data storage and transmission module 604; Data acquisition module 601 is used to acquire plantar pressure data and calf movement data through pressure sensors and IMU sensors; Data processing module 602 is used for data preprocessing, event detection, phase estimation, COP calculation and support phase detection; The scoring and diagnosis module 603 is used to perform periodic scoring and coordination assessment based on gait parameters, and generate evaluation results; The data storage and transmission module 604 is used to upload data to the cloud platform and associate it with medical record information.

[0121] This disclosure also provides an electronic device, please refer to... Figure 6 , Figure 6 This is a schematic diagram of an electronic device illustrated in an embodiment of this disclosure. For example... Figure 6 As shown, the electronic device 100 includes a memory 110 and a processor 120. The memory 110 and the processor 120 are connected via a bus for communication. The memory 110 stores a computer program that can run on the processor 120 to implement the steps in the quantitative evaluation method based on multimodal data disclosed in this embodiment.

[0122] The disclosed embodiments also provide a computer-readable storage medium that, when the instructions in the computer-readable storage medium are executed by a processor of a computer device, enables the computer device to perform steps in the quantitative evaluation method based on multimodal data as described in the embodiments of this disclosure.

[0123] In this example, gait assessment for children aged 3-12 with cerebral palsy is conducted through multi-dimensional data collection, processing, and analysis, combined with doctor-interactive verification, to achieve accurate quantitative assessment of the children's gait status. The specific implementation process is as follows: The sensors are worn by embedding plantar pressure sensors into custom-made orthotic shoes, corresponding to the forefoot and heel areas respectively; sensors integrating accelerometer and gyroscope functions are fixed to the back of the child's lower leg, ensuring a close fit without affecting normal walking. The data collection process is conducted under the full supervision of medical staff. The child walks naturally for 10 minutes daily, and the data collection process continues for a period of time to obtain gait data from multiple time periods, ensuring the comprehensiveness and representativeness of the data.

[0124] In the data preprocessing stage, dedicated software developed with Qt is used to collect sensor data at corresponding time points from the cloud database. The software is set to automatically extract 200 consecutive data points each time and automatically refresh the data sequence according to the time axis to achieve dynamic real-time processing of gait data.

[0125] Please refer to Figure 2 , Figure 2 This is a comparison diagram of normal gait multimodal signals proposed in an embodiment of this application. For example... Figure 2 As shown, the effect demonstration section presents the evaluation results through specific analysis. For a child who underwent rehabilitation intervention, the analysis results showed that: a total of 43 effective gait cycles were analyzed, with an average cycle duration of 0.77 seconds, an average stance phase ratio of 55.3%, an average total score of 4.75 / 5.00, an average phase regularity of 4.81 / 5.00, and a diagnosis of normal gait. The analysis confidence level reached 80%. Detailed data for the first 10 cycles are shown in Table 1. Table 1

[0126] The results show that you should refer to the following: Figure 3 , Figure 3 This is a timing diagram of normal gait events proposed in an embodiment of this application. For example... Figure 3 As shown, the signal timing characteristics of this child are as follows: the timing of acceleration and heel pressure is basically normal, showing a pattern of acceleration followed by heel pressure; the timing of forefoot and hindfoot pressure is basically normal, showing heel pressure followed by forefoot pressure; the acceleration is basically normal, and the lateral acceleration is almost 0.

[0127] Please refer to Figure 4 , Figure 4 This is a comparison diagram of multimodal signals of severe abnormal gait proposed in an embodiment of this application. For example... Figure 4 As shown, the analysis results of another child who did not receive systematic intervention showed that: a total of 43 effective gait cycles were analyzed, with an average cycle duration of 0.67 seconds, an average stance phase ratio of 22.2%, an average total score of 2.58 / 5.00, and an average phase regularity of 2.77 / 5.00. The diagnosis was severe gait abnormality, with an analysis confidence level of 50%. Detailed data for the first 10 cycles are shown in Table 2: Table 2

[0128] Please refer to Figure 5 , Figure 5 This is a timing diagram of severe abnormal gait events proposed in an embodiment of this application. For example... Figure 5 As shown, the results indicate that the child's signal timing characteristics are as follows: the timing of acceleration and heel pressure is very abnormal, with frequent occurrences of abnormal forefoot pressure following acceleration; the timing of forefoot and heel pressure is very abnormal, with frequent occurrences of forefoot pressure followed by simultaneous forefoot and heel pressure; and the acceleration is very abnormal, with lateral acceleration not being zero.

[0129] The doctor-interactive verification process optimizes and applies the assessment system through multi-dimensional comparisons. In designing these comparison dimensions, one focus is on the correlation between the total score and the doctor's assessment level. By statistically analyzing the consistency rate between the abnormality level (mild / moderate / severe) corresponding to the system's total score and the doctor's subjective judgment across multiple datasets, a consistency standard of ≥85% is set. If the consistency rate falls below this threshold, the weighting of the scoring details is adjusted to improve system accuracy. Another aspect is analyzing the effectiveness of auxiliary methods. By comparing gait analysis results after interventions with different auxiliary methods, the actual effectiveness of various auxiliary methods is quantitatively evaluated. The application of the comparison results is reflected in two aspects: first, assisting doctors in decision-making by providing objective and quantitative assessment data, reducing judgment biases caused by differences in doctors' subjective experience, such as differences in the definition of "mild abnormality"; second, supporting the development of personalized rehabilitation plans by clarifying the effectiveness of certain auxiliary methods for specific patients based on the comparison results, helping doctors accurately select intervention methods for children and improving the targeting and effectiveness of rehabilitation treatment.

[0130] This application describes embodiments with reference to flowchart illustrations and / or block diagrams of methods, apparatuses, electronic devices, and computer program products according to embodiments of this application. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing terminal device to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing terminal device, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0131] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing terminal device to operate in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0132] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal equipment, causing a series of operational steps to be performed on the computer or other programmable terminal equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable terminal equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0133] Although preferred embodiments of the present application have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of the embodiments of the present application.

[0134] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other.

[0135] Although preferred embodiments of the present application have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of the embodiments of the present application.

[0136] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or terminal device that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or terminal device. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or terminal device that includes said element.

[0137] The quantitative evaluation method based on multimodal data provided in this application has been described in detail above. Specific examples have been used to illustrate the principles and implementation methods of this application. The description of the above embodiments is only for the purpose of helping to understand the method and its core ideas. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A quantitative evaluation method based on multimodal data, characterized in that, include: S100: Collects plantar pressure data through two pressure sensors located on the forefoot and heel, and collects calf movement data through a 6-axis IMU sensor located on the calf. S200. Perform data processing and feature extraction on plantar pressure data and lower leg movement data to obtain gait parameters, specifically including: S210. Preprocess the plantar pressure data and lower leg motion data to obtain short-time energy signals, acceleration variance and angular velocity integral. S220. Based on short-time energy signals and lower leg movement data, gait event points are identified through an adaptive threshold detection method, and gait cycles are divided according to gait event points; S230. Within the gait event point, obtain the phase observation value through template matching, and update the template based on the phase observation value; S240. Based on phase observations and lower leg movement data, the gait phase is estimated by fusing the extended Kalman filter (EKF) algorithm. S250, based on gait phase and short-time energy signals, the pressure center COP proxy value is obtained; S260. Extract gait parameters in each gait cycle. Gait parameters include acceleration variance, angular velocity integral, pressure center variation range, support phase duration, and support phase percentage. S300: Evaluate the gait parameters and generate evaluation results.

2. The quantitative evaluation method based on multimodal data according to claim 1, characterized in that, The preprocessing in S210 includes high-pass filtering and low-pass filtering of the user's plantar pressure data, and converting the calf movement data into acceleration and angular velocity in physical units; Based on the preprocessed acceleration and angular velocity, the acceleration variance and angular velocity integral are obtained; Based on the preprocessed plantar pressure data, the formula for calculating the short-time energy signal is as follows: Where E[n] is the merged short-time energy signal, W is the half-width of the window used to calculate the short-time energy, and m is the summation index. The process iterates within the window... This is the forefoot pressure signal after high-pass and low-pass filtering. This is the hind foot pressure signal after high-pass and low-pass filtering.

3. The quantitative evaluation method based on multimodal data according to claim 1, characterized in that, The S220 employs an adaptive threshold detection method based on the median and the absolute deviation of the median. The threshold calculation formula is as follows: + in, The adaptive threshold is calculated for the energy signal, and E[n] is the merged short-time energy signal. The median of the combined energy signal E[n] is given by k, where k is the threshold coefficient.

4. The quantitative evaluation method based on multimodal data according to claim 1, characterized in that, The specific steps for S230 template matching are as follows: At the gait event point Extract a pressure signal segment of length L from the vicinity; The pressure signal segment and the template signal are cross-correlated and normalized to obtain a cross-correlation sequence. Find the lag value that maximizes the cross-correlation coefficient sequence and map the lag value to the phase observation value; Specifically, when the maximum cross-correlation sequence exceeds a preset confidence threshold, the template is also updated. Where T is the template vector being maintained and updated, α is the learning rate for template updates, ρ is the confidence threshold for template updates, and S is the learning rate at event point. The current signal segment of length L is captured nearby.

5. The quantitative evaluation method based on multimodal data according to claim 1, characterized in that, The formula for calculating the COP proxy value is as follows: in, Let n be the pressure center proxy value at time n. This is the installation location for the forefoot pressure sensor. This is the installation location for the hindfoot pressure sensor. It is a very small constant; The range of pressure center variation is obtained by taking the difference between the maximum and minimum COP values ​​within one gait cycle.

6. The quantitative evaluation method based on multimodal data according to claim 1, characterized in that, The extraction of the support phase duration and support phase proportion mentioned in S260 includes: Calculate the mean value of z-axis acceleration within one gait cycle. with standard deviation ; by ±κ The first local maximum value is used as the landing point for threshold detection. The last local minimum is taken as the distance from the location. ; The duration of the support phase is: in, Let be the duration of the support phase in the i-th gait cycle; The proportion of support is: in, Let be the total duration of the i-th gait cycle. The proportion of support phase in the i-th gait cycle.

7. The quantitative evaluation method based on multimodal data according to claim 1, characterized in that, The S300 also includes determining the coordination of the ankle and knee joints based on plantar pressure data and calf movement data, respectively. Ankle coordination is assessed by calculating the ratio of forefoot pressure to heel pressure during the initial ground contact phase. If the ratio of forefoot pressure to heel pressure is greater than the statistical threshold for healthy individuals, it is considered an abnormality in ankle function. Knee joint coordination is assessed by calculating the root mean square value of the acceleration in the left and right directions of the lower leg during the gait cycle. If the root mean square value is greater than the statistical threshold for healthy individuals, it is judged as an abnormal knee joint stability.

8. The quantitative evaluation device based on multimodal data according to any one of claims 1-7, characterized in that, The device includes: The data acquisition module is used to collect plantar pressure data and calf movement data through pressure sensors and IMU sensors; The data processing module is used for data preprocessing, event detection, phase estimation, COP calculation, and support phase detection. The scoring and diagnosis module is used to perform periodic scoring and coordination assessment based on gait parameters, and generate evaluation results. The data storage and transmission module is used to upload data to the cloud platform and associate it with medical record information.

9. An electronic device, characterized in that, include: A processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the program, implements the quantitative evaluation method based on multimodal data as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, A computer program is stored on the computer-readable storage medium, which, when executed by a processor, implements the quantitative evaluation method based on multimodal data as described in any one of claims 1 to 7.

Images