Foot motion state reconstruction method and device based on multi-source sensor data fusion

Abstract

This invention discloses a method and apparatus for reconstructing foot motion state based on multi-source sensor data fusion, relating to the field of data processing. The method includes: acquiring data sequences from a plantar pressure sensor array, an IMU data sequence, and an electromagnetic positioning data sequence, performing time synchronization and feature extraction to obtain electromagnetic positioning features, inertial features, pressure features, and constraint features at the current moment, respectively, to construct a multi-dimensional feature vector; inputting the multi-dimensional feature vector into a trained confidence evaluation model to obtain a confidence vector at the current moment; using an error state Kalman filter to dynamically modulate the filtering parameters based on the confidence vector at the current moment to obtain the optimal posterior state vector and posterior error covariance matrix at the current moment; and reconstructing the foot trajectory and plantar pressure distribution using forward kinematics based on the optimal posterior state vectors at different moments, generating gait features. This invention addresses the problems of poor long-term accuracy, weak environmental adaptability, and low efficiency in utilizing multi-source data.

Description

Technical Field

[0001] This invention relates to the field of data processing, and specifically to a method and apparatus for reconstructing foot motion state based on multi-source sensor data fusion. Background Technology

[0002] In fields reliant on precise motion capture, the digital reconstruction of foot movements is a key technology bridging the physical world and the digital space. Existing solutions suffer from the following bottlenecks at the data processing algorithm level:

[0003] 1. The algorithm model lacks fundamental suppression of error accumulation:

[0004] The current mainstream algorithms based on microelectromechanical systems (MEMS) inertial measurement units (IMUs) have state recursion equations. It depends on the integration of acceleration and angular velocity. Due to the inherent zero-bias instability and random walk noise of the sensor, the error covariance matrix... The error will increase exponentially or even cubically over time, leading to the phenomenon of "integral drift". Simple zero-speed correction can only correct the error at a static instant and cannot solve the problem of accumulated error in the dynamic process, resulting in poor long-term accuracy.

[0005] 2. Rigid algorithmic framework for multi-source data fusion:

[0006] Existing multi-sensor fusion algorithms (such as Extended Kalman Filter (EKF) or complementary filters) typically employ static stochastic models. Their observation noise covariance matrix... The fixed-gain matrix, set as a fixed constant during initialization, cannot respond to real-time changes in sensor data quality. For example, when an electromagnetic positioning sensor experiences multipath effects due to interference from environmental metal objects, or when a pressure sensor produces artifacts due to slippage inside a shoe, the fixed-gain algorithm will still fully accept this erroneous data, leading to divergent state estimation. Existing technologies lack an adaptive "rejection" or "soft weighting" mechanism based on real-time feedback of data quality.

[0007] 3. Semantic constraints in cross-modal data are not fully encoded:

[0008] Existing methods mostly remain at a shallow fusion level, either at the data layer (direct splicing) or the feature layer (vector concatenation), failing to embed high-level biomechanical / physical semantics, such as "the center of plantar pressure (COP) must be located within the support surface" and "a specific foot posture angle is strongly correlated with the direction of ground reaction force," into the algorithm framework as mathematical constraints. This results in the inability to utilize prior physical knowledge for intelligent arbitration and consistency repair when data from different sensors conflict, leading to weak environmental adaptability. Summary of the Invention

[0009] The purpose of this application is to propose a method and device for reconstructing foot motion state based on multi-source sensor data fusion to address the aforementioned technical problems.

[0010] In a first aspect, the present invention provides a method for reconstructing foot motion state based on multi-source sensor data fusion, comprising the following steps:

[0011] S1, Construct and train the confidence assessment model to obtain the trained confidence assessment model;

[0012] S2, acquire the plantar pressure sensor array data sequence, IMU data sequence and electromagnetic positioning data sequence and synchronize them in time to obtain the synchronized plantar pressure sensor array data sequence, IMU data sequence and electromagnetic positioning data sequence. Extract features from the synchronized plantar pressure sensor array data sequence, IMU data sequence and electromagnetic positioning data sequence to obtain the electromagnetic positioning features, inertial features and pressure features at the current time. Extract the constraint features at the current time based on the predicted attitude at the previous time and the electromagnetic positioning data sequence at the current time based on physical consistency.

[0013] S3. Construct a multi-dimensional feature vector based on the electromagnetic positioning features, inertial features, pressure features, and constraint features at the current moment. Input the multi-dimensional feature vector into the trained confidence evaluation model to obtain the confidence vector at the current moment.

[0014] S4. Based on the confidence vector at the current moment, the error state Kalman filter is used to dynamically modulate the filtering parameters to obtain the optimal posterior state vector and posterior error covariance matrix at the current moment.

[0015] S5. Repeat steps S2-S4 until the optimal posterior state vector and posterior error covariance matrix at different times are obtained. Based on the optimal posterior state vector at different times, the trajectory of the foot and the plantar pressure distribution are reconstructed using forward kinematics, and gait features are generated.

[0016] Preferably, the electromagnetic positioning features include electromagnetic signal quality indicators and position change amplitude; the inertial features include acceleration magnitude, angular velocity magnitude and local slip variance; the pressure features include pressure distribution entropy, total pressure and pressure change rate; and the constraint features include observation residuals and pressure bottoming-out indicators.

[0017] As a preferred method, the extraction process of electromagnetic positioning features is as follows:

[0018] Select the data from the previous time in the synchronized electromagnetic positioning data sequence. The synchronized electromagnetic positioning data at each moment is used to calculate the electromagnetic signal quality index at the current moment, as shown in the following formula:

[0019] ;

[0020] in, Represents the normalization function. Indicates the first Electromagnetic positioning data synchronized at each moment. Indicates the first Electromagnetic positioning data synchronized at each moment. This indicates the quality index of the electromagnetic signal at the current moment. This represents a very small coefficient. Describes the Euclidean norm. Indicates the current moment;

[0021] The range of position change is Its expression is: ;

[0022] The process of extracting pressure features is as follows:

[0023] Select the current moment's plantar pressure sensor array data from the synchronized plantar pressure sensor array data sequence, and perform a weighted calculation by combining the fixed position coordinates of each pressure sensor in the insole coordinate system with the measured pressure value to obtain the pressure center coordinates at the current moment, as shown in the following formula:

[0024] ;

[0025] The coordinates of each pressure sensor in the insole coordinate system are obtained by transforming the coordinates of the foot pressure sensor array and the first extrinsic parameter transformation matrix. Indicates the coordinates of the pressure center at the current moment. and This represents the x-axis and y-axis coordinates of the pressure center at the current moment. The fixed position coordinates of the pressure sensor in the insole coordinate system are: , Indicates the first The x-axis and y-axis coordinates of each pressure sensor in the insole coordinate system To represent the transpose of a matrix, This indicates the total number of pressure sensors in the plantar pressure sensor array; Indicates the current time's... The pressure value measured by a pressure sensor;

[0026] The normalized probability distribution value corresponding to each pressure sensor at the current moment is calculated as follows:

[0027] ;

[0028] in, Indicates the current time's... The pressure value measured by a pressure sensor. Indicates the current time. The normalized probability distribution values ​​corresponding to each pressure sensor;

[0029] The pressure distribution entropy at the current moment is obtained by calculating the Shannon entropy of the normalized probability distribution value corresponding to each pressure sensor at the current moment. As shown in the following formula:

[0030] ;

[0031] The total pressure at the current moment is calculated based on the pressure values ​​measured by each pressure sensor at the current moment, as shown in the following formula:

[0032] ;

[0033] in, This represents the total pressure at the current moment;

[0034] The rate of change of pressure at the current moment is calculated by comparing the total pressure at the current moment with the total pressure at the previous moment, as shown in the following formula:

[0035] ;

[0036] in, This represents the total pressure at the previous moment. Indicates the sampling time interval. This represents the rate of change of pressure at the current moment;

[0037] The process of extracting inertial features is as follows:

[0038] Select the three-axis acceleration vector at the current moment from the synchronized IMU data sequence. and the three-axis angular velocity vector , These represent the accelerations along the x-axis, y-axis, and z-axis, respectively. The angular velocities are the x-axis, y-axis, and z-axis, respectively, using a length of... The sliding time window establishes the window index set for the current moment as follows: , Indicates the window index;

[0039] The acceleration magnitude and angular velocity magnitude at the current moment are calculated using the following formulas:

[0040] ;

[0041] ;

[0042] The local mean within the set of window indices at the current time is calculated using the following formula. :

[0043] ;

[0044] The local sliding variance at the current time is calculated using the following formula. :

[0045] ;

[0046] The process of extracting constraint features is as follows:

[0047] Construct a rotation matrix from the IMU coordinate system to the world coordinate system based on the predicted attitude from the previous moment. As shown in the following formula:

[0048] ;

[0049] in, This indicates the predicted posture at the previous moment. A quaternion representing the predicted attitude;

[0050] The virtual plantar pressure center observation value at the current moment is calculated based on the ground anchor point coordinates, the rotation matrix from the IMU coordinate system to the world coordinate system, and the position vector of the current plantar pressure center in the IMU coordinate system, as shown in the following formula:

[0051] ;

[0052] in, This represents the position vector of the plantar pressure center in the IMU coordinate system. , , This indicates the distance between the IMU mounting location and the sole of the shoe; Indicates the coordinates of the ground anchor point; This represents the current virtual plantar pressure center observation value;

[0053] like If the pressure exceeds a preset pressure threshold and the duration reaches a preset time, the pressure contact flag is set to 1; otherwise, the pressure contact flag is set to 0. When the pressure contact flag is 1, the Euclidean distance residual between the current electromagnetic positioning data and the observed value of the virtual plantar pressure center is calculated to obtain the observation residual, as shown in the following formula:

[0054] ;

[0055] in, This represents the observation residual.

[0056] Preferably, the confidence assessment model includes a multilayer perceptron, and the confidence vector at the current time is... ,in These represent the confidence levels of electromagnetic positioning data, pressure center data, and IMU data, respectively.

[0057] Preferably, step S4 specifically includes:

[0058] In the state definition phase, the state vector of foot movement is constructed. ,in, For the feet in the world coordinate system The position vector in the middle, Represents the set of real numbers; Let be the velocity vector of the foot in the world coordinate system; To describe from the foot coordinate system To the world coordinate system A unit quaternion vector with rotational relationships; These are the time-varying zero bias of the acceleration timing and the time-varying zero bias of the gyroscope, respectively;

[0059] Constructing the error state vector ,in, For the feet in the world coordinate system The position error vector in the middle; This is the velocity error vector of the foot in the world coordinate system; Here is the attitude error angle vector; These are the zero bias errors of the accelerometer and the gyroscope, respectively.

[0060] In the prediction step, the prior state estimation vector, prediction error covariance matrix, and error state transition matrix for the current moment are calculated based on the posterior state estimate, posterior error covariance matrix, and triaxial acceleration and angular velocity data from the previous moment, as shown in the following equation:

[0061] ;

[0062] ;

[0063] in, To control the input vector, its expression is: ; Let represent the optimal posterior state vector at the previous time step, and let the initial posterior state estimate be . , This represents the prior state estimate vector at the current moment. Let represent the posterior error covariance matrix of the previous time step. The initial posterior error covariance matrix is: ; This represents the prediction error covariance matrix at the current time. Represents a nonlinear state transition function. This represents the error state transition matrix at the current moment. This represents the process noise covariance matrix at the current moment.

[0064] Based on the confidence vector at the current moment The electromagnetic positioning observation noise sub-matrix, pressure center observation noise sub-matrix, and attitude observation noise sub-matrix at the current moment are calculated using the confidence mapping function, and the dynamic observation noise covariance matrix at the current moment is constructed in real time. As shown in the following formula: ; ; in, For confidence level categories, for , or , Indicates the reference noise. Indicates the current time after dynamic adjustment The corresponding observation noise covariance submatrix, Represents a nonlinear expansion function. Represents the block diagonal matrix construction operator;

[0065] During the correction step, the Kalman gain at the current time is calculated using the dynamic observation noise covariance matrix and the prediction error covariance matrix. The error state is then updated using the current Kalman gain and the prior state estimation vector to obtain the posterior error state vector at the current time, as shown in the following equation:

[0066] ;

[0067] ;

[0068] in, This represents the observation Jacobian matrix at the current time. Represents a nonlinear observation function. This represents the Kalman gain at the current moment. This represents the posterior error state vector at the current moment; This represents the actual observation vector at the current moment;

[0069] The posterior error state vector at the current time is injected into the prior state estimation vector at the current time to obtain the optimal posterior state vector at the current time. ;

[0070] The posterior error covariance matrix at the current time is calculated based on the prediction error covariance matrix and the Kalman gain, as shown in the following equation:

[0071] ;

[0072] in, This represents the posterior error covariance matrix at the current time. Represents the identity matrix.

[0073] Preferably, the foot trajectory and plantar pressure distribution are reconstructed using forward kinematics based on the optimal posterior state vectors at different times, and gait features are generated, specifically including:

[0074] Extract the position vector from the optimal posterior state vector at each time step to construct a set of foot trajectories in the world coordinate system. ;

[0075] The plantar pressure distribution is obtained by spatial interpolation of discrete pressure points using Gaussian radial basis functions.

[0076] Gait characteristics include cadence, foot clearance height, energy, inversion angle and / or eversion angle;

[0077] Extract the unit quaternion vector from the optimal posterior state vector at each time step and convert it into Euler angles, where the roll angle corresponds to the inward or outward roll angle, the pitch angle corresponds to the backbend angle or plantar flexion angle, and the yaw angle corresponds to the inward or outward spin angle.

[0078] Extract the velocity vector from the optimal posterior state vector at each time step, and define a decision function. As shown in the following formula:

[0079] ;

[0080] in, For indicator functions, Indicates the pressure threshold. Indicates the speed threshold. This represents the velocity vector of the foot in the world coordinate system at the current moment;

[0081] like If the jump changes to 1, it indicates a landing; if If the jump value becomes 0, it indicates that the foot has left the ground; the stride frequency is determined based on the time period from one heel strike to the next.

[0082] The step length is obtained by calculating the Euclidean distance of the foot on the horizontal plane between two consecutive landing events, as shown in the following formula:

[0083] ;

[0084] in, Indicates the first The position vector in the optimal posterior state vector at the time corresponding to the next landing event. Indicates the first The position vector in the optimal posterior state vector at the time corresponding to the next landing event;

[0085] The foot contour height is calculated using the following formula:

[0086] ;

[0087] in, Indicates a time of departure from the ground. By the time of landing Time period This represents the z-axis component of the position vector in the optimal posterior state vector within the time period. Indicates the ground reference height. Indicates foot contour height;

[0088] Based on a time of departure By the time of landing The moment of takeoff is calculated from the velocity vector, triaxial acceleration vector, and angular velocity vector in the optimal posterior state vector within the time period. By the time of landing Energy consumed within the time period As shown in the following formula:

[0089] ;

[0090] in, This represents the first derivative of the angular velocity vector with respect to time. For foot quality, Here is the rotational inertia matrix. It represents the acceleration due to gravity.

[0091] Secondly, the present invention provides a foot motion state reconstruction device based on multi-source sensor data fusion, comprising:

[0092] The model building module is configured to build and train a confidence assessment model to obtain a trained confidence assessment model.

[0093] The feature extraction module is configured to acquire and time-synchronize the plantar pressure sensor array data sequence, IMU data sequence, and electromagnetic positioning data sequence to obtain the synchronized plantar pressure sensor array data sequence, IMU data sequence, and electromagnetic positioning data sequence. It then performs feature extraction on the synchronized plantar pressure sensor array data sequence, IMU data sequence, and electromagnetic positioning data sequence to obtain the electromagnetic positioning features, inertial features, and pressure features at the current moment, respectively. Based on the predicted attitude at the previous moment and the electromagnetic positioning data sequence at the current moment, it extracts the constraint features at the current moment based on physical consistency.

[0094] The evaluation module is configured to construct a multi-dimensional feature vector based on the electromagnetic positioning features, inertial features, pressure features, and constraint features at the current moment, and input the multi-dimensional feature vector into the trained confidence evaluation model to obtain the confidence vector at the current moment.

[0095] The modulation module is configured to dynamically modulate the filtering parameters using the error state Kalman filter based on the confidence vector at the current time, so as to obtain the optimal posterior state vector and posterior error covariance matrix at the current time.

[0096] The repetition module is configured to repeatedly execute the feature extraction module to the modulation module until the optimal posterior state vector and posterior error covariance matrix at different times are obtained. Based on the optimal posterior state vector at different times, the trajectory of the foot and the plantar pressure distribution are reconstructed using forward kinematics, and gait features are generated.

[0097] Thirdly, the present invention provides an electronic device including one or more processors; and a storage device for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the method as described in any implementation of the first aspect.

[0098] Fourthly, the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method as described in any of the implementations of the first aspect.

[0099] Fifthly, the present invention provides a computer program product, including a computer program that, when executed by a processor, implements the method as described in any of the implementations in the first aspect.

[0100] Compared with the prior art, the present invention has the following beneficial effects:

[0101] (1) The foot motion state reconstruction method based on multi-source sensor data fusion mentioned in this invention transforms the biomechanical constraints between "pressure-posture-position" into mathematical residuals, and uses them as input to the confidence assessment model to dynamically adjust the filtering parameters. This closed-loop mechanism enables the automatic reliance on other physical constraints to maintain high-precision estimation when a single sensor fails (such as electromagnetic blockage), solving the problem of "intelligent arbitration when multi-source data conflict" and possessing physical semantic-driven closed-loop robustness.

[0102] (2) The foot motion state reconstruction method based on multi-source sensor data fusion mentioned in this invention uses a continuously differentiable expansion function. adjust It avoids the state jump caused by the traditional "threshold switching method" and can smoothly transition between "strong observation dependence" and "strong inference dependence" according to the environmental noise level, and has smooth adaptive anti-interference capability.

[0103] (3) The foot motion state reconstruction method based on multi-source sensor data fusion mentioned in this invention decouples the processing of attitude (manifold space) and position (Euclidean space) based on the error state representation of ESKF. New sensors (such as ultra-wideband UWB or visual odometry) only need to provide the observation equation. It can be seamlessly integrated into this framework and has extremely strong scalability. Attached Figure Description

[0104] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0105] Figure 1 This is a flowchart illustrating a foot motion state reconstruction method based on multi-source sensor data fusion, as an embodiment of this application.

[0106] Figure 2 The flowchart shows the mathematical derivation of the adaptive error state Kalman filter (ESKF) for the foot motion state reconstruction method based on multi-source sensor data fusion, which is an embodiment of this application.

[0107] Figure 3 A comparison of trajectory reconstruction errors between the foot motion state reconstruction method based on multi-source sensor data fusion and the traditional fixed covariance EKF method in a complex electromagnetic interference environment, as shown in the embodiments of this application.

[0108] Figure 4 This is a schematic diagram of a foot motion state reconstruction device based on multi-source sensor data fusion, as an embodiment of this application.

[0109] Figure 5 This is a schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0110] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this invention, and not all of them. Based on the embodiments of this invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this invention.

[0111] Figure 1 This application illustrates an embodiment of a foot motion state reconstruction method based on multi-source sensor data fusion, comprising the following steps:

[0112] S1. Construct and train the confidence assessment model to obtain the trained confidence assessment model.

[0113] In a specific embodiment, the confidence assessment model includes a multilayer perceptron, and the confidence vector at the current time is... ,in These represent the confidence levels of electromagnetic positioning data, pressure center data, and IMU data, respectively.

[0114] Specifically, embodiments of this application utilize a trained confidence assessment model. The input to this model is a multidimensional feature vector. The output is the normalized confidence vector corresponding to each data source. Multidimensional feature vectors It mainly consists of electromagnetic signal quality index (SNR), local sliding variance, pressure distribution entropy, and observation residual. It consists of electromagnetic positioning characteristics, inertial characteristics, pressure characteristics, and constraint characteristics. Represents the confidence vector The first component represents the reliability of the electromagnetic positioning data. The closer the value is to 1, the better the signal quality (no interference). A value close to 0 indicates the presence of severe metal interference or magnetic field distortion. Represents the confidence vector The second component represents the reliability of the center of pressure (COP) data, which is mainly used to determine whether the foot is actually in a landing state at the current moment. Represents the confidence vector The third component represents the dynamic adaptability of the IMU data. For example, this value will change during extremely vigorous movement (exceeding the measurement range) or prolonged periods of stillness (where the zero bias has a significant impact).

[0115] In one example, the confidence evaluation model employs a lightweight Multilayer Perceptron (MLP) to meet the real-time requirements of embedded systems, specifically including an input layer. The input layer consists of three layers: hidden layers and output layer. The number of nodes in the input layer is... (corresponding to the input feature vector) The hidden layer (a dimension) is used to receive normalized sensor feature data. The hidden layer consists of a first hidden layer (...) connected in sequence. ) and the second hidden layer ( The first hidden layer contains 16 neurons. The activation function is ReLU, used to introduce non-linear features and accelerate convergence; the second hidden layer contains 8 neurons. Output Layer... The dataset consists of three neurons, corresponding to the confidence levels of electromagnetic positioning data, pressure center data, and IMU data, respectively. The activation function used is the Sigmoid function, which forces the output values ​​to be compressed to the (0, 1) interval, making it convenient for direct use in weighting the covariance matrix.

[0116] S2, acquire the plantar pressure sensor array data sequence, IMU data sequence, and electromagnetic positioning data sequence and synchronize them in time to obtain the synchronized plantar pressure sensor array data sequence, IMU data sequence, and electromagnetic positioning data sequence. Extract features from the synchronized plantar pressure sensor array data sequence, IMU data sequence, and electromagnetic positioning data sequence to obtain the electromagnetic positioning features, inertial features, and pressure features at the current moment. Extract the constraint features at the current moment based on the predicted attitude at the previous moment and the electromagnetic positioning data sequence at the current moment according to physical consistency.

[0117] In a specific embodiment, the electromagnetic positioning features include electromagnetic signal quality indicators and position change amplitude; the inertial features include acceleration magnitude, angular velocity magnitude and local slip variance; the pressure features include pressure distribution entropy, total pressure and pressure change rate; and the constraint features include observation residuals and pressure bottoming-out indicators.

[0118] In a specific embodiment, the process of extracting electromagnetic positioning features is as follows:

[0119] Select the data from the previous time in the synchronized electromagnetic positioning data sequence. The synchronized electromagnetic positioning data at each moment is used to calculate the electromagnetic signal quality index at the current moment, as shown in the following formula:

[0120] ;

[0121] in, Represents the normalization function. Indicates the first Electromagnetic positioning data synchronized at each moment. Indicates the first Electromagnetic positioning data synchronized at each moment. This indicates the quality index of the electromagnetic signal at the current moment. This represents a very small coefficient. Describes the Euclidean norm. Indicates the current moment;

[0122] The range of position change is Its expression is: ;

[0123] The process of extracting pressure features is as follows:

[0124] Select the current moment's plantar pressure sensor array data from the synchronized plantar pressure sensor array data sequence, and perform a weighted calculation by combining the fixed position coordinates of each pressure sensor in the insole coordinate system with the measured pressure value to obtain the pressure center coordinates at the current moment, as shown in the following formula:

[0125] ;

[0126] The coordinates of each pressure sensor in the insole coordinate system are obtained by transforming the coordinates of the foot pressure sensor array and the first extrinsic parameter transformation matrix. Indicates the coordinates of the pressure center at the current moment. and This represents the x-axis and y-axis coordinates of the pressure center at the current moment. The fixed position coordinates of the pressure sensor in the insole coordinate system are: , Indicates the first The x-axis and y-axis coordinates of each pressure sensor in the insole coordinate system To represent the transpose of a matrix, This indicates the total number of pressure sensors in the plantar pressure sensor array; Indicates the current time's... The pressure value measured by a pressure sensor;

[0127] The normalized probability distribution value corresponding to each pressure sensor at the current moment is calculated as follows:

[0128] ;

[0129] in, Indicates the current time's... The pressure value measured by a pressure sensor. Indicates the current time. The normalized probability distribution values ​​corresponding to each pressure sensor;

[0130] The pressure distribution entropy at the current moment is obtained by calculating the Shannon entropy of the normalized probability distribution value corresponding to each pressure sensor at the current moment. As shown in the following formula:

[0131] ;

[0132] The total pressure at the current moment is calculated based on the pressure values ​​measured by each pressure sensor at the current moment, as shown in the following formula:

[0133] ;

[0134] in, This represents the total pressure at the current moment;

[0135] The rate of change of pressure at the current moment is calculated by comparing the total pressure at the current moment with the total pressure at the previous moment, as shown in the following formula:

[0136] ;

[0137] in, This represents the total pressure at the previous moment. Indicates the sampling time interval. This represents the rate of change of pressure at the current moment;

[0138] The process of extracting inertial features is as follows:

[0139] Select the three-axis acceleration vector at the current moment from the synchronized IMU data sequence. and the three-axis angular velocity vector , These represent the accelerations along the x-axis, y-axis, and z-axis, respectively. The angular velocities are the x-axis, y-axis, and z-axis, respectively, using a length of... The sliding time window establishes the window index set for the current moment as follows: , Indicates the window index;

[0140] The acceleration magnitude and angular velocity magnitude at the current moment are calculated using the following formulas:

[0141] ;

[0142] ;

[0143] The local mean within the set of window indices at the current time is calculated using the following formula. :

[0144] ;

[0145] The local sliding variance at the current time is calculated using the following formula. :

[0146] ;

[0147] The process of extracting constraint features is as follows:

[0148] Construct a rotation matrix from the IMU coordinate system to the world coordinate system based on the predicted attitude from the previous moment. As shown in the following formula:

[0149] ;

[0150] in, This indicates the predicted posture at the previous moment. A quaternion representing the predicted attitude;

[0151] The virtual plantar pressure center observation value at the current moment is calculated based on the ground anchor point coordinates, the rotation matrix from the IMU coordinate system to the world coordinate system, and the position vector of the current plantar pressure center in the IMU coordinate system, as shown in the following formula:

[0152] ;

[0153] in, This represents the position vector of the plantar pressure center in the IMU coordinate system. , , This indicates the distance between the IMU mounting location and the sole of the shoe; Indicates the coordinates of the ground anchor point; This represents the current virtual plantar pressure center observation value;

[0154] like If the pressure exceeds a preset pressure threshold and the duration reaches a preset time, the pressure contact flag is set to 1; otherwise, the pressure contact flag is set to 0. When the pressure contact flag is 1, the Euclidean distance residual between the current electromagnetic positioning data and the observed value of the virtual plantar pressure center is calculated to obtain the observation residual, as shown in the following formula:

[0155] ;

[0156] in, This represents the observation residual.

[0157] Specifically, the data sequence of the plantar pressure sensor array is acquired in parallel. IMU data sequences Electromagnetic positioning data sequence Timestamps of high-frequency IMU data sequences Using Hermite interpolation as a benchmark, a resampling algorithm is employed to align the low-frequency plantar pressure sensor array data sequence and the electromagnetic positioning data sequence to... At any time, generate a synchronization data frame. Further feature extraction yields electromagnetic positioning features, inertial features, pressure features, and constraint features. The electromagnetic positioning features include electromagnetic signal quality indicators. and the magnitude of positional change Inertial characteristics include acceleration magnitude. angular velocity magnitude and local sliding variance Pressure characteristics include pressure distribution entropy. Total pressure Pressure change rate Constraint features include observation residuals And pressure contact indicator (0 / 1). The above 10-dimensional feature data are connected to form a multi-dimensional feature vector.

[0158] The plantar pressure sensor array contains A separate pressure sensor, the first Each sensing unit is in the insole coordinate system The fixed position coordinates in are At the current moment The measured pressure value (or normalized voltage value) is Therefore, by calculating the pressure-weighted geometric center, the coordinates of the center of pressure (COP) can be obtained. .

[0159] Pressure distribution moment Used to describe the dispersion of pressure distribution, such as distinguishing between single-point strike and full-foot strike. Calculate about The pressure distribution moment can be obtained from the weighted covariance matrix. .

[0160] Calculating the pressure distribution entropy In the process, the pressure percentage of each pressure sensor sampling point is first calculated using a normalization method. Then, the Shannon entropy of the pressure distribution is calculated to obtain the pressure distribution entropy, which is used to measure the "disorder" or "uniformity" of the plantar pressure distribution. The larger the value, the more uniform the pressure distribution (e.g., full foot strike), and the calculated COP (center of pressure) is very stable and reliable with high confidence. The smaller the value, the more concentrated the pressure (e.g., toe strike or accidental heel strike due to insole slippage), and the COP is extremely unstable, and the confidence level should be reduced.

[0161] The total pressure is then calculated using the pressure values ​​collected by all pressure sensors. The rate of change was calculated using the first-order backward difference method. This is to ensure real-time causality.

[0162] Calculate the acceleration magnitude, angular velocity magnitude, and local slip variance based on the synchronized IMU data sequence. This is used for subsequent static state detection. A sliding window algorithm is used. A time window is set. (For example, W = 0.2 seconds). The local sliding variance can be obtained by calculating the variance of the acceleration magnitude within this time window. This applies if and only if... Approximately gravitational acceleration and Below the preset threshold When the foot is determined to be stationary (ZeroVelocityInterval), the Zero Velocity Adjustment (ZUPT) mechanism can be triggered.

[0163] By utilizing the local jitter of electromagnetic data, electromagnetic positioning data from the past N frames is acquired, and its standard deviation is calculated. If the IMU shows a stationary or uniform motion state, but the standard deviation of the electromagnetic positioning data is large, it indicates the presence of magnetic interference noise. Therefore, an electromagnetic signal quality index is proposed as one of the characteristics of electromagnetic positioning. The position change amplitude is calculated by the magnitude of the difference between the electromagnetic positioning data at the current moment and the electromagnetic positioning data at the previous moment.

[0164] In addition to the sensor characteristics mentioned above, data quality assessment under physical semantic constraints is also required. Based on rigid body kinematics and biomechanical models, a physical consistency index is calculated. This is based on the predicted attitude from the previous moment. and The COP readings are used to calculate the spatial position of the foot at the assumed ground contact point using forward kinematics, generating virtual plantar pressure center observations. The specific process is as follows:

[0165] Constructing a rotation matrix from the IMU coordinate system to the world coordinate system using predicted pose According to rigid body geometry, the position of the contact point in the world coordinate system is the sum of the position of the IMU in the world coordinate system and the rotated COP offset vector. Therefore, rearranging the terms yields the final virtual plantar pressure center observation value. .

[0166] Calculate electromagnetic positioning data The observation residual is obtained by calculating the Euclidean distance residual between the observed value and the virtual plantar pressure center. This observation residual reflects the degree of conflict between "electromagnetic perception" and "mechanical perception".

[0167] S3. Construct a multi-dimensional feature vector based on the electromagnetic positioning features, inertial features, pressure features, and constraint features at the current moment. Input the multi-dimensional feature vector into the trained confidence evaluation model to obtain the confidence vector at the current moment.

[0168] For details, please refer to Figure 2 The confidence vector at the current time is obtained by inputting the multidimensional feature vector at the current time into the trained confidence evaluation model.

[0169] S4. Based on the confidence vector at the current moment, the error state Kalman filter is used to dynamically modulate the filtering parameters to obtain the optimal posterior state vector and posterior error covariance matrix at the current moment.

[0170] In a specific embodiment, step S4 specifically includes:

[0171] In the state definition phase, the state vector of foot movement is constructed. ,in, For the feet in the world coordinate system The position vector in the middle, Represents the set of real numbers; Let be the velocity vector of the foot in the world coordinate system; To describe from the foot coordinate system To the world coordinate system A unit quaternion vector with rotational relationships; These are the time-varying zero bias of the acceleration timing and the time-varying zero bias of the gyroscope, respectively;

[0172] Constructing the error state vector ,in, For the feet in the world coordinate system The position error vector in the middle; This is the velocity error vector of the foot in the world coordinate system; Here is the attitude error angle vector; These are the zero bias errors of the accelerometer and the gyroscope, respectively.

[0173] In the prediction step, the prior state estimation vector, prediction error covariance matrix, and error state transition matrix for the current moment are calculated based on the posterior state estimate, posterior error covariance matrix, and triaxial acceleration and angular velocity data from the previous moment, as shown in the following equation:

[0174] ;

[0175] ;

[0176] in, To control the input vector, its expression is: ; Let represent the optimal posterior state vector at the previous time step, and let the initial posterior state estimate be . , This represents the prior state estimate vector at the current moment. Let represent the posterior error covariance matrix of the previous time step. The initial posterior error covariance matrix is: ; This represents the prediction error covariance matrix at the current time. Represents a nonlinear state transition function. This represents the error state transition matrix at the current moment. This represents the process noise covariance matrix at the current moment.

[0177] Based on the confidence vector at the current moment The electromagnetic positioning observation noise sub-matrix, pressure center observation noise sub-matrix, and attitude observation noise sub-matrix at the current moment are calculated using the confidence mapping function, and the dynamic observation noise covariance matrix at the current moment is constructed in real time. As shown in the following formula: ; ; in, For confidence level categories, for , or , Indicates the reference noise. Indicates the current time after dynamic adjustment The corresponding observation noise covariance submatrix, Represents a nonlinear expansion function. Represents the block diagonal matrix construction operator;

[0178] During the correction step, the Kalman gain at the current time is calculated using the dynamic observation noise covariance matrix and the prediction error covariance matrix. The error state is then updated using the current Kalman gain and the prior state estimation vector to obtain the posterior error state vector at the current time, as shown in the following equation:

[0179] ;

[0180] ;

[0181] in, This represents the observation Jacobian matrix at the current time. Represents a nonlinear observation function. This represents the Kalman gain at the current moment. This represents the posterior error state vector at the current moment; This represents the actual observation vector at the current moment;

[0182] The posterior error state vector at the current time is injected into the prior state estimation vector at the current time to obtain the optimal posterior state vector at the current time. ;

[0183] The posterior error covariance matrix at the current time is calculated based on the prediction error covariance matrix and the Kalman gain, as shown in the following equation:

[0184] ;

[0185] in, This represents the posterior error covariance matrix at the current time. Represents the identity matrix.

[0186] Specifically, the embodiments of this application use Error State Kalman Filter (ESKF) as the core fusion engine, and dynamically modulate the filter parameters through confidence vector.

[0187] Construct the nominal state vector of foot movement :

[0188] ;

[0189] in, For the feet in the world coordinate system The three-dimensional position in; The three-dimensional velocity of the foot in the world coordinate system; To describe from the foot body coordinate system To the world coordinate system Unit quaternions of rotational relations (using Hamiltonian specification); These are the time-varying zero biases of the accelerometer and gyroscope, respectively. This nominal state vector records the most intuitive physical quantities of the foot at each moment (position, velocity, attitude quaternion, zero bias). It is used to perform the kinematic integration in step S4, that is, to calculate the pose at the next moment based on the IMU data.

[0190] Initialize the nominal state vector and its error covariance matrix Load the pre-calibrated set of extrinsic transformation matrices. It is used to unify the coordinate systems of various sensors.

[0191] Define the error state vector :

[0192] ;

[0193] in, The error state vector is of size . The column vector does not directly represent the trajectory of the foot, but rather the small deviation between the "currently calculated trajectory" and the "true trajectory". In the ESKF architecture, the filter only estimates this error state vector, instead of directly estimating the nominal state vector, which ensures numerical stability on the rotating manifold. It represents the true state vector, the absolute motion state of the foot in the physical world. This represents the nominal state vector, calculated solely from kinematic integration of IMU data (acceleration and angular velocity). Since it is uncorrected, it includes accumulated error. This represents the generalized subtraction operator for manifolds. For Euclidean space variables (position, velocity, zero bias), it is a standard vector subtraction. For non-Euclidean space attitude quaternions... They cannot be directly subtracted. The operation is defined as follows: It represents mapping the difference between two quaternions onto a rotation vector in the tangent space. Wherein, The inverse of the nominal pose quaternion. The quaternion represents the true posture of foot movement. This represents the position error vector. It is the difference between the true position and the nominal position in the three-dimensional coordinate system of the world coordinate system. ), This represents the position error along the X-axis in the world coordinate system. This represents the position error along the Y-axis in the world coordinate system. This represents the position error along the Z-axis in the world coordinate system. This represents the velocity error vector, which is the difference between the actual velocity and the nominal velocity in the world coordinate system. This represents the attitude error angle vector, which is the biggest difference from the nominal state vector. The nominal state vector uses 4-dimensional quaternions. The attitude is described, while the error state is described using a 3D rotation vector as a "tiny correction rotation." This avoids the problem of covariance matrix singularity and reduces the state dimension from 16 to 15. Physically, it refers to the surrounding area. A tiny angular deviation of the axis. This represents the zero bias error of the accelerometer, which is the difference between the true zero bias value and the currently estimated zero bias value. It is used to correct the constant error of the accelerometer as it drifts with temperature or time online. This represents the gyroscope zero bias error, which is the difference between the actual gyroscope zero bias and the currently estimated zero bias. This is a key correction term for suppressing angle drift. It is a combined vector containing position, pressure center mapping value, and attitude features.

[0194] In one example, a discrete-time error state recursion model is defined. For the estimated nominal state vector... The median integral method is used for prediction:

[0195] Location update: ;

[0196] Posture update: ,in, This is quaternion multiplication. This represents the attitude quaternion estimated based on the previous time step. The rotation matrix constructed from the IMU coordinate system to the world coordinate system. This represents the estimated time-varying zero bias of the accelerometer at the previous moment. This represents the estimated value of the time-varying zero bias of the gyroscope at the previous moment.

[0197] Corresponding error state vector linearized transition matrix for The matrix, whose key block matrix structure is as follows:

[0198]

[0199] in Represents the antisymmetric matrix operator of vectors. This represents the estimated value of the time-varying zero bias of the accelerometer. This represents the estimated value of the time-varying zero bias of the gyroscope.

[0200] In the prediction step (Time Update) phase, IMU data is used to drive the kinematic integration of the nominal state and derive the covariance propagation of the error state. The nonlinear state transition function used in this process describes the physical equations of the system dynamics (also known as the mechanical orchestration equations). It defines how to use acceleration and angular velocity to map the state from the previous moment to the current moment through physical integral formulas. IMU data is used as the control input vector. This is used to drive the evolution of the system state. Over time, due to the integral effect, the uncertainty (variance) naturally increases. The magnitude of the prediction error covariance matrix reflects this diffusion. The error state transition matrix is ​​a nonlinear function. The Jacobian matrix for error states. It describes how errors from the previous time step (e.g., angle error) propagate linearly and affect other errors (e.g., position error) at the current time step. The discrete process noise covariance matrix represents the new uncertainty introduced into the system from the previous time step to the current time step due to the noise of the IMU sensor itself (e.g., white noise, random walk), preventing the covariance matrix from being... Over-convergence (becoming too small) during the filtering process helps maintain the filter's sensitivity to new changes.

[0201] Based on the confidence vector Real-time construction of dynamic observation noise covariance matrix It mathematically defines the uncertainty of all observed data at the current moment. This differs from the fixed noise covariance matrix used in traditional methods. Unlike other applications, the dynamic observation noise covariance matrix in this application... It is a variable The function is such that it changes dynamically with changes in the environment.

[0202] In one example, to generate the confidence vector from the neural network output Converting this to a variance inflation coefficient, we design the following exponential mapping function as a nonlinear inflation function:

[0203] ;

[0204] ;

[0205] in The reference noise covariance is the one calibrated at the sensor's factory. To prevent small amounts from being divided by zero. When (At high confidence) denominator , The filter is working normally. When (In the case of strong interference) the denominator , Kalman gain The filter automatically ignores the observation and switches to pure inertial estimation.

[0206] In the Measurement Update phase: the Kalman gain is calculated based on the dynamic observation noise covariance matrix. And update the error state. Finally, update the posterior error state vector at the current time step. Inject the prior state estimation vector at the current time step After completing the correction, the optimal posterior state vector at the current time is obtained, and the error state vector is reset to zero. The injection operation depends on the mathematical space of the state variables.

[0207] S5. Repeat steps S2-S4 until the optimal posterior state vector and posterior error covariance matrix at different times are obtained. Based on the optimal posterior state vector at different times, the trajectory of the foot and the plantar pressure distribution are reconstructed using forward kinematics, and gait features are generated.

[0208] In a specific embodiment, the trajectory of the foot and the plantar pressure distribution are reconstructed using forward kinematics based on the optimal posterior state vectors at different times, and gait features are generated, specifically including:

[0209] Extract the position vector from the optimal posterior state vector at each time step to construct a set of foot trajectories in the world coordinate system. ;

[0210] The plantar pressure distribution is obtained by spatial interpolation of discrete pressure points using Gaussian radial basis functions.

[0211] Gait characteristics include cadence, foot clearance height, energy, inversion angle and / or eversion angle;

[0212] Extract the unit quaternion vector from the optimal posterior state vector at each time step and convert it into Euler angles, where the roll angle corresponds to the inward or outward roll angle, the pitch angle corresponds to the backbend angle or plantar flexion angle, and the yaw angle corresponds to the inward or outward spin angle.

[0213] Extract the velocity vector from the optimal posterior state vector at each time step, and define a decision function. As shown in the following formula:

[0214] ;

[0215] in, For indicator functions, Indicates the pressure threshold. Indicates the speed threshold. This represents the velocity vector of the foot in the world coordinate system at the current moment;

[0216] like If the jump changes to 1, it indicates a landing; if If the jump value becomes 0, it indicates that the foot has left the ground; the stride frequency is determined based on the time period from one heel strike to the next.

[0217] The step length is obtained by calculating the Euclidean distance of the foot on the horizontal plane between two consecutive landing events, as shown in the following formula:

[0218] ;

[0219] in, Indicates the first The position vector in the optimal posterior state vector at the time corresponding to the next landing event. Indicates the first The position vector in the optimal posterior state vector at the time corresponding to the next landing event;

[0220] The foot contour height is calculated using the following formula:

[0221] ;

[0222] in, Indicates a time of departure from the ground. By the time of landing Time period This represents the z-axis component of the position vector in the optimal posterior state vector within the time period. express, Indicates foot contour height;

[0223] Based on a time of departure By the time of landing The moment of takeoff is calculated from the velocity vector, triaxial acceleration vector, and angular velocity vector in the optimal posterior state vector within the time period. By the time of landing Energy consumed within the time period As shown in the following formula:

[0224] ;

[0225] in, This represents the first derivative of the angular velocity vector with respect to time. For foot quality, Here is the rotational inertia matrix. It represents the acceleration due to gravity.

[0226] Specifically, steps S2-S4 are executed iteratively to obtain the optimal posterior state vector at each time step. The system reconstructs the three-dimensional trajectory, velocity curve, and plantar pressure distribution heatmap of the foot using forward kinematics, and generates gait features, including indicators such as stride length, stride frequency, and inversion / pronation angle.

[0227] By directly extracting the first three dimensions of the optimal posterior state vector at the current time step, the position vector at the current time step can be obtained. To eliminate high-frequency jitter, a 5-point cubic smoothing algorithm can be used to smooth the position vector, as shown in the following equation:

[0228] ;

[0229] in, It is the smoothed position vector. , , , This represents the position vectors of the two frames before and after the current moment. The position vector of the current moment is weighted and averaged using the position vectors of the two frames before and after. The weighting coefficients are derived based on local cubic polynomial fitting.

[0230] The unit quaternion in the current best posterior state vector. Convert to Euler angles (Roll, Pitch, Yaw), as shown in the following formula:

[0231]

[0232]

[0233]

[0234] , , These represent the roll angle, pitch angle, and yaw angle at the current moment, respectively.

[0235] Further automatic segmentation and feature calculation of gait cycles are performed to obtain stride length, foot clearance height, plantar pressure distribution (in the form of plantar pressure heatmap), and to estimate the energy consumed in each gait cycle.

[0236] In one of the complex workflow scenarios, the user's wearable device passes through a large metal medical device (generating strong magnetic interference) and undergoes the following process:

[0237] 1. Interference: The electromagnetic sensor readings change abruptly, resulting in a large deviation between the calculated position and the predicted position at the previous moment.

[0238] 2. Quality assessment: In step S2 of the embodiment of this application, the following was detected. The surge in confidence assessment, based on the decrease in input RSSI and the increase in residuals, leads the trained confidence assessment model to output the confidence level of the electromagnetic positioning data. .

[0239] 3. Parameter adjustment: In step S4 magnified times.

[0240] 4. State Update: When calculating the gain in ESKF, the electromagnetic observation weight is extremely low. It primarily relies on IMU integration to maintain the trajectory and utilizes... (If feet are on the ground) Limit drift.

[0241] 5. Interference Disappears: After the user leaves the electromagnetic interference area, the signal is restored, and the confidence level of the electromagnetic positioning data output by the trained confidence assessment model is restored. near It smoothly switches back to multi-source fusion mode, correcting the cumulative drift of IMU data during interference.

[0242] refer to Figure 3 This figure illustrates the comparison of foot position reconstruction accuracy between the traditional fixed-parameter fusion method and the adaptive fusion method proposed in this invention under a complex experimental environment with intermittent strong electromagnetic interference. The horizontal axis represents the system running time (in seconds, s), and the vertical axis represents the absolute position error of the reconstructed 3D trajectory (in meters, m). During normal operation (0 ~ 1.5 seconds): In the initial stage without external interference, both the red "traditional method" and the blue "method of this invention" maintain low pose errors close to 0, indicating comparable basic fusion accuracy. During the electromagnetic interference stage (approximately 1.5 seconds ~ 5 seconds, light blue shaded area in the figure): When entering a strong electromagnetic interference area, the readings of the electromagnetic positioning sensor experience severe jumps. If the traditional method (red dashed line) is used, because its filter's observation noise covariance matrix is ​​fixed, the algorithm cannot perceive the deterioration of data quality and still fully incorporates contaminated observations into the state update. This leads to rapid error divergence, with a peak position error as high as approximately 3.5 meters, resulting in complete trajectory failure. When using the method of this invention (blue solid line), thanks to the closed-loop feedback of the physical constraints and confidence evaluation network, the system keenly detects the abnormal increase in residuals. The algorithm triggers "adaptive..." The "suppression" mechanism instantaneously and dynamically amplifies the noise variance of electromagnetic observations, smoothly reducing their Kalman gain weights. During this period, the system mainly relies on gravity-constrained IMU integration to maintain trajectory extrapolation, strictly controlling the maximum position error within an extremely low range of approximately 0.5 meters. In the interference decay phase (after 5 seconds): after leaving the interference zone, the electromagnetic signal returns to normal. The method of this invention can converge rapidly and smoothly to zero error by relying on the recovery of confidence; although traditional methods also gradually converge, due to the large accumulated deviation in the early stage, their recovery process is accompanied by slight oscillations (downshoot).

[0243] Further reference Figure 4 As an implementation of the methods shown in the above figures, this application provides an embodiment of a foot motion state reconstruction device based on multi-source sensor data fusion. This device embodiment is similar to... Figure 1 Corresponding to the method embodiments shown, this device can be specifically applied to various electronic devices.

[0244] This application provides a foot motion state reconstruction device based on multi-source sensor data fusion, including:

[0245] Model building module 1 is configured to build and train a confidence assessment model to obtain a trained confidence assessment model;

[0246] Feature extraction module 2 is configured to acquire and synchronize the plantar pressure sensor array data sequence, IMU data sequence, and electromagnetic positioning data sequence, and obtain synchronized plantar pressure sensor array data sequence, IMU data sequence, and electromagnetic positioning data sequence. Feature extraction is performed on the synchronized plantar pressure sensor array data sequence, IMU data sequence, and electromagnetic positioning data sequence to obtain the electromagnetic positioning feature, inertial feature, and pressure feature at the current moment, respectively. Based on the predicted attitude at the previous moment and the electromagnetic positioning data sequence at the current moment, constraint features at the current moment are extracted based on physical consistency.

[0247] Evaluation module 3 is configured to construct a multi-dimensional feature vector based on the electromagnetic positioning features, inertial features, pressure features and constraint features at the current moment, and input the multi-dimensional feature vector into the trained confidence evaluation model to obtain the confidence vector at the current moment.

[0248] Modulation module 4 is configured to dynamically modulate the filtering parameters using error state Kalman filtering based on the confidence vector at the current time, so as to obtain the optimal posterior state vector and posterior error covariance matrix at the current time.

[0249] The repeating module 5 is configured to repeatedly execute the feature extraction module 2 to the modulation module 4 until the optimal posterior state vector and posterior error covariance matrix at different times are obtained. Based on the optimal posterior state vector at different times, the trajectory of the foot and the plantar pressure distribution are reconstructed using forward kinematics, and gait features are generated.

[0250] Figure 5 This is a schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present invention. For example... Figure 5 As shown, the electronic device of this embodiment includes a processor 501 and a memory 502; wherein the memory 502 is used to store computer execution instructions; and the processor 501 is used to execute the computer execution instructions stored in the memory to implement the various steps performed by the electronic device in the above embodiment. For details, please refer to the relevant descriptions in the foregoing method embodiments.

[0251] Alternatively, the memory 502 can be either standalone or integrated with the processor 501.

[0252] When the memory 502 is set up independently, the electronic device also includes a bus 503 for connecting the memory 502 and the processor 501.

[0253] This invention also provides a computer storage medium storing computer execution instructions, which, when executed by processor 501, implement the above method.

[0254] This invention also provides a computer program product, including a computer program that, when executed by a processor 501, implements the above-described method.

[0255] In the embodiments provided by this invention, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative; for instance, the division of modules is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple modules may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be indirect coupling or communication connection through some interfaces, devices, or modules, and may be electrical, mechanical, or other forms.

[0256] The modules described as separate components may or may not be physically separate. The components shown as modules may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to implement the solution of this embodiment according to actual needs.

[0257] Furthermore, the functional modules in the various embodiments of this invention can be integrated into one processing unit, or each module can exist physically separately, or two or more modules can be integrated into one unit. The unit formed by the above modules can be implemented in hardware or in the form of hardware plus software functional units.

[0258] The integrated modules implemented as software functional modules described above can be stored in a computer-readable storage medium. These software functional modules, stored in a storage medium, include several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor 501 to execute some steps of the methods of the various embodiments of this application.

[0259] It should be understood that the processor 501 described above can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), etc. The general-purpose processor can be a microprocessor, or the processor 501 can be any conventional processor 501. The steps of the method disclosed in this invention can be directly manifested as the hardware processor 501 executing the steps, or as a combination of hardware and software modules within the processor 501 executing the steps.

[0260] The memory 502 may include high-speed RAM memory, and may also include non-volatile memory NVM, such as at least one disk storage device, and may also be a USB flash drive, portable hard drive, read-only memory, disk or optical disc, etc.

[0261] Bus 503 can be an Industry Standard Architecture (ISA), a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. Bus 503 can be divided into address bus, data bus, control bus, etc. For ease of illustration, the bus 503 in the accompanying drawings of this application is not limited to only one bus 503 or one type of bus 503.

[0262] The aforementioned storage medium can be implemented from any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk. The storage medium can be any available medium accessible to general-purpose or special-purpose computers.

[0263] An exemplary storage medium is coupled to processor 501, enabling processor 501 to read information from and write information to the storage medium. Alternatively, the storage medium can be an integral part of processor 501. Processor 501 and storage medium can reside in application-specific integrated circuits (ASICs). Alternatively, processor 501 and storage medium can exist as discrete components in an electronic device or host device.

[0264] Those skilled in the art will understand that all or part of the steps of the above-described method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When executed, the program performs the steps of the above-described method embodiments; and the aforementioned storage medium includes various media capable of storing program code, such as ROM, RAM, magnetic disks, or optical disks.

[0265] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for reconstructing foot motion state based on multi-source sensor data fusion, characterized in that, Includes the following steps: S1, Construct and train the confidence assessment model to obtain the trained confidence assessment model; S2, acquire the plantar pressure sensor array data sequence, IMU data sequence and electromagnetic positioning data sequence and synchronize them in time to obtain the synchronized plantar pressure sensor array data sequence, IMU data sequence and electromagnetic positioning data sequence. Extract features from the synchronized plantar pressure sensor array data sequence, IMU data sequence and electromagnetic positioning data sequence to obtain the electromagnetic positioning features, inertial features and pressure features at the current time. Extract the constraint features at the current time based on the predicted attitude at the previous time and the electromagnetic positioning data sequence at the current time based on physical consistency. S3. Construct a multi-dimensional feature vector based on the electromagnetic positioning features, inertial features, pressure features, and constraint features at the current moment, and input the multi-dimensional feature vector into the trained confidence evaluation model to obtain the confidence vector at the current moment. S4. Based on the confidence vector at the current moment, the error state Kalman filter is used to dynamically modulate the filtering parameters to obtain the optimal posterior state vector and posterior error covariance matrix at the current moment. S5. Repeat steps S2-S4 until the optimal posterior state vector and posterior error covariance matrix at different times are obtained. Based on the optimal posterior state vector at different times, the trajectory of the foot and the plantar pressure distribution are reconstructed using forward kinematics, and gait features are generated.

2. The method for reconstructing foot motion state based on multi-source sensor data fusion according to claim 1, characterized in that, The electromagnetic positioning features include electromagnetic signal quality indicators and position change amplitude; the inertial features include acceleration magnitude, angular velocity magnitude, and local sliding variance; the pressure features include pressure distribution entropy, total pressure, and pressure change rate; and the constraint features include observation residuals and pressure bottoming-out indicators.

3. The method for reconstructing foot motion state based on multi-source sensor data fusion according to claim 2, characterized in that, The extraction process of the electromagnetic positioning features is as follows: Select the data from the previous time in the synchronized electromagnetic positioning data sequence. The synchronized electromagnetic positioning data at each moment is used to calculate the electromagnetic signal quality index at the current moment, as shown in the following formula: ； in, Represents the normalization function. Indicates the first Electromagnetic positioning data synchronized at each moment. Indicates the first Electromagnetic positioning data synchronized at each moment. This indicates the quality index of the electromagnetic signal at the current moment. This represents a very small coefficient. Describes the Euclidean norm. Indicates the current moment; The position change range is Its expression is: ; The extraction process of the pressure feature is as follows: Select the current moment's plantar pressure sensor array data from the synchronized plantar pressure sensor array data sequence, and perform a weighted calculation by combining the fixed position coordinates of each pressure sensor in the insole coordinate system with the measured pressure value to obtain the pressure center coordinates at the current moment, as shown in the following formula: ； The coordinates of each pressure sensor in the insole coordinate system are obtained by transforming the coordinates of the foot pressure sensor array and the first extrinsic parameter transformation matrix. Indicates the coordinates of the pressure center at the current moment. and This represents the x-axis and y-axis coordinates of the pressure center at the current moment. The fixed position coordinates of the pressure sensor in the insole coordinate system are: , Indicates the first The x-axis and y-axis coordinates of each pressure sensor in the insole coordinate system To represent the transpose of a matrix, This indicates the total number of pressure sensors in the plantar pressure sensor array; Indicates the current time's... The pressure value measured by a pressure sensor; The normalized probability distribution value corresponding to each pressure sensor at the current moment is calculated as follows: ； in, Indicates the current time's... The pressure value measured by a pressure sensor. Indicates the current time. The normalized probability distribution values ​​corresponding to each pressure sensor; The pressure distribution entropy at the current moment is obtained by calculating the Shannon entropy of the normalized probability distribution value corresponding to each pressure sensor at the current moment. As shown in the following formula: ； The total pressure at the current moment is calculated based on the pressure values ​​measured by each pressure sensor at the current moment, as shown in the following formula: ； in, This represents the total pressure at the current moment; The rate of change of pressure at the current moment is calculated by comparing the total pressure at the current moment with the total pressure at the previous moment, as shown in the following formula: ； in, This represents the total pressure at the previous moment. Indicates the sampling time interval. This represents the rate of change of pressure at the current moment; The extraction process of the inertial features is as follows: Select the triaxial acceleration vector at the current moment from the synchronized IMU data sequence. and the three-axis angular velocity vector , These represent the accelerations along the x-axis, y-axis, and z-axis, respectively. The angular velocities are the x-axis, y-axis, and z-axis, respectively, using a length of... The sliding time window establishes the window index set for the current moment as follows: , Indicates the window index; The acceleration magnitude and angular velocity magnitude at the current moment are calculated using the following formulas: ； ； The local mean within the set of window indices at the current time is calculated using the following formula. : ； The local sliding variance at the current time is calculated using the following formula. : ； The process of extracting the constraint features is as follows: Construct a rotation matrix from the IMU coordinate system to the world coordinate system based on the predicted attitude from the previous moment. As shown in the following formula: ； in, This indicates the predicted posture at the previous moment. A quaternion representing the predicted attitude; The virtual plantar pressure center observation value at the current moment is calculated based on the ground anchor point coordinates, the rotation matrix from the IMU coordinate system to the world coordinate system, and the position vector of the current plantar pressure center in the IMU coordinate system, as shown in the following formula: ； in, This represents the position vector of the plantar pressure center in the IMU coordinate system. , , This indicates the distance between the IMU mounting location and the sole of the shoe; Indicates the coordinates of the ground anchor point; This represents the current virtual plantar pressure center observation value; like If the pressure exceeds a preset pressure threshold and the duration reaches a preset time, the pressure contact flag is set to 1; otherwise, the pressure contact flag is set to 0. When the pressure contact flag is 1, the Euclidean distance residual between the current electromagnetic positioning data and the observed value of the virtual plantar pressure center is calculated to obtain the observation residual, as shown in the following formula: ； in, This represents the observation residual.

4. The method for reconstructing foot motion state based on multi-source sensor data fusion according to claim 3, characterized in that, The confidence assessment model includes a multilayer perceptron, and the confidence vector at the current moment is: ,in These represent the confidence levels of electromagnetic positioning data, pressure center data, and IMU data, respectively.

5. The method for reconstructing foot motion state based on multi-source sensor data fusion according to claim 4, characterized in that, Step S4 specifically includes: In the state definition phase, the state vector of foot movement is constructed. ,in, For the feet in the world coordinate system The position vector in the middle, Represents the set of real numbers; Let be the velocity vector of the foot in the world coordinate system; To describe from the foot coordinate system To the world coordinate system A unit quaternion vector with rotational relationships; These are the time-varying zero bias of the acceleration timing and the time-varying zero bias of the gyroscope, respectively; Constructing the error state vector ,in, For the feet in the world coordinate system The position error vector in the middle; This is the velocity error vector of the foot in the world coordinate system; Here is the attitude error angle vector; These are the zero bias errors of the accelerometer and the gyroscope, respectively. In the prediction step, the prior state estimation vector, prediction error covariance matrix, and error state transition matrix for the current moment are calculated based on the posterior state estimate, posterior error covariance matrix, and triaxial acceleration and angular velocity data from the previous moment, as shown in the following equation: ； ； in, To control the input vector, its expression is: ; Let represent the optimal posterior state vector at the previous time step, and let the initial posterior state estimate be . , This represents the prior state estimate vector at the current moment. Let represent the posterior error covariance matrix of the previous time step. The initial posterior error covariance matrix is: ; This represents the prediction error covariance matrix at the current time. Represents a nonlinear state transition function. This represents the error state transition matrix at the current moment. This represents the process noise covariance matrix at the current moment. Based on the confidence vector at the current moment The electromagnetic positioning observation noise sub-matrix, pressure center observation noise sub-matrix, and attitude observation noise sub-matrix at the current moment are calculated using the confidence mapping function, and the dynamic observation noise covariance matrix at the current moment is constructed in real time. As shown in the following formula: ； ； in, For confidence level categories, for , or , Indicates the reference noise. Indicates the current time after dynamic adjustment The corresponding observation noise covariance submatrix, Represents a nonlinear expansion function. Represents the block diagonal matrix construction operator; During the correction step, the Kalman gain at the current time is calculated using the dynamic observation noise covariance matrix and the prediction error covariance matrix. The error state is then updated using the current Kalman gain and the prior state estimation vector to obtain the posterior error state vector at the current time, as shown in the following equation: ； ； in, This represents the observation Jacobian matrix at the current time. Represents a nonlinear observation function. This represents the Kalman gain at the current moment. This represents the posterior error state vector at the current moment; This represents the actual observation vector at the current moment; The posterior error state vector at the current time is injected into the prior state estimation vector at the current time to obtain the optimal posterior state vector at the current time. ; The posterior error covariance matrix at the current time is calculated based on the prediction error covariance matrix and the Kalman gain, as shown in the following equation: ； in, This represents the posterior error covariance matrix at the current time. Represents the identity matrix.

6. The method for reconstructing foot motion state based on multi-source sensor data fusion according to claim 5, characterized in that, Based on the optimal posterior state vectors at different times, the foot trajectory and plantar pressure distribution are reconstructed using forward kinematics, and gait features are generated, specifically including: Extract the position vector from the optimal posterior state vector at each time step to construct a set of foot trajectories in the world coordinate system. ; The plantar pressure distribution is obtained by spatial interpolation of discrete pressure points using Gaussian radial basis functions. The gait characteristics include cadence, foot clearance height, energy, inversion angle and / or eversion angle; Extract the unit quaternion vector from the optimal posterior state vector at each time step and convert it into Euler angles, where the roll angle corresponds to the inward or outward roll angle, the pitch angle corresponds to the backbend angle or plantar flexion angle, and the yaw angle corresponds to the inward or outward spin angle. Extract the velocity vector from the optimal posterior state vector at each time step, and define a decision function. As shown in the following formula: ； in, For indicator functions, Indicates the pressure threshold. Indicates the speed threshold. This represents the velocity vector of the foot in the world coordinate system at the current moment; like If the jump changes to 1, it indicates a landing; if If the jump value becomes 0, it indicates that the foot has left the ground; the stride frequency is determined based on the time period from one heel strike to the next. The step length is obtained by calculating the Euclidean distance of the foot on the horizontal plane between two consecutive landing events, as shown in the following formula: ； in, Indicates the first The position vector in the optimal posterior state vector at the time corresponding to the next landing event. Indicates the first The position vector in the optimal posterior state vector at the time corresponding to the next landing event; The following formula is used to calculate the foot contour height: ； in, Indicates a time of departure from the ground. By the time of landing Time period This represents the z-axis component of the position vector in the optimal posterior state vector within the time period. Indicates the ground reference height. Indicates foot contour height; Based on a time of departure By the time of landing The moment of takeoff is calculated from the velocity vector, triaxial acceleration vector, and angular velocity vector in the optimal posterior state vector within the time period. By the time of landing Energy consumed within the time period As shown in the following formula: ； in, This represents the first derivative of the angular velocity vector with respect to time. For foot quality, Here is the rotational inertia matrix. It represents the acceleration due to gravity.

7. A foot motion state reconstruction device based on multi-source sensor data fusion, characterized in that, include: The model building module is configured to build and train a confidence assessment model to obtain a trained confidence assessment model. The feature extraction module is configured to acquire and time-synchronize the plantar pressure sensor array data sequence, IMU data sequence, and electromagnetic positioning data sequence to obtain the synchronized plantar pressure sensor array data sequence, IMU data sequence, and electromagnetic positioning data sequence. It then performs feature extraction on the synchronized plantar pressure sensor array data sequence, IMU data sequence, and electromagnetic positioning data sequence to obtain the electromagnetic positioning features, inertial features, and pressure features at the current moment, respectively. Based on the predicted attitude at the previous moment and the electromagnetic positioning data sequence at the current moment, it extracts the constraint features at the current moment based on physical consistency. The evaluation module is configured to construct a multi-dimensional feature vector based on the electromagnetic positioning features, inertial features, pressure features and constraint features at the current moment, and input the multi-dimensional feature vector into the trained confidence evaluation model to obtain the confidence vector at the current moment. The modulation module is configured to dynamically modulate the filtering parameters using the error state Kalman filter based on the confidence vector at the current time, so as to obtain the optimal posterior state vector and posterior error covariance matrix at the current time. The repetition module is configured to repeatedly execute the feature extraction module to the modulation module until the optimal posterior state vector and posterior error covariance matrix at different times are obtained. Based on the optimal posterior state vector at different times, the trajectory of the foot and the plantar pressure distribution are reconstructed using forward kinematics, and gait features are generated.

8. An electronic device, comprising: One or more processors; Storage device for storing one or more programs. When the one or more programs are executed by the one or more processors, the one or more processors implement the method as described in any one of claims 1-6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the method as described in any one of claims 1-6.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the method as described in any one of claims 1-6.

Images