A plantar posture monitoring method and system based on multi-modal data fusion

Abstract

The present application relates to the technical field of motion state monitoring, in particular to a plantar posture monitoring method and system based on multi-modal data fusion, comprising the following steps: based on the pressure array data of a single foot, the plantar flexion data and the forefoot IMU data and the heel IMU data, unified packaging is carried out according to the preset spatial corresponding relationship; obtaining the plantar multi-modal posture state frame representing the plantar stress distribution and the front and rear partition posture state, wherein the state frame takes the plantar region as the unified reference datum. Based on the plantar multi-modal posture state frame, frame-level continuous analysis is carried out, and the plantar motion state evolution sequence is constructed according to the pressure change continuity and the inertial posture change trend. In the present application, the pressure array, two-way forefoot flexion sensor, forefoot IMU, heel IMU and battery state are packaged according to the fixed frame structure, the frame length is identified to identify the shoe size, the frame header is identified to identify the left and right feet, and the frame tail is checked for integrity, so that the data automatic analysis and consistent processing in the multi-shoe size and double-foot synchronous scene are realized.

Description

Technical Field

[0001] This invention relates to the field of motion state monitoring technology, and in particular to a method and system for monitoring plantar posture based on multimodal data fusion. Background Technology

[0002] Plantar pressure distribution, pressure center changes, forefoot flexion, forefoot posture, and heel posture are key data for gait analysis, exercise training, rehabilitation assessment, motion capture, and human-computer interaction. Traditional pressure plates, treadmills, or motion capture devices are usually deployed in fixed locations, making it difficult to reflect the true movement of the foot under natural walking, everyday wear, or mobile scenarios.

[0003] Existing smart insoles mostly use several discrete pressure points or a single inertial measurement unit for monitoring, which can realize step counting, pressure trend or simple posture display, but still have the following shortcomings: First, the acquisition mode is single, making it difficult to simultaneously reflect plantar pressure, forefoot flexion, forefoot local posture and heel local posture; Second, the number and arrangement of pressure points are different for different shoe sizes, and the host computer often needs to be manually configured, which can easily lead to mismatch of points; Third, there is a lack of clear and stable frame-level identification method when transmitting left and right foot data wirelessly, which may cause data confusion when both feet are working at the same time; Fourth, there is a lack of a unified data frame structure and processing flow between pressure array, flexion sensor, battery detection, dual IMU posture and CSV playback.

[0004] In the manufacturing of pressure sensors, common methods for implementing flexible pressure arrays include flexible circuit boards, electrode patches, or manual wiring. These methods suffer from problems such as numerous process steps, poor wiring consistency, insufficient repeatability of points, and uneven local thickness in scenarios involving curved insoles and multi-size adaptation. This makes it difficult to guarantee the correspondence of pressure points, sampling stability, and batch consistency between different shoe size prototypes.

[0005] Therefore, a wearable insole system is needed that integrates pressure arrays, bending sensors, forefoot IMUs, heel IMUs, main control / WiFi communication, data frame protocols, and host computer processing algorithms into a unified design. This would enable the insole to collect multi-source data at fixed intervals, and the host computer to automatically identify the left and right feet and shoe size. Furthermore, it would enable the display of pressure cloud maps, pressure center, bending progress, posture, data recording, and playback. Summary of the Invention

[0006] This invention provides a method and system for monitoring plantar posture based on multimodal data fusion.

[0007] A method for monitoring plantar posture based on multimodal data fusion includes the following steps:

[0008] The pressure array data, plantar flexion data, forefoot IMU data and heel IMU data of a single foot are uniformly packaged according to a preset spatial correspondence.

[0009] Obtain multimodal attitude state frames of the foot, which characterize the force distribution and the posture state of the front and rear sections of the foot, wherein the state frames use the foot region as a unified reference.

[0010] Based on the multimodal attitude state frames of the foot, frame-level continuous analysis is performed, and the evolution sequence of foot motion state is constructed according to the continuity of pressure change and the trend of inertial attitude change.

[0011] Gait state switching nodes are identified in this sequence, thereby generating temporally consistent plantar posture sequence data.

[0012] Based on the plantar posture sequence data, pressure distribution characteristics, bending change characteristics, and forefoot and hindfoot IMU posture characteristics are fused and calculated.

[0013] By combining the shoe size adaptive mapping rule and the left and right foot frame-level identification rule, standardized foot posture monitoring results are output.

[0014] The beneficial effects of this invention are:

[0015] This invention encapsulates the pressure array, two forefoot flexion sensors, forefoot IMU, heel IMU, and battery status into a fixed frame structure through a unified expression mechanism of multimodal foot posture state frames. It also achieves automatic data parsing and consistency processing in scenarios with multiple shoe sizes and dual-foot synchronization by identifying shoe size through frame length, identifying left and right feet through frame header, and verifying integrity through frame tail.

[0016] This invention uses a foot posture fusion method based on continuous pressure migration, forefoot flexion changes, and front and rear IMU posture differences to identify heel strike, foot weight-bearing, forefoot propulsion, and toe lift-off nodes in the evolution sequence of foot movement states. It also combines magnetic field effectiveness gating to achieve nine-axis fusion and six-axis degradation update, thereby improving the stability of foot posture monitoring in natural walking scenarios. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only for this invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 This is a schematic diagram of the IMU data block structure in Embodiment 1 of the present invention;

[0019] Figure 2This is a schematic diagram of the single-legged UDP data frame structure according to Embodiment 1 of the present invention;

[0020] Figure 3 This is a flowchart of the automatic shoe size recognition and automatic left / right foot distribution process in Embodiment 1 of the present invention;

[0021] Figure 4 This is a flowchart illustrating the pressure display, pressure center, and calibration process of Embodiment 1 of the present invention.

[0022] Figure 5 This is a flowchart of attitude zeroing, magnetic field calibration, and nine-axis fusion processing in Embodiment 1 of the present invention;

[0023] Figure 6 This is a statistical chart of the average total count of the pressure array in Embodiment 1 of the present invention;

[0024] Figure 7 This is a schematic diagram of the system modules in Embodiment 2 of the present invention. Detailed Implementation

[0025] The present invention will now be described in detail with reference to the accompanying drawings and specific embodiments. For some well-known technologies, those skilled in the art may also use other alternative methods to implement the invention. Moreover, the accompanying drawings are only for more specific description of the embodiments and are not intended to specifically limit the present invention.

[0026] Example 1

[0027] like Figures 1-6 As shown, a foot posture monitoring method based on multimodal data fusion includes the following steps:

[0028] S1, the pressure array data, plantar flexion data, forefoot IMU data and heel IMU data of a single foot are uniformly encapsulated according to a preset spatial correspondence to generate a plantar multimodal attitude state frame that characterizes the plantar force distribution and the front and rear partition posture state, wherein the state frame takes the plantar region as a unified reference.

[0029] S11. First, based on the foot structure of the insole, a foot anatomical reference benchmark is preset, and the insole plane is defined as a unified spatial coordinate domain. The unified spatial coordinate domain is used to provide the same spatial expression basis for pressure array data, foot flexion data, forefoot IMU data and heel IMU data, so that the data collected by different sensors can be calibrated and subsequently fused in the same foot area.

[0030] Let the planar region of the insole be represented as: ;in, Indicates shoe size The effective area corresponding to the insole in the plane; Indicates the shoe size type, such as size 38 or size 42; It represents a two-dimensional planar coordinate space.

[0031] In the flat area of ​​the insole Determine the geometric center area of ​​the heel. Based on the outer contour of the insole or the shoe size layout file, determine the overall front-to-back direction of the insole, that is, from the heel to the toe. Then, along this front-to-back direction, measure the planar area of ​​the insole. The length is normalized and divided, and a predetermined proportion area near the heel is selected as the candidate heel region. For example, the area about 15% to 25% of the longitudinal length of the insole is selected as the initial heel region. Then, based on the change in the left and right width of the insole outline within this region, the parts near the edge that are not part of the stable load-bearing area are eliminated. Only the area within the heel load-bearing range, with relatively continuous left and right width and a shape that is approximately elliptical or arc-shaped, is retained to obtain the geometric center region of the heel. The geometric center of the heel's geometric center region is taken as the origin of the unified reference coordinate system for the sole of the foot, and is represented as follows:

[0032] ;

[0033] in, This indicates the spatial origin of the unified reference coordinate system for the sole of the foot; Indicates the geometric center region of the heel; This represents the area of ​​the geometric center region of the heel. Represents the planar coordinates of any point within the geometric center region of the heel; This indicates that the area is being integrated.

[0034] In the discrete point file or insole contour point file, the spatial origin is represented as:

[0035] ;

[0036] in, Indicates the first heel geometric center region The coordinates of a discrete contour point or reference point; This indicates the number of discrete reference points within the geometric center region of the heel.

[0037] Based on the flat area of ​​the insole Determine the toe reference area in the forward and backward direction. The direction from the heel to the toe is taken as the longitudinal direction. A predetermined proportion of the foremost area of ​​the insole is selected as the candidate toe area, such as the area 5% to 15% of the forefoot closest to the toe. Then, based on the outer contour of the insole and the distribution of effective pressure points, the sharp edges, areas without sensing points, or unstable pressure areas are eliminated. The area located between the forefoot and the toe, with a continuous contour and consistent with the direction of the toe, is retained as the toe reference area. And obtain the toe reference point as follows:

[0038] ;

[0039] in, Indicates the toe reference point; Indicates the toe reference area; This represents the area of ​​the toe reference zone; This represents the planar coordinates of any point within the toe reference area.

[0040] From the origin of space Pointing to the toes reference point Using the direction as the vertical reference axis, we obtain the vertical unit vector:

[0041] ;

[0042] in, This represents the vertical unit vector in the unified reference coordinate system for the sole of the foot. This represents the direction vector from the geometric center region of the heel to the toe region; This represents the magnitude of the direction vector.

[0043] With perpendicular to the vertical unit vector Taking the direction as the horizontal reference axis, the horizontal unit vector is represented as: ;in, Represents the lateral unit vector in the unified foot reference coordinate system; matrix An orthogonal rotation matrix in a two-dimensional plane is used to rotate a vertical unit vector into a horizontal unit vector perpendicular to it.

[0044] The unified reference coordinate system for the sole of the foot is represented as follows: ;in, This indicates a unified reference coordinate system for the sole of the foot; Represents the origin of space; Indicates the lateral reference axis extending along the width of the foot; This indicates the longitudinal reference axis extending from the heel to the toe.

[0045] For any sensor mounting point or pressure sampling point within the plane of the insole Its coordinates in the unified reference coordinate system of the foot are represented as follows:

[0046] ;

[0047] in, Point The coordinates after transformation from the original planar coordinates of the insole to the unified reference coordinate system of the foot sole; Point Projection onto the transverse reference axis; Point Projection onto the longitudinal reference axis; Point The lateral coordinates of the sole of the foot; Point The longitudinal coordinates of the sole of the foot.

[0048] In cases where there is a mirror image difference between the left and right foot coordinate display directions, the right foot coordinate is aligned with the left foot display direction through the following mirror transformation:

[0049] ;

[0050] in, This represents the coordinates of the right foot after mirroring. This indicates the horizontal coordinates of the right foot position in front of the mirror image; This indicates the vertical coordinate of the right foot position in front of the mirror image.

[0051] The pressure points, flexion areas, forefoot posture sub-regions, and hindfoot posture sub-regions of the left and right feet can be expressed under a unified display direction.

[0052] S12, after establishing the unified reference coordinate system of the foot, spatial mapping is performed on the pressure array data, foot flexion data, forefoot IMU data and heel IMU data according to the coordinate system, so as to establish the spatial correspondence between pressure, flexion and posture.

[0053] For pressure array data, let the row number of the pressure array be... Column number is The valid point mapping table is as follows:

[0054] ;

[0055] in, Indicates shoe size Time Line 1 The validity marker of the column intersection point; when the intersection point is located in the effective pressure area of ​​the sole, its value is 1; when the intersection point is located in the invalid area of ​​the insole, its value is 0.

[0056] The effective pressure point set is represented as: ;in, Indicates shoe size The set of effective pressure points at that time; Indicates the row number of the pressure array; Indicates the column number of the pressure array.

[0057] According to the preset sorting rules For the set of effective pressure points Linearization is represented as:

[0058] ;

[0059] in, Indicates shoe size Linearization sorting rules for effective pressure points at time; Indicates the first The row and column coordinates corresponding to each effective pressure point; Indicates shoe size The number of effective pressure points on a single foot.

[0060] The number of effective pressure points corresponding to different shoe sizes is expressed as follows:

[0061] ;

[0062] in, Indicates shoe size The number of effective pressure points on a single foot; This indicates a size 38 insole; This indicates a size 42 insole.

[0063] No. The effective pressure point is at the first The pressure sampling count within each sampling period is represented as follows:

[0064] ;

[0065] in, Indicates the first Within the sampling period, the first Pressure sampling count at each effective pressure point; Indicates the first Row numbering of each effective pressure point; Indicates the first Column numbering of each effective pressure point; Indicates the first Within the sampling period, the first Line 1 The analog-to-digital conversion sampling result corresponding to the column intersection point, in the first... At the start of the sampling period, the main control module selects the pressure array's first sample using a row selection circuit or a multiplexer. The row conductive line is set as the current excitation row, forming a detectable resistance change path with the varistor conductive material layer and the column-direction conductive line. Subsequently, the column acquisition circuit selects the first... The first conductive line will be the first line and number The analog voltage signal generated in the column intersection area under the action of plantar pressure is sent to the ADC channel. The ADC performs analog-to-digital conversion according to the reference voltage and outputs the corresponding digital count value. This digital count value is the number of... Within the sampling period, the first Line 1 The analog-to-digital conversion sampling result corresponding to the column intersection point is denoted as .

[0066] Let the first The original planar coordinates of each effective pressure point in the insole point layout file are: Then its coordinates in the unified reference coordinate system of the foot are expressed as:

[0067] ;

[0068] in, Indicates the first The coordinates of each effective pressure point in the unified reference coordinate system of the sole of the foot; Indicates the first The location of each effective pressure point in the original plane coordinate system of the insole; Indicates the first The lateral coordinates of the sole of the foot at each effective pressure point; Indicates the first The longitudinal coordinates of the sole of the foot at each effective pressure point.

[0069] The spatial mapping relationship of pressure array data is represented as follows:

[0070] ;

[0071] in, Indicates the first Pressure space mapping results within each sampling period; Indicates the first Pressure sampling count at each effective pressure point; Indicates the first The coordinates of each effective pressure point in the unified reference coordinate system of the sole of the foot.

[0072] For plantar flexion data, each insole is equipped with two forefoot flexion sensors, denoted as Flex1 and Flex2 respectively; let the... The installation location of the road bending sensor is as follows: Then its coordinates in the unified reference coordinate system of the foot are: ;in, Indicates the first The placement of the road bending sensor in the unified reference coordinate system of the sole of the foot; Indicates the first The installation position of the road bending sensor in the original plane coordinates of the insole; Indicates the first Lateral coordinates of the foot on the road bending sensor; Indicates the first The longitudinal coordinates of the foot on the road bending sensor; Corresponding to Flex1, Corresponding to Flex2.

[0073] Let the first Road bending sensor at the first The original acquisition count within each sampling period is The data is obtained by the main control module at the insole end within a fixed sampling period, and is represented by median window de-jitter processing:

[0074] ;

[0075] in, Indicates the first Road bending sensor at the first Median filtering results within each sampling period; Indicates the first The raw data collection counts from the road bending sensor; This represents the length of the median window, which is 7 in this embodiment; This indicates the midpoint operation.

[0076] Further exponential moving average processing is performed: ;in, Indicates the first Road bending sensor at the first Smooth bend count within each sampling period; This represents the smoothing coefficient, which is set to 0.08 in this embodiment. This represents the result of median filtering; This represents the smooth bend count of the previous sampling period.

[0077] The bending progress obtained from the straightness calibration value and the bending calibration value is expressed as follows:

[0078] ;

[0079] in, Indicates the first Road bending sensor at the first The bending progress within each sampling period; This represents a preset minimum value to prevent the denominator from being zero; This means limiting the result to the range of 0 to 100; Indicates the first The straightness calibration value of the road curvature sensor; Indicates the first The bending calibration value of the road bending sensor; first, place the insole flat with the forefoot not bent, and then continuously read the value in this state. The original counts collected by the road bending sensor over several sampling periods are used, and the median of these counts is taken as the first value. Straightness calibration value of road bending sensor Then, the forefoot area of ​​the insole is bent to the standard bending posture defined in actual use, and the data is continuously read in this state. The original counts collected by the road bending sensor over several sampling periods are used, and the median of these counts is taken as the first value. Bending calibration value of road bending sensor .

[0080] The spatial mapping results of plantar flexion data are represented as follows:

[0081] ;

[0082] in, Indicates the first Spatial mapping results of plantar flexion within each sampling period; Indicates the number of smooth bends; Indicates the progress of bending; Indicates the first The placement of the road bending sensor in the unified reference coordinate system of the foot.

[0083] For data from the forefoot IMU and the heel IMU, let the installation location of the forefoot IMU be... The IMU is then installed in the following location. The spatial positions of the two in the unified reference coordinate system of the foot are respectively represented as:

[0084] ;

[0085] in, This indicates the spatial position of the forefoot IMU in the unified reference coordinate system of the foot. Indicates the mounting position of the forefoot IMU within the insole plane; , , These represent the horizontal, vertical, and lateral coordinates of the forefoot IMU, respectively. Indicates the spatial position of the heel IMU in the unified reference coordinate system of the foot. Indicates the mounting position of the heel IMU within the insole plane; , , These represent the horizontal, vertical, and axial coordinates of the IMU that follows.

[0086] Let the IMU number be represented as: ;in, Indicates the forefoot IMU; It indicates that it is followed by IMU.

[0087] The raw data for each IMU group is converted and represented as follows:

[0088] ;

[0089] ;

[0090] ;

[0091] ;

[0092] in, Indicates the first Acceleration data converted from IMU data; Indicates the first Raw acceleration counts of the IMU group; This represents the converted angular velocity data; Represents the original angular velocity count; This represents the converted Euler angle data; Represents the original Euler angle count; This represents the converted quaternion data; This represents the counting of primitive quaternions.

[0093] The magnetic field count is represented as: ;in, Indicates the first Magnetic field data from the IMU group; Indicates the first The raw magnetic field count of the IMU group.

[0094] Then the first The attitude data vector of the IMU group is represented as follows:

[0095] ;

[0096] in, Indicates the first Within the sampling period, the first The attitude data vector of the IMU group; , , , , These represent acceleration, angular velocity, Euler angles, magnetic field, and quaternion data, respectively.

[0097] The spatial mapping results of the forefoot IMU and the heel IMU are respectively expressed as follows:

[0098] ;

[0099] ;

[0100] in, Indicates the first Spatial attitude mapping results of the forefoot IMU within each sampling period; Indicates the first Spatial attitude mapping results of the IMU within each sampling period; and These represent the spatial positions of the forefoot IMU and heel IMU in the unified reference coordinate system of the foot. and These represent the attitude data vectors of the forefoot IMU and the heel IMU, respectively.

[0101] The spatial correspondence between pressure, bending, and attitude is expressed as follows:

[0102] ;

[0103] in, Indicates the first The spatial correspondence between pressure, bending, and attitude established within each sampling period; This represents the pressure space mapping result; This represents the spatial mapping result of plantar flexion; This indicates the spatial attitude mapping result of the forefoot IMU; This indicates that the IMU spatial attitude mapping result follows.

[0104] S13. After completing the spatial correspondence between pressure, flexion and attitude, the pressure array data is converted into a plantar force distribution vector, the plantar flexion data is converted into a forefoot flexion state vector, the forefoot IMU data and heel IMU data are converted into forefoot attitude vector and hindfoot attitude vector respectively, and then fused and encapsulated according to the preset frame structure order to generate a plantar multimodal attitude state frame.

[0105] The force distribution vector on the sole of the foot is represented as:

[0106] ;

[0107] in, Indicates the first Foot force distribution vector within each sampling period; Indicates the first Pressure sampling count at each effective pressure point; Indicates the first The coordinates of each effective pressure point in the unified reference coordinate system of the sole of the foot; Indicates shoe size The number of effective pressure points on a single foot.

[0108] The forefoot flexion state vector is represented as:

[0109] ;

[0110] in, Indicates the first Forefoot flexion state vector within each sampling period; and These represent the smooth bending counts of the two bending sensors, respectively. and These represent the bending progress of the two bending sensors, respectively. and These represent the positions of the two bending sensors in the unified reference coordinate system of the foot.

[0111] The forefoot posture vector is represented as: ;in, Indicates the first Forefoot posture vector within each sampling period; This indicates the spatial position of the forefoot INU in the unified reference coordinate system of the foot. This represents the attitude data vector of the forefoot IMU.

[0112] The hindfoot posture vector is represented as: ;in, Indicates the first Hind foot posture vector within each sampling period; Indicates the spatial position of the heel IMU in the unified reference coordinate system of the foot. This represents the attitude data vector following the IMU.

[0113] The aforementioned plantar force distribution vector, forefoot flexion state vector, forefoot posture vector, and hindfoot posture vector are uniformly encapsulated according to a preset frame structure, including the frame header, pressure array, first-channel flexion sensor data, second-channel flexion sensor data, battery detection data, forefoot IMU data block, heel IMU data block, and frame tail, to obtain the plantar multimodal posture state frame representation as follows:

[0114] ;

[0115] in, Indicates the first Foot plantar multimodal pose state frames within a sampling period; Indicates the frame header field; This represents the force distribution vector on the sole of the foot; This represents the forefoot flexion state vector; This field represents the battery detection count or power supply status. This represents the forefoot posture vector; This represents the hind foot posture vector; This indicates the frame end field.

[0116] During actual data transmission, the foot multimodal posture state frames are organized according to the following frame structure:

[0117] ;

[0118] in, Indicates the frame header field; Represents a pressure array; and These represent the bending acquisition counts for Flex1 and Flex2, respectively; Indicates the battery detection count; Indicates the front hand IMU data block; This indicates that it is followed by an IMU data block; This indicates the frame end field.

[0119] The frame header field is used to identify the left and right feet, and its values ​​are represented as follows:

[0120] ;

[0121] in, The frame header field representing the multimodal attitude state frame of the foot. Indicates the left foot frame header; This indicates the right foot frame header.

[0122] The frame tail field is represented as: ;in, The frame tail field represents the foot multimodal attitude state frame and is used to verify the integrity of the data frame.

[0123] The length of a single-leg data frame is represented as: ;in, Indicates shoe size Length of the single-leg data frame; Indicates the length of the frame header field; This indicates the length of the pressure array, with each pressure point occupying 2 bytes; 6 represents Flex1, Flex2, and... The total length of the three fields; Indicates the length of the front hand IMU data block; Indicates the length of the IMU data block that follows; Indicates the length of the frame tail field.

[0124] When the shoe size is 38: ;in, This indicates the length of a 38-bit single-leg data frame, in bytes.

[0125] When the shoe size is 42: ;in, This indicates the length of a 42-bit single-leg data frame, in bytes.

[0126] The multimodal attitude state frame of the foot is based on the unified reference coordinate system of the foot, and encapsulates the force distribution of the foot, the forefoot flexion state, the forefoot attitude state and the hindfoot attitude state in the same frame structure.

[0127] S2, perform frame-level continuous parsing on the multimodal attitude state frames of the foot, construct the evolution sequence of foot motion state based on the continuity of pressure change and the trend of inertial attitude change, and determine the gait state switching node in the sequence, thereby generating foot attitude sequence data with temporal consistency.

[0128] S21, after the host computer continuously receives the multimodal attitude status frames of the foot sent by the insole, it performs frame header recognition, frame tail verification and frame length judgment on each frame according to the preset frame structure order of the multimodal attitude status frames of the foot, so as to determine the left and right foot recognition results, shoe size recognition results and validity status corresponding to the frame, and forms a valid frame sequence of consecutive frames that pass the verification according to the sampling period order.

[0129] Suppose the host computer receives the first... The multimodal attitude state frame of the foot is represented as follows:

[0130] ;

[0131] in, Indicates the first Frame of foot plantar multimodal pose state; Indicates the first The frame header field of the frame; Indicates the first The pressure array in the frame; Indicates shoe size The number of effective pressure points at that time; Indicates the first The first path of the bending sensor data in the frame; Indicates the first Second-path bending sensor data in the frame; Indicates the first Battery detection data in the frame; Indicates the first Forehand IMU data in the frame; Indicates the first The frame is followed by IMU data; Indicates the first The frame end field of the frame.

[0132] For the The frame length is determined, and the shoe size recognition result is represented as follows:

[0133] ;

[0134] in, Indicates the first The shoe size recognition result corresponding to the frame; Indicates the first The frame length of the multimodal attitude state frame of the foot; This indicates an invalid shoe size identification result. When the frame length does not meet the preset frame length, the frame is judged as an abnormal frame.

[0135] Determine the number of pressure points based on shoe size recognition results:

[0136] ;

[0137] in, Indicates the first The frame corresponds to the number of effective pressure points under the shoe size; when At that time, the number of effective pressure points was 192; when At 42, the number of effective pressure points is 233.

[0138] For the Frame header recognition is performed to obtain the left and right foot recognition results:

[0139] ;

[0140] in, Indicates the first The recognition results of the left and right feet corresponding to the frame; Indicates the left foot; Indicates the right foot; Indicates the first The frame header field of the frame; This indicates an invalid left or right foot recognition result.

[0141] For the The frame end check is represented as follows:

[0142] ;

[0143] in, Indicates the first The frame end check result; Indicates the first The frame end field of the frame. This indicates the preset frame end identifier, consisting of two fixed bytes, used as the basis for determining whether the multimodal attitude state frame of the foot is complete, truncated, or misaligned; when When it indicates that the frame end check has passed; when This indicates that the frame end check failed.

[0144] By combining frame length determination, frame header identification, and frame tail verification, the first... The validity marker for a frame is represented as follows:

[0145] ;

[0146] in, Indicates the first Frame validity marker; when When, it indicates the first The frame is a valid frame; when When, it indicates the first The frame is invalid and needs to be discarded.

[0147] Arrange the continuously received valid frames in order of their sampling periods to obtain the valid frame sequence as follows:

[0148] ;

[0149] in, Indicates a valid frame sequence; Indicates the first The foot plantar multimodal posture state frames corresponding to each valid sampling time; Indicates the first The sampling period number corresponding to each valid frame; Indicates the total number of valid frames.

[0150] After obtaining the valid frame sequence, the pressure array, plantar flexion data, battery detection data, forefoot IMU data, and heel IMU data are parsed from each valid frame according to the preset frame structure order and represented as follows:

[0151] ;

[0152] in, Indicates the first The multimodal data set obtained by parsing each valid frame; Represents a pressure array; This indicates plantar flexion data; This indicates battery test data; This indicates data from the front-hand IMU. This indicates that IMU data follows.

[0153] The pressure array and plantar flexion data are represented as follows:

[0154] ;

[0155] ;

[0156] in, Indicates the first The pressure array in each valid frame; Indicates the first The effective pressure point is at the first Pressure sampling count in each valid frame; Indicates by the first The number of effective pressure points determined by the shoe size recognition results of each effective frame; Indicates the first Foot flexion data in each valid frame; and These represent the data from the first bending sensor and the second bending sensor, respectively.

[0157] S22, after obtaining the effective frame sequence, based on the changes in pressure array, pressure center, plantar flexion data, and inertial attitude changes in forefoot IMU data and heel IMU data between adjacent effective frames, a continuous frame motion association result is generated; the continuous frame motion association result is used to characterize the continuity of force changes and attitude changes between adjacent plantar multimodal attitude state frames.

[0158] The total pressure calculated from the pressure array in the valid frame is expressed as follows:

[0159] ;

[0160] in, Indicates the first The total pressure corresponding to each valid frame; Indicates the first Pressure sampling count at each effective pressure point; Indicates the first Each valid frame corresponds to the number of valid pressure points under the shoe size.

[0161] The pressure center is calculated based on the coordinates of the pressure point in the unified reference coordinate system of the sole of the foot, and is represented as follows:

[0162] ;

[0163] ;

[0164] in, Indicates the first The horizontal coordinate of the pressure center of each valid frame; Indicates the first The longitudinal coordinates of the pressure center of each valid frame; Indicates the first The lateral coordinates of each effective pressure point in the unified reference coordinate system of the sole of the foot; Indicates the first The longitudinal coordinates of each effective pressure point in the unified reference coordinate system of the sole of the foot; This represents a preset minimum value to prevent the denominator from being zero.

[0165] The change in the pressure array between adjacent valid frames is represented as follows:

[0166] ;

[0167] in, Indicates the first The first valid frame and the first The change in the pressure array between valid frames; Indicates the first A pressure array of valid frames; Indicates the first A pressure array of valid frames.

[0168] The change in total pressure between adjacent valid frames is expressed as:

[0169] ;

[0170] in, This indicates the total change in pressure between adjacent valid frames; Indicates the first Total pressure of each valid frame; Indicates the first Total pressure of each valid frame.

[0171] The change in pressure center between adjacent valid frames is expressed as:

[0172] ;

[0173] in, Indicates the first The first valid frame and the first The change in the center of pressure between valid frames; , They represent the first The horizontal and vertical coordinates of the pressure center of each valid frame; , They represent the first The horizontal and vertical coordinates of the pressure center of each valid frame.

[0174] The change in plantar flexion data between adjacent valid frames is represented as follows:

[0175] ;

[0176] in, This indicates the amount of change in plantar flexion data between adjacent valid frames; Indicates the first Foot flexion data in each valid frame; Indicates the first Foot flexion data in each valid frame.

[0177] The inertial attitude changes represented by the forefoot IMU data and the heel IMU data are obtained as follows:

[0178] ;

[0179] ;

[0180] in, Indicates the first The inertial attitude vector of the forefoot IMU in each valid frame; Indicates the first The inertial attitude vector of the IMU follows each valid frame; This represents the angular velocity data of the forefoot IMU; This represents the Euler angle data of the forefoot IMU; This represents the quaternion data of the foreground IMU; This indicates the angular velocity data following the IMU reading; This indicates that the Euler angle data is followed by the IMU. This indicates the quaternion data following the IMU.

[0181] The inertial attitude changes of the forefoot IMU and the heel IMU between adjacent valid frames are respectively expressed as:

[0182] ;

[0183] ;

[0184] in, This represents the change in inertial attitude of the forefoot IMU between adjacent valid frames; This represents the change in inertial attitude of the IMU between adjacent valid frames.

[0185] Based on all the changes, the resulting motion correlation of consecutive frames is represented as follows:

[0186] ;

[0187] in, Indicates the first The first valid frame and the first Motion correlation results between consecutive frames of valid frames; This represents the change in the pressure array; This indicates the change in total pressure. Indicates the change in the center of pressure; This indicates the change in plantar flexion data; This indicates the change in inertial attitude of the forefoot IMU; This indicates the change in inertial attitude following the IMU.

[0188] S23, combine the multimodal data set obtained from parsing each valid frame with the corresponding motion association results of consecutive frames to form a foot motion state unit:

[0189] ;

[0190] in, Indicates the first Foot motion state unit corresponding to each valid frame; Indicates the first The multimodal data set obtained by parsing each valid frame; Indicates the first The first valid frame and the first Motion correlation results between consecutive frames of valid frames.

[0191] Arranging each foot motion state unit in the order of sampling period, the resulting foot motion state evolution sequence is represented as follows:

[0192] ;

[0193] in, This represents the evolutionary sequence of foot movement states; Indicates the first Each valid frame corresponds to a foot motion state unit; Indicates the total number of valid frames.

[0194] Based on identifying gait state transition nodes, the heel pressure region, midfoot pressure region, and forefoot pressure region are first divided. The effective pressure point set is then divided according to the longitudinal coordinates in the unified plantar reference coordinate system as follows:

[0195] ;

[0196] ;

[0197] ;

[0198] in, This represents the set of effective pressure point numbers corresponding to the pressure region that follows. This represents the set of effective pressure point numbers corresponding to the midfoot pressure zone. This represents the set of effective pressure point numbers corresponding to the forefoot pressure zone. Indicates the first The longitudinal coordinates of the sole of the foot at each effective pressure point; Indicates the vertical coordinate range of the following region; Indicates the longitudinal coordinate range of the midfoot region; This indicates the range of vertical coordinates for the forefoot region.

[0199] Calculate the pressure in the heel area, midfoot area, and forefoot area separately:

[0200] ;

[0201] ;

[0202] ;

[0203] in, Indicates the first Pressure in the following region of each valid frame; Indicates the first Pressure in the midfoot region of each valid frame; Indicates the first Pressure in the forefoot region of each valid frame.

[0204] When the pressure in the heel region changes from below the ground contact threshold to above the ground contact threshold, and the heel IMU attitude change satisfies the ground contact impact change condition, the corresponding sampling period is determined as the heel ground contact node, represented as:

[0205] ;

[0206] in, Indicates the heel-ground node; The magnitude of the change in inertial attitude following the IMU; The heel contact pressure threshold is determined by continuously collecting the total pressure at pressure points in the heel area under unloaded conditions to obtain the unloaded noise range of the heel area. Then, the total pressure in the heel area is collected under normal standing conditions or with the heel lightly touching the ground to obtain the effective heel pressure range. Finally, a threshold is selected between the upper limit of the unloaded noise and the lower limit of the effective heel pressure as the heel contact pressure threshold. ; This represents the heel IMU ground contact attitude change threshold. The angular velocity, Euler angle, or quaternion changes of the heel IMU are collected under conditions of the insole being stationary and normal swinging without ground contact, obtaining the baseline of attitude change in the non-ground contact state. Then, during several normal walks by the user, the inertial attitude change of the heel IMU at the moment of heel strike is collected, obtaining the ground contact impact change range. Finally, a threshold is selected between the upper limit of the non-ground contact attitude change and the lower limit of the ground contact impact change as the heel IMU ground contact attitude change threshold. .

[0207] When the total pressure reaches the load-bearing threshold, and the pressure distribution expands from the heel area to the midfoot area or the entire sole area, the corresponding sampling period is determined as the plantar load-bearing node, represented as:

[0208] ;

[0209] in, Indicates the foot's load-bearing node; The total pressure load-bearing threshold is determined by continuously collecting the total pressure at all effective pressure points when the insole is unloaded, thus obtaining the no-load noise range. Then, the user is allowed to stand naturally with both feet or bear weight briefly on one foot, collecting the total pressure range under overall foot load. Finally, a value is selected between the upper limit of the no-load noise and the lower limit of the stable load-bearing pressure that can distinguish between no load, light contact with the ground, and the effective load-bearing state of the foot. This value is used as the total pressure load-bearing threshold. ; The midfoot weight-bearing threshold is defined by first dividing the effective pressure points into the midfoot region using a unified plantar reference coordinate system. The total pressure in the midfoot region is then collected under no-load conditions to obtain the midfoot noise baseline. Next, the midfoot pressure range is collected when the heel just touches the ground but the midfoot has not yet significantly participated in weight-bearing, as well as the midfoot pressure range when the plantar surface expands from the heel to the midfoot, entering full or near-full-foot weight-bearing. Finally, a threshold is selected between the upper limit of midfoot pressure during the heel-only contact phase and the lower limit of pressure during the midfoot's effective weight-bearing phase, serving as the midfoot region weight-bearing threshold. .

[0210] When the pressure in the forefoot region increases, the pressure center shifts along the longitudinal reference axis towards the toe, and the plantar flexion data continues to increase, the corresponding sampling period is defined as the forefoot flexion propulsion node, represented as follows:

[0211] ;

[0212] in, Indicates the forefoot bending propulsion node; This indicates the amount of change in the center of pressure along the longitudinal reference axis. This indicates the total bending change from the two bending sensors; To determine the forefoot pressure threshold, the forefoot pressure region is first divided based on a unified foot reference coordinate system and shoe size layout file. The total forefoot pressure is then collected under unloaded conditions to obtain the forefoot noise baseline. Next, the user completes several steps of normal walking, recording the forefoot pressure changes during heel strike, plantar weight-bearing, and forefoot propulsion phases. Since the forefoot flexion and propulsion phase typically shows a significant increase in forefoot pressure with the pressure center shifting towards the toes, a threshold is selected between the upper limit of forefoot pressure during full-foot weight-bearing and the lower limit of the stable high value of forefoot pressure during the forefoot propulsion phase as the forefoot pressure threshold. ; The forefoot flexion change threshold is determined by first collecting raw or smoothed counts of Flex1 and Flex2 when the insole is flat and the forefoot is not flexed, thus obtaining the forefoot flexion noise range. Then, when the user is standing and bearing weight but the forefoot is not significantly flexed, the changes in the two flexion sensors are collected to obtain the upper limit of flexion disturbance in the non-propulsion state. Subsequently, the user performs several normal walking or tiptoeing propulsion movements, and the range of the total changes in adjacent frames of the two flexion sensors during the forefoot flexion propulsion phase is collected. A threshold is selected between the upper limit of flexion disturbance in the non-propulsion state and the lower limit of flexion change in the forefoot propulsion state as the forefoot flexion change threshold. .

[0213] The total bending change of the two bending sensors is expressed as follows:

[0214] ;

[0215] in, Indicates the first The effective frame is relative to the first The total bending change of each valid frame; and They represent the first The first and second bending sensor data in each valid frame.

[0216] When the pressure in the forefoot region changes from not lower than the ground clearance threshold to lower than the ground clearance threshold, and the forefoot INU posture change meets the lift-off condition, the corresponding sampling period is determined as the toe-off node, represented as:

[0217] ;

[0218] in, Indicates the point at which the toes leave the ground; The modulus representing the change in inertial attitude of the forefoot IMU; To represent the toe-off pressure threshold, the forefoot pressure region is first divided according to the unified foot reference coordinate system and shoe size layout file. The total pressure in the forefoot region is collected under unloaded insole conditions to obtain the forefoot unloaded noise range. Then, the user completes several steps of normal walking, recording the forefoot pressure changes during the forefoot propulsion phase, the toe-off phase, and the complete toe-off phase. Since toe-off typically manifests as a rapid decrease in forefoot pressure from a high value to near the unloaded level, a threshold is selected between the upper limit of the forefoot pressure noise after complete toe-off and the lower limit of the effective forefoot pressure before toe-off, as the toe-off pressure threshold. ; The threshold for forefoot IMU lift-off posture change is determined by first collecting angular velocity, Euler angle, or quaternion changes of the forefoot IMU under conditions of insole rest, normal standing with weight-bearing, and slight forefoot disturbance without lifting off the ground. This obtains the baseline for posture change in the non-lift-off state. Then, the user completes several normal walking cycles, and the posture change of the forefoot IMU at the moment of toe lift-off is collected to obtain the range of inertial change during toe lift-off. Finally, a threshold is selected between the upper limit of posture change in the non-lift-off state and the lower limit of posture change in the toe lift-off state as the forefoot IMU lift-off posture change threshold. .

[0219] The set of gait state switching nodes is represented as follows:

[0220] ;

[0221] in, Represents the set of nodes for gait state switching; Indicates the heel-ground node; Indicates the foot's load-bearing node; Indicates the forefoot bending propulsion node; This indicates the point at which the toes leave the ground.

[0222] The plantar motion state evolution sequence not only retains the pressure array, plantar flexion data, forefoot IMU data and heel IMU data in each sampling period, but also marks the state transition process of the foot from heel strike, plantar weight-bearing, forefoot propulsion to toe lift-off in the sequence, thereby improving the temporal interpretability of plantar posture sequence data.

[0223] S24, Sequence of foot movement state evolution based on gait state switching node set. Dividing the movement into multiple stages is represented as follows:

[0224] ;

[0225] ;

[0226] ;

[0227] in, This indicates the movement phase from heel strike to weight-bearing on the sole of the foot. This indicates the movement phase between weight-bearing on the sole of the foot and forefoot flexion and propulsion. This refers to the movement phase between the forefoot bending and the toes lifting off the ground; , , , These represent the heel strike point, sole bearing point, forefoot flexion propulsion point, and toe lift-off point, respectively.

[0228] For each valid frame, a gait phase identifier is assigned based on the motion phase to which its sampling period belongs, as follows:

[0229] ;

[0230] in, Indicates the first Gait phase identifiers corresponding to each valid frame; This indicates the stage from heel strike to weight-bearing on the sole of the foot. This indicates the stage from weight-bearing on the sole of the foot to the forefoot flexion and propulsion. This indicates the stage where the forefoot bends and propels the foot until the toes leave the ground; This indicates other states that have not been classified into the above stages.

[0231] The left and right foot recognition results, shoe size recognition results, pressure array, plantar flexion data, battery detection data, forefoot IMU data, heel IMU data, and gait phase identifiers of each valid frame are merged in chronological order to generate the plantar posture sequence data as follows:

[0232] ;

[0233] in, This represents temporally consistent plantar posture sequence data; Indicates the first Each valid frame corresponds to a plantar posture recording unit; Indicates the total number of valid frames.

[0234] The plantar posture recording unit is represented as follows:

[0235] ;

[0236] in, Indicates the first Each valid frame corresponds to a plantar posture recording unit; Indicates the sampling period number or time identifier; This indicates the results of left and right foot recognition; This indicates the shoe size recognition result; Indicates gait phase markers; Represents a pressure array; This indicates plantar flexion data; This indicates battery test data; This indicates data from the front-hand IMU. This indicates that IMU data follows.

[0237] The generated plantar posture sequence data retains the pressure array, Flex1, Flex2, battery detection data, forefoot IMU data, and heel IMU data from the original plantar multimodal posture state frames. It also adds left and right foot recognition results, shoe size recognition results, and gait stage identifiers, enabling real-time display, CSV recording, CSV playback, pressure center trajectory analysis, forefoot flexion progress analysis, and forefoot and rearfoot posture difference analysis to be performed under a unified time series. Since each plantar posture recording unit comes from a valid frame sequence that has completed frame header recognition, frame tail verification, and frame length determination, it can ensure that the number of points, field order, left and right foot distribution relationship, and data temporal meaning remain consistent.

[0238] S3. Based on the foot posture sequence data, the pressure distribution characteristics, bending change characteristics and forefoot and hindfoot IMU posture characteristics are fused and calculated, and combined with the shoe size adaptive mapping rule and the left and right foot frame-level identification rule, the standardized foot posture monitoring results are output.

[0239] S31, after obtaining the foot posture sequence data, first read the shoe size recognition result and left and right foot recognition result of each foot posture recording unit in the foot posture sequence data, and call the corresponding shoe size point layout file according to the shoe size recognition result; the point layout file is used to record the number, row and column position of the effective pressure point under the corresponding shoe size and the plane coordinate in the unified reference coordinate system of the foot, so that the pressure sampling count in the pressure array can correspond one-to-one with the spatial position of the foot.

[0240] The m-th foot posture recording unit is Based on the shoe size recognition result, the corresponding shoe size's point layout file is retrieved as follows: ;in, Indicates the relationship with the first The point layout file corresponding to the shoe size recognition result of each foot posture recording unit; This indicates the shoe size recognition result; Indicates the effective pressure point number; Indicates the first The row number corresponding to each effective pressure point; Indicates the first The column number corresponding to each effective pressure point; Indicates the first The lateral coordinates of each effective pressure point in the unified reference coordinate system of the sole of the foot; Indicates the first The longitudinal coordinates of each effective pressure point in the unified reference coordinate system of the sole of the foot; This indicates the number of effective pressure points determined by the shoe size identification results.

[0241] For different shoe sizes, the number of effective pressure points is expressed as follows:

[0242] ;

[0243] in, Indicates the first The number of effective pressure points corresponding to each foot posture recording unit under the shoe size; when the shoe size recognition result is size 38, the number of effective pressure points is 192; when the shoe size recognition result is size 42, the number of effective pressure points is 233.

[0244] According to the left and right foot frame-level identification rules, a fixed frame header identifier is set in each frame of the foot multimodal posture state. The current data frame is directly determined by the frame header to determine whether it comes from the left foot or the right foot, and the foot posture sequence data is distributed to the left foot processing channel or the right foot processing channel. This is expressed as follows:

[0245] ;

[0246] in, Indicates the first The processing channel corresponding to each foot posture recording unit; Indicates the left foot processing channel; Indicates the right foot processing channel; This indicates the results of left and right foot recognition; Indicates the left foot; It indicates the right foot.

[0247] When the foot posture recording unit enters the right foot processing channel, the coordinates of the right foot position are mirrored to ensure that the left and right feet have a consistent foot orientation representation during display and fusion.

[0248] ;

[0249] in, Indicates the first The first foot posture recording unit The coordinates of each effective pressure point after processing in the left and right foot directions; Indicates the first The lateral coordinates of each effective pressure point; Indicates the first The longitudinal coordinates of each effective pressure point.

[0250] Mapping the pressure array to the corresponding point coordinates yields the shoe size adaptive pressure distribution data as follows:

[0251] ;

[0252] in, Indicates the first Adaptive pressure distribution data of shoe size corresponding to each foot posture recording unit; Indicates the first The first foot posture recording unit Pressure sampling count at each effective pressure point; Indicates the first The coordinates of each effective pressure point after processing in the left and right foot directions; This indicates the number of effective pressure points corresponding to the current shoe size.

[0253] The pressure arrays for different shoe sizes can be converted into pressure distribution data with clear spatial coordinates based on the corresponding shoe size's point layout file. The left and right foot data can also enter the corresponding processing channel according to the left and right foot frame-level identification rules, thereby avoiding mismatch of shoe size points and confusion of left and right foot data.

[0254] S32, after obtaining the shoe size adaptive pressure distribution data, calculate the pressure distribution features based on the shoe size adaptive pressure distribution data, calculate the bending change features based on the foot bending data in the foot posture sequence data, calculate the forefoot and heel IMU posture features based on the forefoot IMU data and heel IMU data in the foot posture sequence data, and generate a foot multimodal fusion feature set.

[0255] After performing zero-point subtraction, noise floor thresholding, and sensitivity multiplier correction on the pressure sampling count, the displayed pressure value is expressed as follows:

[0256] ;

[0257] in, Indicates the first The first foot posture recording unit Display pressure values ​​for each effective pressure point; Indicates the first Pressure sampling count at each effective pressure point; Indicates the first Zero-point offset value of each effective pressure point; Indicates the first Sensitivity ratio of each effective pressure point; The noise threshold is defined as follows: the insole is placed in a stable, unworn, unpressurized state. The main control module continuously collects the original pressure sampling counts of each effective pressure point according to the normal sampling cycle. The host computer statistically analyzes the unloaded sampling value of each pressure point to obtain the fluctuation range of the pressure point under no-pressure conditions. Then, the larger value among the upper limit of the unloaded fluctuation of each pressure point, the unloaded average value plus a preset multiple standard deviation, or the statistical value of the unloaded fluctuation of all effective pressure points can be taken as the noise threshold.

[0258] The total plantar pressure calculated based on the displayed pressure value is expressed as follows:

[0259] ;

[0260] in, Indicates the first Total plantar pressure corresponding to each plantar posture recording unit; Indicates the first Display pressure values ​​for each effective pressure point; This indicates the number of effective pressure points corresponding to the current shoe size.

[0261] The coordinates of the pressure center are represented as follows:

[0262] ;

[0263] ;

[0264] in, The horizontal coordinates of the standardized pressure center; The longitudinal coordinates representing the standardized center of pressure; Indicates the first The horizontal coordinates of each effective pressure point after processing in the left and right foot directions; Indicates the first The longitudinal coordinates of each effective pressure point after processing in the left and right foot directions; This represents a preset minimum value to prevent the denominator from being zero.

[0265] Based on the set of effective pressure point numbers corresponding to the heel pressure zone, midfoot pressure zone, and forefoot pressure zone, the zoned pressure is calculated as follows:

[0266] ;

[0267] ;

[0268] ;

[0269] in, Indicates pressure in the following area; This indicates pressure in the midfoot area; Indicates pressure in the forefoot area; This represents the set of effective pressure point numbers corresponding to the pressure region that follows. This represents the set of effective pressure point numbers corresponding to the midfoot pressure zone. This represents the set of effective pressure point numbers corresponding to the forefoot pressure zone. Indicates the first The displayed pressure value for each effective pressure point.

[0270] Based on the coordinates of the points, the pressure values ​​will be projected onto a two-dimensional display grid to generate a pressure cloud map, as shown below:

[0271] ;

[0272] in, Indicates the first Each foot posture recording unit is located at the grid position. Pressure cloud map value at the location; and This indicates a two-dimensional display of grid coordinates; Indicates the first A spatial kernel function that projects each effective pressure point onto a two-dimensional display grid; and Indicates the first The coordinates of each effective pressure point after processing in the left and right foot directions; The smoothing function first projects discrete pressure points onto a two-dimensional display grid of preset size based on the displayed pressure values ​​and corresponding coordinates of each effective pressure point, thus obtaining an initial pressure grid. Then, for each grid position in the initial pressure grid, the grid values ​​in its preset neighborhood are selected as smoothing inputs, and a weighted average is performed according to the principle that the closer the distance, the greater the weight, and the farther the distance, the smaller the weight, to obtain the smoothed pressure value of that grid position. The same processing is then performed on all grid positions in sequence to weaken isolated noise points and form a continuous transition between adjacent pressure regions. Finally, grid values ​​below the display threshold can be set to zero, and the processed two-dimensional grid is output as a pressure cloud map.

[0273] The pressure distribution characteristics are represented as follows:

[0274] ;

[0275] in, Indicates the first Pressure distribution characteristics corresponding to each foot posture recording unit.

[0276] For plantar flexion data, let the first... The data from the two bending sensors in the foot posture recording unit are represented as follows: ;in, This indicates plantar flexion data; This represents the data from the first bending sensor. This indicates data from the second bending sensor.

[0277] The bending progress mapping of the two bending sensor data is expressed as follows:

[0278] ;

[0279] in, Indicates the first Road bending sensor at the first Standardized bending progress in each foot posture recording unit; Indicates the first Road bending sensor data; Indicates the first The straightness calibration value of the road curvature sensor; Indicates the first The bending calibration value of the road bending sensor; This represents a preset minimum value to prevent the denominator from being zero; This means limiting the result to the range of 0 to 100; correspond , correspond .

[0280] The amount of bending change between adjacent recording units is expressed as follows:

[0281] ;

[0282] in, Indicates the first The change in bending progress between adjacent foot posture recording units by the road bending sensor; Indicates the first Standardized bending progress of each foot posture recording unit; Indicates the first Standardized bending progress of each foot posture recording unit.

[0283] The difference in bending between the two paths is expressed as follows: ;in, Indicates the first The dual-path bending difference in each foot posture recording unit is used to reflect the degree of inconsistency in the inner and outer bending of the forefoot.

[0284] The bending variation characteristics are represented as follows:

[0285] ;

[0286] in, Indicates the first The bending change characteristics corresponding to each foot posture recording unit.

[0287] For the forefoot IMU data and the heel IMU data, extract the forefoot IMU attitude vector and the heel IMU attitude vector respectively:

[0288] ;

[0289] ;

[0290] in, This represents the attitude vector of the forefoot IMU; This indicates that it is followed by the IMU attitude vector; , , These represent the roll angle, pitch angle, and yaw angle of the forefoot IMU, respectively. , , , This represents the quaternion components of the forehand IMU; , Ya These represent the roll angle, pitch angle, and yaw angle of the IMU following it, respectively. , , , This indicates that it is followed by an IMU quaternion component.

[0291] The difference in forefoot and hindfoot posture is calculated as follows: ;in, Indicates the first The posture difference between the forefoot IMU and the heel IMU in each foot posture recording unit; This represents the attitude vector of the forefoot IMU; This indicates that the IMU attitude vector follows.

[0292] The obtained forefoot and hindfoot IMU pose features are represented as follows: ;in, Indicates the first The forefoot and hindfoot IMU posture features corresponding to each plantar posture recording unit.

[0293] The pressure distribution characteristics, bending variation characteristics, and forefoot and hindfoot IMU posture characteristics are organized into a multimodal fusion feature set of the plantar surface, represented as follows:

[0294] ;

[0295] in, Indicates the first The set of multimodal fusion features of the foot plantar posture recording unit; Indicates pressure distribution characteristics; Indicates the characteristics of bending changes; Indicates the posture characteristics of the forefoot and hindfoot IMUs; Indicates gait phase markers; This indicates the results of left and right foot recognition; This indicates the shoe size recognition result.

[0296] S33, after obtaining the foot multimodal fusion feature set, perform posture fusion and abnormal degradation processing based on the foot multimodal fusion feature set, and uniformly map the pressure distribution features, bending change features and forefoot and rearfoot IMU posture features into standardized foot posture monitoring results, and display, store and play back them in the same field order as the shoe size recognition results and left and right foot frame-level identification results. First, perform magnetic field calibration on the forefoot IMU and the rearfoot IMU respectively.

[0297] For the For a group of IMUs, the magnetic field offset and scaling are expressed as follows:

[0298] ;

[0299] ;

[0300] in, , Indicates the forefoot IMU. Indicates that it is followed by IMU; Indicates the axis of the magnetic field; Indicates the first Group IMU in the Magnetic field offset on the axis; Indicates the first Group INU in The maximum value of the magnetic field collected on the axis; Indicates the first Group IMU in the The minimum magnetic field value collected on the axis; Indicates the first Group IMU in the Magnetic field scaling on the axis; Indicates the first Group IMU in the Half-amplitude value on the axis; Indicates the first The average of the half-amplitude values ​​of the three axes of the IMU group.

[0301] in:

[0302] ;

[0303] .

[0304] The calibrated magnetic field data is represented as follows: ;in, Indicates the first Group IMU in the The first foot posture recording unit Magnetic field data after shaft calibration; Indicates the first Group IMU in the Raw magnetic field data on the axis; Indicates the magnetic field offset; This indicates the scaling factor of the magnetic field.

[0305] The validity assessment of the calibrated magnetic field is expressed as follows:

[0306] ;

[0307] in, Indicates the first Group IMU in the Magnetic field validity markers in each plantar posture recording unit; Indicates the first The calibrated magnetic field vector of the IMU group; Indicates the magnitude of the calibrated magnetic field vector; Indicates the lower limit of the magnetic field modulus; Indicates the upper limit of the magnetic field modulus; Indicates the first The exponential moving average of the historical magnetic field modulus of the group IMU; This indicates the allowable deviation ratio.

[0308] The attitude fusion and anomaly degradation processing based on the magnetic field effectiveness marker are represented as follows:

[0309] ;

[0310] in, Indicates the first Group IMU in the The fused posture results from each plantar posture recording unit; Indicates the first Acceleration data from the IMU group; Indicates the first Angular velocity data of the IMU group; Indicates the first Post-calibration magnetic field data of the IMU group; Indicates the validity of the magnetic field; This represents the nine-axis attitude fusion function, first reading the first... The IMU collects acceleration, angular velocity, and calibrated magnetic field data from the current plantar attitude recording unit. The acceleration and magnetic field vectors are normalized. The angular velocity is then converted from the sensor's raw units to the units required for attitude update. The attitude quaternion from the previous sampling period is used as the initial value for the current attitude prediction. Next, the attitude quaternion is integrated based on the angular velocity to obtain a short-term attitude prediction result. The pitch and roll angles are then corrected using the acceleration direction, and the yaw angle is corrected using the calibrated magnetic field direction. The predicted attitude and correction errors are then weighted and updated according to a preset fusion gain to obtain the fusion quaternion for the current sampling period. Finally, the fusion quaternion is normalized and converted into roll, pitch, and yaw angles, which are then used as the fused attitude result output by the nine-axis attitude fusion function. This represents the six-axis attitude fusion function. First, read the first... The acceleration and angular velocity data in the current plantar attitude recording unit of the IMU group no longer use magnetic field data. Then, the acceleration vector is normalized, and the attitude quaternion of the previous sampling period is used as the initial value for the current attitude prediction. Then, the short-time attitude prediction result is obtained based on the angular velocity integral. Then, the deviation of the gravity direction from the acceleration direction is corrected, thereby correcting the roll and pitch angles. Since the six-axis attitude fusion does not introduce magnetic field information, the yaw angle is not corrected by the magnetic field. Instead, the yaw angle is continuously updated mainly by the angular velocity integral. Then, the angular velocity prediction result and the acceleration correction result are fused according to the preset fusion gain to obtain the fused quaternion of the current sampling period. Finally, the fused quaternion is normalized and converted into roll, pitch and yaw angles as the fused attitude result output by the six-axis attitude fusion function.

[0311] Based on the forefoot IMU fusion attitude results and the heel IMU fusion attitude results, the forefoot and hindfoot fusion attitude differences are represented as follows:

[0312] ;

[0313] in, This indicates the pose difference between the pose fusion results of the forefoot IMU and the pose fusion results of the heel IMU; This indicates the attitude fusion result of the forefoot IMU; This indicates the pose result after IMU fusion.

[0314] When mapping pressure distribution features, flexion variation features, forefoot and rearfoot IMU posture features, fused posture results, shoe size recognition results, and left / right foot recognition results to standardized plantar posture monitoring results, each plantar posture recording unit is processed first. Shoe size recognition results are used as the basis for selecting the number and layout of points, determining the pressure point coordinates and number of fields corresponding to shoe sizes 38, 42, or others for the current recording unit. Then, left / right foot recognition results are used as the basis for channel distribution and coordinate direction processing, ensuring that left foot data enters the left foot standard output channel and right foot data enters the right foot standard output channel. When display is required, the right foot pressure point coordinates are mirrored to ensure that both feet are represented in the same plantar direction. Finally, the total plantar pressure, heel area pressure, midfoot area pressure, and forefoot area pressure from the pressure distribution features are processed. Force and pressure center coordinates and pressure cloud map are written into the standardized results as part of the plantar force state. The two-way bending progress and the difference between the two-way bending in the bending change characteristics are written into the standardized results as part of the forefoot bending state. The forefoot IMU attitude, heel IMU attitude, and the attitude difference between the two are written into the standardized results as part of the forefoot and rearfoot partition attitude state. At the same time, the forefoot fusion attitude, heel fusion attitude, difference between forefoot and rearfoot fusion attitude, and magnetic field validity mark in the fusion attitude results are written into the standardized results to indicate whether the current attitude result is obtained by nine-axis fusion or degenerates to six-axis update when the magnetic field is abnormal. Finally, the gait stage identifier, battery detection data, shoe size recognition result, and left and right foot recognition result are written into the standardized plantar attitude monitoring results as auxiliary state fields to form a standardized plantar attitude monitoring result with a unified field order.

[0315] Finally, the continuous standardized plantar posture monitoring results are output in chronological order as a standardized plantar posture monitoring result sequence, as follows: ;in, This represents a sequence of standardized plantar posture monitoring results; Indicates the first Standardized plantar posture monitoring results corresponding to each plantar posture recording unit; This indicates the total number of foot posture recording units.

[0316] The standardized foot posture monitoring result sequence is displayed, stored, and replayed according to the field order consistent with the shoe size recognition result and the left and right foot frame-level identification result. During display, the pressure cloud map, pressure center trajectory, forefoot flexion progress, forefoot posture, heel posture, forefoot and rearfoot posture difference, and battery status are output. During storage, the Time, pressure point field, Flex field, Bat field, forefoot IMU field, heel IMU field, fused posture field, and MagValid field are written to a CSV file in a fixed order. During playback, the shoe size is automatically identified based on the number of pressure points in the CSV header, and the corresponding point layout file is loaded to ensure that the real-time display, storage, and playback are consistent in terms of the number of pressure points, field order, left and right foot distribution relationship, and posture meaning.

[0317] Example 2

[0318] like Figure 7 As shown, a foot posture monitoring system based on multimodal data fusion includes the following modules:

[0319] A multimodal acquisition module for the sole of the foot is used to acquire pressure array data, plantar flexion data, forefoot IMU data, heel IMU data and battery detection data of a single foot.

[0320] The state frame encapsulation module is used to encapsulate the pressure array data, plantar flexion data, forefoot IMU data and heel IMU data into a plantar multimodal attitude state frame based on a unified plantar reference coordinate system.

[0321] The frame-level parsing and temporal reconstruction module is used to perform frame header recognition, frame tail verification, and frame length determination on the continuously received multimodal foot posture state frames, determine the left and right foot recognition results, shoe size recognition results, and valid frame sequence, and construct a foot motion state evolution sequence based on the continuity of pressure change, pressure center change, foot flexion change, and inertial attitude change of forefoot IMU and heel IMU between adjacent valid frames, identify the heel strike node, foot weight-bearing node, forefoot flexion propulsion node, and toe lift-off node, and generate foot posture sequence data with temporal consistency.

[0322] The multimodal fusion output module is used to generate shoe size adaptive pressure distribution data based on the foot posture sequence data, calculate pressure distribution characteristics, bending change characteristics and forefoot and hindfoot IMU posture characteristics, perform posture fusion and abnormal degradation processing, and output standardized foot posture monitoring results.

[0323] This invention encompasses any substitutions, modifications, equivalent methods, and solutions made within the spirit and scope of this invention. To provide the public with a thorough understanding of this invention, specific details are described in detail in the following preferred embodiments; however, those skilled in the art will fully understand the invention even without these details. Furthermore, to avoid unnecessary misunderstanding of the essence of this invention, well-known methods, processes, procedures, components, and circuits are not described in detail.

[0324] The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A method for monitoring plantar posture based on multimodal data fusion, characterized in that, Includes the following steps: The pressure array data, plantar flexion data, forefoot IMU data and heel IMU data of a single foot are uniformly packaged according to a preset spatial correspondence. Obtain multimodal attitude state frames of the foot, which characterize the force distribution and posture state of the front and rear sections of the foot, wherein the state frames use the foot region as a unified reference. Based on the multimodal attitude state frames of the foot, frame-level continuous analysis is performed, and the evolution sequence of foot motion state is constructed according to the continuity of pressure change and the trend of inertial attitude change. Gait state switching nodes are identified in this sequence, thereby generating temporally consistent plantar posture sequence data; Based on the plantar posture sequence data, pressure distribution characteristics, bending change characteristics, and forefoot and hindfoot IMU posture characteristics are fused and calculated. By combining the shoe size adaptive mapping rule and the left and right foot frame-level identification rule, standardized foot posture monitoring results are output.

2. The foot posture monitoring method based on multimodal data fusion according to claim 1, characterized in that, Based on the foot structure of the insole, a pre-defined anatomical reference datum for the foot is established. The insole plane is defined as a unified spatial coordinate domain, with the geometric center of the heel as the spatial origin. A longitudinal reference axis is defined along the toe direction, and a transverse reference axis is defined along the foot width direction, thus forming a unified foot reference datum coordinate system represented as follows: ;in, This indicates a unified reference coordinate system for the sole of the foot; Represents the origin of space; Indicates the lateral reference axis extending along the width of the foot; This indicates the longitudinal reference axis extending from the heel to the toe.

3. The foot posture monitoring method based on multimodal data fusion according to claim 2, characterized in that, Based on the aforementioned unified foot reference coordinate system, each pressure sampling point in the pressure array data is mapped to its corresponding foot plane coordinate position. Simultaneously, the foot flexion sensor placement is mapped to the forefoot local flexion region, the forefoot IMU data is mapped to the forefoot posture sub-region, and the heel IMU data is mapped to the hindfoot posture sub-region. This establishes the spatial correspondence between pressure, flexion, and posture as follows: ； in, Indicates the first The spatial correspondence between pressure, bending, and attitude established within each sampling period; This represents the pressure space mapping result; This represents the spatial mapping result of plantar flexion; This indicates the spatial attitude mapping result of the forefoot IMU; This indicates that the IMU spatial attitude mapping result follows.

4. The foot posture monitoring method based on multimodal data fusion according to claim 3, characterized in that, After completing the spatial correspondence mapping, the pressure array data is converted into a plantar force distribution vector, the plantar flexion data is converted into a forefoot flexion state vector, and the forefoot IMU data and heel IMU data are converted into forefoot attitude vectors and hindfoot attitude vectors, respectively. These four types of vectors are then fused and encapsulated according to a preset frame structure order to generate a multimodal plantar attitude state frame representation that characterizes the plantar force distribution and the forefoot and anterior / posterior partition attitude states. ； in, Indicates the first Foot plantar multimodal pose state frames within a sampling period; Indicates the frame header field; This represents the force distribution vector on the sole of the foot; This represents the forefoot flexion state vector; Indicates the battery detection count; This represents the forefoot posture vector; This represents the hindfoot posture vector; This indicates the frame tail field, where each modal data is expressed in a coordinate-consistent manner based on the foot unified reference datum.

5. The foot posture monitoring method based on multimodal data fusion according to claim 1, characterized in that, According to the preset frame structure order of the foot multimodal posture state frames, the continuously received foot multimodal posture state frames are subjected to frame header recognition, frame tail verification, and frame length determination to determine the corresponding left and right foot recognition results, shoe size recognition results, and valid frame sequences. The pressure array, foot flexion data, battery detection data, forefoot IMU data, and heel IMU data are parsed from the valid frame sequences and represented as follows: ； in, Indicates the first The multimodal data set obtained by parsing each valid frame; Represents a pressure array; This indicates plantar flexion data; This indicates battery test data; This indicates data from the front-hand IMU. This indicates that IMU data follows.

6. The foot posture monitoring method based on multimodal data fusion according to claim 5, characterized in that, Based on the changes in pressure array, pressure center, plantar flexion data, and inertial attitude changes in forefoot IMU and heel IMU data in adjacent frames of the effective frame sequence, the continuous frame motion correlation result is generated as follows: ； in, Indicates the first The first valid frame and the first Motion correlation results between consecutive frames of valid frames; This represents the change in the pressure array; This indicates the change in total pressure. Indicates the change in the center of pressure; This indicates the change in plantar flexion data; This indicates the change in inertial attitude of the forefoot IMU; This indicates the change in inertial attitude following the IMU.

7. The foot posture monitoring method based on multimodal data fusion according to claim 6, characterized in that, Based on the continuous frame motion association results, a foot motion state evolution sequence is constructed according to the sampling period order. Within this sequence, the heel strike node, foot weight-bearing node, forefoot flexion propulsion node, and toe-off node are identified, resulting in the gait state switching node set represented as follows: ； in, Represents the set of nodes for gait state switching; Indicates the heel-ground node; Indicates the foot's load-bearing node; Indicates the forefoot bending propulsion node; Indicates the point at which the toes leave the ground; Based on the gait state switching node set, the pressure array, plantar flexion data, forefoot IMU data, and heel IMU data in the foot motion state evolution sequence are temporally merged to generate a temporally consistent foot posture sequence data representation as follows: ;in, This represents temporally consistent plantar posture sequence data; Indicates the first Each valid frame corresponds to a plantar posture recording unit; Indicates the total number of valid frames.

8. The foot posture monitoring method based on multimodal data fusion according to claim 1, characterized in that, Based on the shoe size recognition result in the foot posture sequence data, the corresponding shoe size point layout file is called, and the foot posture sequence data is distributed to the left foot processing channel or the right foot processing channel according to the left and right foot frame-level identification rules. The pressure array in the foot posture sequence data is mapped to point coordinates to generate shoe size adaptive pressure distribution data. The pressure distribution features are calculated based on the shoe size adaptive pressure distribution data, and the bending change features are calculated based on the foot bending data in the foot posture sequence data. The forefoot and heel IMU posture features are calculated based on the forefoot IMU data and heel IMU data in the foot posture sequence data, generating a multimodal fusion feature set of the foot.

9. The foot posture monitoring method based on multimodal data fusion according to claim 8, characterized in that, Based on the foot multimodal fusion feature set, posture fusion and abnormal degradation processing are performed. The pressure distribution features, bending change features and forefoot and hindfoot IMU posture features are uniformly mapped into standardized foot posture monitoring results, and displayed, stored and played back according to the field order consistent with the shoe size recognition results and the left and right foot frame-level identification results.

10. A foot posture monitoring system based on multimodal data fusion, used to implement the foot posture monitoring method based on multimodal data fusion as described in any one of claims 1-9, characterized in that, Includes the following modules: A multimodal acquisition module for the sole of the foot is used to acquire pressure array data, plantar flexion data, forefoot IMU data, heel IMU data and battery detection data of a single foot. The state frame encapsulation module is used to encapsulate the pressure array data, plantar flexion data, forefoot IMU data and heel IMU data into a plantar multimodal attitude state frame based on a unified plantar reference coordinate system. The frame-level parsing and temporal reconstruction module is used to perform frame header recognition, frame tail verification, and frame length determination on the continuously received multimodal foot posture state frames, determine the left and right foot recognition results, shoe size recognition results, and valid frame sequence, and construct a foot motion state evolution sequence based on the continuity of pressure change, pressure center change, foot flexion change, and inertial attitude change of forefoot IMU and heel IMU between adjacent valid frames, identify the heel strike node, foot weight-bearing node, forefoot flexion propulsion node, and toe lift-off node, and generate foot posture sequence data with temporal consistency. The multimodal fusion output module is used to generate shoe size adaptive pressure distribution data based on the foot posture sequence data, calculate pressure distribution characteristics, bending change characteristics and forefoot and hindfoot IMU posture characteristics, perform posture fusion and abnormal degradation processing, and output standardized foot posture monitoring results.

Images