Exercise training analysis method for multi-modal sensing data joint modeling

Abstract

The invention discloses an exercise training analysis method for multi-modal sensing data joint modeling, and relates to the technical field of biomechanical measurement. Comprising the following steps: S1, setting weights of IMU data, sEMG signal data and pressure data according to an angular velocity, and carrying out data fusion; s2, according to the fused three-dimensional vector, setting a Transform model, determining an attention weight, and according to the attention weight, performing feature reconstruction; and S3, according to the fusion features, the sampling frequency and the basic rigidity are adjusted. According to the method, abnormal data points and thermal expansion compensation are removed through the calibration plate, the rotation matrix and the translation vector are determined in combination with singular value decomposition, coordinate system unification of multiple sensors is achieved, meanwhile, the corresponding attention weight is obtained by querying the key matrix, the weight of key motion features can be strengthened, and the accuracy and the reliability of the system are improved. And the IMU weight can be adaptively adjusted according to different motion modes, so that the pertinence of feature fusion is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of biomechanical measurement, in particular to a motion training analysis method based on multi-modal sensor data joint modeling. BACKGROUND

[0002] With the development of sports science and rehabilitation medicine, precise motion training analysis technology has become a key support for improving athletes' competitive level and ensuring the safety of rehabilitation training. The current mainstream motion analysis system is mainly based on the following three technical routes:

[0003] (1) Mechanical measuring device: such as the Chinese utility model patent with publication number CN222622959U, which discloses a scoliosis measuring device, which realizes static posture measurement through mechanical structures such as adjusting frame and angle ruler.

[0004] (2) Single sensor system: using inertial measurement unit (IMU) or surface electromyography (sEMG) to collect motion data.

[0005] (3) Optical motion capture: three-dimensional optical tracking system based on marker points.

[0006] However, the mechanical measuring device currently used requires the measured person to remain completely still in order to obtain effective measurement results. For example, in the Chinese utility model patent with publication number CN222622959U, the hanging cone 605 needs to be stably pointed to the center position of the first angle ruler 601 to achieve a measurable state, which will result in about 78% of the dynamic motion data being unable to be effectively collected in the actual motion training scene. When the subject performs a scoliosis motion with an amplitude exceeding 15°, the hanging cone 605 will swing violently, resulting in a reading error of the first pointer 606 up to ±7°.

[0007] At the same time, due to the lack of dynamic parameter optimization mechanism based on real-time training data in the mechanical measuring device currently used, the key adjustment parameters (such as spring pre-tightening force in scoliosis measurement) are kept fixed, and cannot be adaptively adjusted according to the immediate motion state of the measured person. That is, not only will there be systematic deviation between the motion evaluation results and the individual's true physiological state, but also specific parameter optimization schemes for different trainers cannot be provided, thus fundamentally restricting the scientificity and effectiveness of training guidance. SUMMARY

[0008] The purpose of the present application is to provide a motion training analysis method based on multi-modal sensor data joint modeling to solve the problems raised in the background art.

[0009] To achieve the above object, the application provides the following technical scheme: a multi-modal sensor data joint modeling sports training analysis method, comprising:

[0010] S1: setting dynamic-static measurement mechanism: setting the weight size of IMU data, sEMG signal data and pressure data according to the angular velocity size, and performing data fusion according to the weight size;

[0011] S2: multi-modal space-time alignment: setting the Transformer model according to the fusion three-dimensional vector, determining the attention weight, and performing feature reconstruction according to the attention weight, comprising:

[0012] S2.1: data synchronization processing: performing synchronization alignment processing according to the time stamp and spatial coordinate system of the sensor, obtaining the aligned time stamp and spatial coordinate system;

[0013] S2.2: feature level fusion: constructing the fusion three-dimensional vector according to the aligned feature vector, and setting the query matrix, key matrix and value matrix according to the fusion three-dimensional vector, setting the attention weight matrix according to the query matrix and key matrix, and determining the reconstruction feature matrix according to the attention weight matrix and value matrix;

[0014] S2.3: state adjustment: adjusting the weight coefficient according to the motion mode of the patient, obtaining the adjusted weight coefficient;

[0015] S2.4: determining the fusion parameter: determining the fusion feature according to the adjusted weight coefficient and reconstruction feature matrix, specifically:

[0016]

[0017] Wherein: Z final is the fusion feature, is the adjusted IMU weight coefficient, Z IMU is the IMU modal feature segment, Z mech is the mechanical modal feature segment;

[0018] S3: parameter optimization control: adjusting the sampling frequency and basic stiffness according to the fusion feature.

[0019] Further, the data fusion comprises:

[0020] S1.1: multi-modal data preprocessing: obtaining IMU data, sEMG signal data and pressure data through the set inertial measurement unit, surface electromyography sensor and flexible pressure sensor array;

[0021] S1.2: determining a motion state: determining a motion mode of the patient according to the angular velocity size;

[0022] S1.3: data fusion: setting a weight coefficient according to the motion mode, and obtaining fused data through a Kalman filtering algorithm, specifically:

[0023] X fused = ω IMU × X IMU + ω mech × X mech

[0024] X fused is a primary fusion three-dimensional vector, X mech is joint angle data, X IMU is preprocessed IMU data, ω IMU is an IMU weight coefficient, and ω mech is a mechanical weight coefficient.

[0025] Further, determining the motion mode of the patient includes:

[0026] S1.2.1: determining an angular velocity: setting an anatomical coordinate system according to the constructed spatial sensor array, and obtaining an X-axis angular velocity component, a Y-axis angular velocity component, and a Z-axis angular velocity component according to the anatomical coordinate system to determine the angular velocity size;

[0027] S1.2.2: determining a motion state: obtaining an output voltage according to the angular velocity size, comparing the output voltage signal with a preset voltage threshold range, and determining the motion state of the patient according to the comparison result, specifically:

[0028] When the output voltage signal is greater than an upper threshold of the preset voltage threshold range, the motion state of the patient is a dynamic mode; when the output voltage signal is within the preset voltage threshold range, the motion state of the patient is a mixed mode; and when the output voltage signal is less than a lower threshold of the preset voltage threshold range, the motion state of the patient is a static mode.

[0029] Further, the upper threshold of the preset voltage threshold range is set according to a temperature difference between a current environment temperature and a reference temperature and a reference threshold voltage, specifically:

[0030]

[0031] wherein: is the upper threshold of the preset voltage threshold range, is the reference threshold voltage, k T is a temperature coefficient, T act is the current environment temperature, and Tthe is a reference temperature;

[0032] According to the resistance voltage dividing network, the lower threshold of the preset voltage threshold range is set, specifically:

[0033]

[0034] Wherein: is the lower threshold of the preset voltage threshold range, is the reference threshold voltage, R2 is the lower arm resistance in the resistance voltage dividing network, and R1 is the upper arm resistance in the resistance voltage dividing network.

[0035] Further, the fused data is obtained, including:

[0036] S1.3.1: determining the weight coefficient: according to the angular velocity, the IMU weight coefficient and the mechanical weight coefficient are set, specifically:

[0037]

[0038] Wherein: ω IMU is the IMU weight coefficient, ω mech is the mechanical weight coefficient, e is the natural logarithm base, and ||ω‖ is the combined magnitude of the three-dimensional space angular velocity;

[0039] S1.3.2: determining the fused three-dimensional vector: according to the IMU weight coefficient and the mechanical weight coefficient, the IMU data, the sEMG signal data and the pressure data are fused to obtain a primary fused three-dimensional vector, and according to the motion joint angle data and the IMU data, a data residual is obtained, and the data residual is compared with a preset residual threshold, according to the comparison result, the final fused three-dimensional vector is determined, specifically:

[0040] When the data residual is less than the preset residual threshold, the primary fused three-dimensional vector is the final fused three-dimensional vector; otherwise, the primary fused three-dimensional vector is corrected by Kalman filtering algorithm to obtain a corrected fused three-dimensional vector, and the corrected fused three-dimensional vector is the final fused three-dimensional vector.

[0041] Further, the primary fused three-dimensional vector is corrected by Kalman filtering algorithm, including:

[0042] S1.3.2.1: determining the predicted state vector: according to the sampling time interval, the state transition matrix is set, and according to the state transition matrix, the predicted state vector at the current time is determined;

[0043] S1.3.2.2: determining the observation vector: according to the primary fused three-dimensional vector, the observation vector is constructed;

[0044] S1.3.2.3: determining a gain matrix: constructing a prediction covariance matrix according to joint angles, angular velocities and angular accelerations, and determining a gain matrix;

[0045] S1.3.2.4: determining a corrected fused three-dimensional vector: correcting the primary fused three-dimensional vector according to the prediction state vector, the gain matrix and the observation vector, and obtaining a corrected fused three-dimensional vector, specifically:

[0046]

[0047] wherein X final is the corrected fused three-dimensional vector, is a prediction state vector at a current time r, K r is the gain matrix, and Z r is an observation vector, and H is an observation matrix.

[0048] Further, an aligned time stamp and a spatial coordinate system are obtained, including:

[0049] S2.1.1: time synchronization processing: correcting a time stamp of a sensor through clock offset compensation, and obtaining a corrected time stamp, specifically:

[0050] t cor = t raw + (t mas - t sla ) - t p-d

[0051] wherein t cor is the corrected time stamp, t raw is an original time stamp, t mas is a PTP master clock reference time, t sla is a node local clock reading, and t p-d is a signal transmission physical delay time.

[0052] S2.1.2: data rejection compensation: determining an abnormal data point through a set calibration plate, rejecting the abnormal data point, and simultaneously performing temperature compensation on the remaining data points according to a constructed temperature field;

[0053] S2.1.3: determining a spatial conversion parameter: obtaining a centralized coordinate according to a temperature-compensated spatial coordinate and a single-point measurement spatial coordinate, constructing a covariance matrix according to the centralized coordinate, simultaneously singular value decomposing the covariance matrix, obtaining a left singular vector, a singular value matrix and a right singular vector, and determining a translation vector, specifically:

[0054] Ttra = mean(p i )-R rot · mean(P i cor )

[0055] where: T tra is a translation vector, mean(p i ) is a global coordinate mean value, mean(P i cor ) is a sensor coordinate mean value, R rot is a rotation matrix.

[0056] Further, the remaining data points are temperature compensated, including:

[0057] S2.1.2.1: Abnormal data point rejection: Obtain the mean value of each coordinate component of each data point through the calibration plate, determine the three-dimensional space standard deviation, set the abnormal judgment threshold according to the three-dimensional space standard deviation, and compare the Euclidean norm between the single-point measurement space coordinates and the coordinate component mean value with the abnormal judgment threshold, according to the comparison result, the data point corresponding to the single-point measurement space coordinates is rejected, specifically:

[0058] When the Euclidean norm is greater than the abnormal judgment threshold, the data point corresponding to the single-point measurement space coordinates is rejected; otherwise, the data point corresponding to the single-point measurement space coordinates is retained;

[0059] S2.1.2.2: Temperature compensation: According to the material thermal expansion coefficient of the calibration plate and the current environment temperature, set the relative thermal expansion rate, and obtain the space coordinates after temperature compensation according to the relative thermal expansion rate, specifically:

[0060]

[0061] where: P i cor is the space coordinates after temperature compensation, p i is the single-point measurement space coordinates, and A is the relative thermal expansion rate.

[0062] Further, when the motion mode of the patient is a dynamic mode, the initial IMU weight coefficient is adjusted according to the maximum setting value of the angular velocity size and the IMU weight coefficient, to determine the adjusted IMU weight coefficient, specifically:

[0063]

[0064] wherein: is the adjusted IMU weight coefficient in the dynamic mode, ||ω|| is the combined value of the three-dimensional space angular velocity, and ω IMU is the IMU weight coefficient. is the angular velocity corresponding to the upper threshold of the preset voltage threshold range.

[0065] When the motion mode of the patient is the static mode, the initial IMU weight coefficient is adjusted according to the angular velocity, and the adjusted IMU weight coefficient is determined, specifically as follows:

[0066]

[0067] wherein: is the adjusted IMU weight coefficient in the dynamic mode, ||ω|| is the combined value of the three-dimensional space angular velocity, and ω IMU is the IMU weight coefficient.

[0068] Further, the sampling frequency and the basic stiffness are adjusted, including:

[0069] S3.1: Stiffness adjustment: according to the patient's weight, the maximum allowed stiffness is set, and the initial stiffness is dynamically adjusted according to the initial stiffness, sampling interval and sampling acceleration, specifically as follows:

[0070] S = S0 + k s · (Σa j × Δt)

[0071] wherein: S is the adjusted stiffness, S0 is the initial stiffness, k s is the acceleration integral coefficient, a j is the jth sampling point acceleration, and Δt is the sampling interval.

[0072] S3.2: Sampling frequency adjustment: according to the motion mode of the patient, the sampling frequency is set, specifically as follows:

[0073] When the motion mode of the patient is the static mode, the fixed sampling frequency is set;

[0074] When the motion mode of the patient is the mixed mode, the fixed sampling frequency is adjusted according to the angular velocity, specifically as follows:

[0075] f adj = f0 + k f · ||ω||

[0076] wherein: f adj is the adjusted sampling frequency, f0 is the basic sampling frequency, k f is the adjustment coefficient, and ||ω|| is the combined value of the three-dimensional space angular velocity.

[0077] When the motion mode of the patient is a dynamic mode, then according to the maximum sampling frequency and the adjusted sampling frequency, the sampling frequency in the dynamic mode is determined, specifically:

[0078] f = min (f max ,f adj )

[0079] Wherein: f is the sampling frequency in the dynamic mode, f max is the maximum sampling frequency, and f adj is the adjusted sampling frequency.

[0080] Compared with the prior art, the beneficial effects of the present application are:

[0081] Firstly, the present application eliminates abnormal data points and thermal expansion compensation through the calibration plate, and determines the rotation matrix and translation vector by combining singular value decomposition, so that the coordinate system of the multi-sensor is unified, and the corresponding attention weight is obtained through the query-key matrix, which not only can strengthen the weight of the key motion feature, but also can adaptively adjust the IMU weight according to different motion modes, thereby improving the pertinence of feature fusion.

[0082] Secondly, the present application distinguishes three different motion modes of static, mixed and dynamic through the comparison between the angular velocity threshold and the voltage of temperature compensation, and simultaneously adapts to the three different motion modes of static, mixed and dynamic in real time through the dynamic fusion of multi-modal sensors, so as to not only improve the data acquisition range, but also ensure the effectiveness of the data in different motion states.

[0083] Thirdly, the present application sets the maximum stiffness according to the patient's weight, and dynamically adjusts the stiffness size through acceleration integration, so as to avoid excessive constraint or insufficient support, and simultaneously in the dynamic mode, the sampling frequency can be linearly adjusted with the angular velocity, thereby effectively balancing the data amount. BRIEF DESCRIPTION OF DRAWINGS

[0084] Figure 1 It is a flowchart of the motion training analysis method in the present application. DETAILED DESCRIPTION

[0085] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application.

[0086] Reference Figure 1The embodiment provides a sports training analysis method based on multi-modal sensor data joint modeling.

[0087] Step S1: setting a dynamic-static measurement mechanism. That is, according to the set sensor array, IMU data, sEMG signal data and pressure data are obtained, and according to the comparison of angular velocity, the weight size of the IMU data, sEMG signal data and pressure data is obtained, and according to the determined weight size, data fusion is performed, and the specific steps are as follows:

[0088] Step S1.1: multi-modal data preprocessing. That is, at the position of the key anatomical landmarks of the human body (such as C7 vertebrae, sacrum, etc.), an inertial measurement unit, that is, an industrial 6-axis inertial measurement unit, is fixedly arranged through a medical-grade adhesive material. At the same time, according to the target muscle direction (such as the erector spinae muscle, the rectus abdominis muscle and other key muscle groups), a surface electromyography sensor, that is, an 8-channel double-differential electrode, is attached through a conductive gel. Further, a flexible pressure sensor array is arranged through a capacitive pressure sensing film, and a silica gel protective layer is arranged on the contact surface.

[0089] In the embodiment, the IMU data obtained by the inertial measurement unit is transmitted through Bluetooth, the sEMG signal obtained by the surface electromyography sensor is transmitted through a wire (such as a shielded twisted pair), and the pressure data obtained by the flexible pressure sensor array is transmitted through a CAN bus, that is, a distributed acquisition architecture is constructed.

[0090] Further, the IMU data is denoised through a fourth-order Butterworth low-pass filter to align the coordinate system of the denoised IMU data with the human anatomical coordinate system. At the same time, after the sEMG signal is removed from the baseline drift through a band-pass filter, the corresponding envelope is obtained through Hilbert transformation. At the same time, the pixel-pressure mapping relationship is determined according to the checkerboard calibration method, and spatial interpolation processing is performed through a cubic interpolation algorithm.

[0091] Step S1.2: determining the motion state. That is, according to the set spatial sensor array, the corresponding angular velocity size is obtained, and the corresponding motion mode is determined according to the obtained angular velocity size.

[0092] Specifically:

[0093] Step S1.2.1: determining the angular velocity. That is, TDKICM-42688 modules are arranged at the L4 lumbar spine, the T12 vertebrae and the sacrum, respectively, to construct a spatial sensor array. Specifically, the TDKICM-42688 module arranged at the L4 lumbar spine is set as a main sensor, and the TDKICM-42688 modules arranged at the T12 vertebrae and the sacrum are set as auxiliary sensors.

[0094] In the embodiment, the main sensor is set as the origin, the horizontal left direction is set as the X-axis, the horizontal forward direction is set as the Y-axis, and the vertical upward direction is set as the Z-axis. Further, the X-Z plane formed by the X-axis and the Z-axis is set as the coronal plane, the Y-Z plane formed by the Y-axis and the Z-axis is set as the sagittal plane, and the X-Y plane formed by the X-axis and the Y-axis is set as the transverse plane, that is, according to the set coronal plane, sagittal plane and transverse plane, the anatomical coordinate system is set.

[0095] Further, according to the set anatomical coordinate system, the angular velocity components corresponding to each plane in the anatomical coordinate system are obtained, that is, the X-axis angular velocity component, the Y-axis angular velocity component and the Z-axis angular velocity component. Specifically, the X-axis angular velocity component is obtained by the TDKICM-42688 module set at the T12 vertebral body. The Y-axis angular velocity component is obtained by the TDKICM-42688 module set at the L4 lumbar spine. The Z-axis angular velocity component is obtained by the TDKICM-42688 module set at the sacral center.

[0096] Further, according to the obtained X-axis angular velocity component, Y-axis angular velocity component and Z-axis angular velocity component, the corresponding angular velocity magnitude is determined, specifically:

[0097]

[0098] Wherein: ||ω|| is the combined magnitude of the three-dimensional space angular velocity, ω x is the X-axis angular velocity component, ω y is the Y-axis angular velocity component, and ω z is the Z-axis angular velocity component.

[0099] Step S1.2.2: Determine the motion state. That is, according to the angular velocity magnitude determined in step S1.2.1, the corresponding output voltage is determined, specifically:

[0100] V out = k ω-V × ||ω z ||

[0101] Wherein: V out is the output voltage signal, k ω-V is the conversion coefficient, and ||ω|| is the combined magnitude of the three-dimensional space angular velocity.

[0102] Further, the determined output voltage signal is compared with the preset voltage threshold range, and according to the comparison result, the current state of the patient is determined, specifically:

[0103] When the determined output voltage signal is greater than the upper threshold of the preset voltage threshold range, the current state of the corresponding patient is a dynamic mode. When the determined output voltage signal is within the preset voltage threshold range, the current state of the corresponding patient is a mixed mode. When the determined output voltage signal is less than the lower threshold of the preset voltage threshold range, the current state of the corresponding patient is a static mode.

[0104] In the embodiment, the upper threshold of the preset voltage threshold range is set according to the temperature difference between the current environment temperature and the reference temperature, and the set reference threshold voltage, specifically as follows:

[0105]

[0106] Wherein: is the upper threshold of the preset voltage threshold range, is the reference threshold voltage, k T is the temperature coefficient, T act is the current environment temperature, T the is the reference temperature.

[0107] Further, the sensor array set in step S1.1 is connected to the signal conditioning circuit and the resistance voltage dividing network (the signal conditioning circuit and the resistance voltage dividing network in the embodiment are both conventional circuit designs, so they are not specifically described in the embodiment). Specifically, according to the set resistance voltage dividing network, the lower threshold of the preset voltage threshold range is set, specifically as follows:

[0108]

[0109] Wherein: is the lower threshold of the preset voltage threshold range, is the reference threshold voltage, R2 is the lower arm resistance in the resistance voltage dividing network, and R1 is the upper arm resistance in the resistance voltage dividing network.

[0110] Step S1.3: Data fusion. That is, according to the current state of the patient determined in step S1.2.2, the IMU weight coefficient corresponding to the IMU data, the mechanical weight coefficient corresponding to the sEMG signal data and the pressure data are determined, and the fused data are obtained through the Kalman filtering algorithm and the determined IMU weight coefficient and mechanical weight coefficient. Specifically as follows:

[0111] Step S1.3.1: Determine the weight coefficient. That is, according to the angular velocity determined in step S1.2.2, the IMU weight coefficient and the mechanical weight coefficient are set, specifically as follows:

[0112]

[0113] wherein: ω IMU is the IMU weight coefficient, ω mech is the mechanical weight coefficient, e is the natural logarithm base, and ||ω|| is the resultant value of the three-dimensional angular velocity.

[0114] It is worth noting that in the process of determining the IMU weight coefficient and the mechanical weight coefficient according to the angular velocity, when the angular velocity determined in step S1.2.2 is greater than the preset angular velocity threshold, the IMU weight coefficient and the mechanical weight coefficient are re-set through a nonlinear compensation algorithm, specifically:

[0115]

[0116] wherein: ω IMU-sat is the nonlinearly compensated IMU weight coefficient, ω mech-sat is the nonlinearly compensated mechanical weight coefficient, ω base is the basic angular velocity, k ω is the adjustment coefficient, ω the is the preset angular velocity threshold, and e is the natural logarithm base. ||ω|| is the resultant value of the three-dimensional angular velocity.

[0117] Step S1.3.2: Determine the fused three-dimensional vector. That is, according to the IMU weight coefficient and the mechanical weight coefficient determined in step S1.3.1, the preprocessed IMU data obtained in step S1.1, and the motion joint angle data determined through the sEMG signal and the pressure data, data fusion is performed to obtain a primary fused three-dimensional vector, specifically:

[0118] X fused = ω IMU × X IMU + ω mech × X mech

[0119] wherein: X fused is the primary fused three-dimensional vector, X mech is the motion joint angle data, X IMU is the preprocessed IMU data, ω IMU is the IMU weight coefficient, and ω mech is the mechanical weight coefficient.

[0120] Further, in the present embodiment, the data residual size between the motion joint angle data and the preprocessed IMU data is obtained, and the obtained data residual is compared with a preset residual threshold, and according to the comparison result, the final fused three-dimensional vector is determined, specifically:

[0121] When the obtained data residual is less than the preset residual threshold, the obtained primary fusion three-dimensional vector is the final fusion three-dimensional vector. Otherwise, the primary fusion three-dimensional vector is corrected by the Kalman filtering algorithm to obtain a corrected fusion three-dimensional vector, and the corrected fusion three-dimensional vector is the final fusion three-dimensional vector.

[0122] In the embodiment, the primary fusion three-dimensional vector is corrected by the Kalman filtering algorithm to obtain a corrected fusion three-dimensional vector. Specifically, the following steps are performed.

[0123] Step S1.3.2.1: determining a predicted state vector. That is, a state transition matrix is set according to a set sampling time interval, and specifically, the state transition matrix is set as follows:

[0124]

[0125] wherein F is the state transition matrix, and Δt is the sampling time interval.

[0126] Further, the predicted state vector at the current time is determined according to the set state transition matrix and the predicted state vector corresponding to the previous time, and specifically, the predicted state vector at the current time is determined as follows:

[0127]

[0128] wherein: is the predicted state vector at the current time r, X r-1 is the predicted state vector at the previous time r-1, and F is the state transition matrix.

[0129] Step S1.3.2.2: determining an observation vector. That is, the observation vector is constructed according to the obtained primary fusion three-dimensional vector, and specifically, the observation vector is constructed as follows:

[0130] Z r = H · X fused + v

[0131] wherein Z r is the observation vector, H is the observation matrix, X fused is the primary fusion three-dimensional vector, and v is the observation noise.

[0132] Step S1.3.2.3: determining a gain matrix. That is, the predicted covariance matrix is constructed according to the joint angle, angular velocity and angular acceleration obtained in step S1.1, and specifically, the predicted covariance matrix is constructed as follows:

[0133]

[0134] wherein: is the predicted covariance matrix, is the angle variance, is the angle velocity variance, σ α2 is the angle acceleration variance, σ θω , σ ωθ is the angle-angle velocity covariance, σ θα , σ αθ is the angle-acceleration covariance, σ ωα , σ αω is the angle velocity-acceleration covariance.

[0135] Further, according to the constructed prediction covariance matrix, the corresponding gain matrix is obtained, specifically:

[0136]

[0137] wherein K r is the gain matrix, H T is the inversion of the observation matrix, R is the observation noise covariance, is the prediction covariance matrix, and H is the observation matrix.

[0138] Step S1.3.2.4: determining the corrected fusion three-dimensional vector. That is, according to the determined prediction state vector, gain matrix and observation vector, the primary fusion three-dimensional vector is corrected to obtain the corrected fusion three-dimensional vector, specifically:

[0139]

[0140] wherein X final is the corrected fusion three-dimensional vector, is the prediction state vector at the current time r, K r is the gain matrix, Z r is the observation vector, and H is the observation matrix.

[0141] Step S2: multi-modal space-time alignment. That is, according to the fusion three-dimensional vector obtained in step S1.3.2, a Transformer model is set, and the corresponding attention weight size is determined through the set Transformer model, and feature reconstruction is performed according to the determined attention weight size. Specifically as follows:

[0142] Step S2.1: data synchronization processing. That is, the time stamp and spatial coordinate system corresponding to each sensor are synchronously aligned to determine the aligned time stamp and spatial coordinate system. Specifically as follows:

[0143] Step S2.1.1: time synchronization processing. That is, according to the time stamp corresponding to the sensor data obtained by each sensor, clock offset compensation is performed to determine the corresponding corrected time stamp, specifically:

[0144] tcor =t raw +(t mas -t sla )-t p-d

[0145] Where: t cor is the timestamp after correction, t raw is the original timestamp, t mas is the PTP master clock reference time, t sla is the node local clock reading, t p-d It is the physical delay time of signal transmission.

[0146] Furthermore, linear interpolation is performed based on the determined corrected timestamp and the sensor data corresponding to the corresponding set original timestamp to obtain the sensor data corresponding to the corrected timestamp, specifically:

[0147]

[0148] in: For time Sensor data, For time Sensor data, For time Sensor data, is the nth corrected timestamp, is the n-1th corrected timestamp, The nth original timestamp.

[0149] Step S2.1.2: Data rejection compensation. A calibration plate was constructed from a 300 x 300 mm medical-grade aluminum alloy. Specifically, nine markers were arranged in a 3 x 3 array on the plate. The marker in the lower left corner of the plate was used as the origin of the global coordinate system. The long side of the plate pointing rightward was designated as the X-axis, the short side pointing upward was designated as the Y-axis, and the direction perpendicular to the plate and outward was designated as the Z-axis.

[0150] In this embodiment, the calibration plate is set up to determine the abnormal data points, and the determined abnormal data points are eliminated. At the same time, temperature compensation is performed on the remaining data points after the abnormal data points are eliminated based on the constructed temperature field. The details are as follows:

[0151] Step S2.1.2.1: Eliminate abnormal data points. That is, through the calibration plate, obtain the mean value of each coordinate component corresponding to each data point, and determine the corresponding three-dimensional space standard deviation based on the obtained mean value of each coordinate component and the corresponding actual measurement value. Specifically,

[0152]

[0153] wherein: σ p is the three-dimensional space standard deviation, p i is the single-point measurement space coordinate, p mean is the coordinate component mean, m is the effective measurement point number, and ||·|| is the Euclidean norm.

[0154] In this embodiment, according to the determined three-dimensional space standard deviation, the three times three-dimensional space standard deviation is set as the abnormality determination threshold, the Euclidean norm between the single-point measurement space coordinate and the coordinate component mean is compared with the abnormality determination threshold, and according to the comparison result, the data point corresponding to the single-point measurement space coordinate is removed. Specifically:

[0155] When the obtained Euclidean norm is greater than the abnormality determination threshold, the data point corresponding to the single-point measurement space coordinate is removed. Conversely, when the obtained Euclidean norm is not greater than the abnormality determination threshold, the data point corresponding to the single-point measurement space coordinate is retained.

[0156] Step S2.1.2.2: temperature compensation. That is, according to the material thermal expansion coefficient corresponding to the calibration plate and the current environmental temperature, the corresponding relative thermal expansion rate is determined, specifically:

[0157] A = a x (T act -T the )

[0158] wherein: A is the relative thermal expansion rate, a is the material thermal expansion coefficient, T act is the current environmental temperature, and T the is the reference temperature.

[0159] Further, according to the obtained relative thermal expansion rate, the data points retained in step S2.2.1 are temperature compensated to obtain the corresponding temperature-compensated space coordinates, specifically:

[0160]

[0161] wherein: P i cor is the temperature-compensated space coordinate, p i is the single-point measurement space coordinate, and A is the relative thermal expansion rate.

[0162] Step S2.1.3: determining the space conversion parameter. That is, according to the temperature-compensated space coordinates and the single-point measurement space coordinates obtained in step S2.1.2.2, the corresponding global coordinate mean and sensor coordinate mean are determined, and the corresponding centralized coordinates are obtained, specifically:

[0163]

[0164] Where: p cen is the coordinate after global coordinate centering, is the sensor’s centered coordinate, mean(p i ) is the global coordinate mean, mean(P i cor ) is the mean value of sensor coordinates, P i cor is the spatial coordinate after temperature compensation, p i Measure the spatial coordinates of a single point.

[0165] Furthermore, based on the obtained global coordinates and sensor coordinates, a covariance matrix is ​​constructed, specifically:

[0166]

[0167] Where: B is the covariance matrix, The coordinates after sensor centering The inversion of p cen The coordinates are the global coordinates after centering.

[0168] In this embodiment, singular value decomposition is performed based on the constructed covariance matrix to obtain the corresponding left singular vectors, singular value matrix, and right singular vectors, and the corresponding rotation matrix is ​​determined based on the obtained left singular vectors, singular value matrix, and right singular vectors, specifically:

[0169]

[0170] Among them: U is the left singular vector, S is the singular value matrix, V is the right singular vector, R rot is the rotation matrix, U T is the inversion of the left singular vector U.

[0171] Furthermore, based on the determined rotation matrix, sensor coordinate mean, and global coordinate mean, the corresponding translation vector is determined as follows:

[0172] T tra =mean(p i )-R rot ·mean(P i cor )

[0173] Where: T tra is the translation vector, mean(p i ) is the global coordinate mean, mean(P i cor ) is the mean value of sensor coordinates, R rot is the rotation matrix.

[0174] Step S2.2: Feature level fusion. That is, according to the time-aligned feature vectors obtained in step S2.1.1, a corresponding fusion three-dimensional vector is constructed, and a query matrix, a key matrix and a value matrix are set according to the constructed fusion three-dimensional vector, specifically:

[0175]

[0176] Wherein: Q is the query matrix, X fused is the primary fusion three-dimensional vector, W Q is the query transformation weight matrix, K is the key matrix, W K is the key transformation weight matrix, V is the value matrix, W V is the value transformation weight matrix.

[0177] Further, according to the obtained query matrix and key matrix, a corresponding attention weight matrix is set, specifically:

[0178]

[0179] Wherein: C is the attention weight matrix, K T is the inverse of the key matrix K, d k is the key vector dimension.

[0180] In this embodiment, according to the obtained value matrix and attention weight matrix, a corresponding reconstructed feature matrix is determined, specifically:

[0181] Z = C · V

[0182] Wherein: Z is the reconstructed feature matrix, C is the attention weight matrix, and V is the value matrix.

[0183] Step S2.3: State adjustment. That is, according to the motion state determined in step S1.2.2, i.e. dynamic mode, mixed mode and static mode, the IMU weight coefficient is adjusted to obtain the corresponding adjusted IMU weight coefficient. It is worth noting that when the motion state is mixed mode, the IMU weight coefficient remains unchanged and is not adjusted.

[0184] In this embodiment, when the motion state is dynamic mode, the initial IMU weight coefficient is adjusted according to the current angular velocity, and compared with the maximum set value of the IMU weight coefficient to determine the adjusted IMU weight coefficient, specifically:

[0185]

[0186] Wherein: is the adjusted IMU weight coefficient in dynamic mode, ||ω|| is the combined magnitude of three-dimensional space angular velocity, and ωIMU is an IMU weight coefficient, is the upper threshold value corresponding to the angular velocity of the preset voltage threshold range.

[0187] Further, when the motion state is the static mode, the initial IMU weight coefficient is adjusted according to the current angular velocity, and the adjusted IMU weight coefficient is determined, specifically:

[0188]

[0189] wherein: is the adjusted IMU weight coefficient in the static mode, ||ω|| is the combined magnitude of the three-dimensional space angular velocity, and ω IMU is an IMU weight coefficient.

[0190] Step S2.4: Determine the fusion parameter. That is, according to the adjusted IMU weight coefficient determined in step S2.3 and the reconstructed feature matrix obtained in step 2.2, the corresponding fusion feature is determined, specifically:

[0191]

[0192] wherein: Z final is a fusion feature, is the adjusted IMU weight coefficient, Z IMU is an IMU modal feature segment, Z mech is a mechanical modal feature segment.

[0193] Step S3: Parameter optimization control. That is, according to the fusion feature obtained in step S2.4, the sampling frequency and the basic stiffness are adjusted. Specifically as follows:

[0194] Step S3.1: Stiffness adjustment. That is, according to the size of the patient's weight, the maximum allowed stiffness is set, specifically:

[0195] S max = 2 × BW

[0196] wherein: S max is the maximum allowed stiffness, and BW is the patient's weight.

[0197] Further, according to the size of the set initial stiffness and the size of the set sampling interval and the acceleration of the sampling point, the initial stiffness is dynamically adjusted, specifically:

[0198] S = S0 + k s · (∑a j × Δt)

[0199] wherein: S is the adjusted stiffness, S0 is the initial stiffness, k s is an acceleration integral coefficient, and aj is the acceleration of the jth sampling point, and Δt is the sampling interval.

[0200] It is worth noting that in the process of real-time adjustment, the obtained adjusted stiffness cannot be greater than the set maximum allowable stiffness.

[0201] Step S3.2: Sampling frequency adjustment. That is, according to the motion state determined in step S1.2.2, i.e. dynamic mode, mixed mode and static mode, the sampling frequency of the corresponding mode is set. Specifically, when the motion state is static mode, the data points are sampled according to the set fixed sampling frequency (for example, 50 Hz).

[0202] Further, when the motion state is mixed mode, the fixed sampling frequency in the static mode is set as the basic sampling frequency, and the sampling frequency is adjusted in real time according to the size of the angular velocity to obtain the adjusted sampling frequency, which is specifically:

[0203] f adj = f0+k f ·||ω||

[0204] Wherein: f adj is the adjusted sampling frequency, f0 is the basic sampling frequency, k f is the adjustment coefficient, and ||ω|| is the combined magnitude of the three-dimensional angular velocity.

[0205] Further, when the motion state is dynamic mode, the corresponding sampling frequency is determined according to the set maximum sampling frequency and the corresponding adjusted sampling frequency in the mixed mode, which is specifically:

[0206] f = min(f max ,f adj )

[0207] Wherein: f is the sampling frequency in dynamic mode, f max is the maximum sampling frequency, and f adj is the adjusted sampling frequency.

[0208] Although embodiments of the present application have been shown and described, it will be understood by those of ordinary skill in the art that various changes, modifications, substitutions and alterations can be made thereto without departing from the principles and spirit of the present application, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A sports training analysis method based on joint modeling of multimodal sensory data, characterized in that: Includes: S1: Setting up a dynamic-static measurement mechanism: setting weights of IMU data, sEMG signal data, and pressure data according to the angular velocity, and performing data fusion based on the weights; S2: Multimodal spatiotemporal alignment: Based on the fused 3D vectors, a Transformer model is set up to determine the attention weights, and feature reconstruction is performed based on the attention weights, including: S2.1: Data synchronization processing: Perform synchronization alignment processing based on the sensor's timestamp and spatial coordinate system to obtain the aligned timestamp and spatial coordinate system; S2.2: Feature-level fusion: Construct a fused three-dimensional vector based on the aligned feature vectors, and set a query matrix, a key matrix, and a value matrix based on the fused three-dimensional vector. Set an attention weight matrix based on the query matrix and the key matrix, and determine a reconstructed feature matrix based on the attention weight matrix and the value matrix. S2.3: State adjustment: Adjust the weight coefficient according to the patient's movement pattern to obtain the adjusted weight coefficient; S2.4: Determine fusion parameters: Determine fusion features based on the adjusted weight coefficients and the reconstructed feature matrix, specifically: Where: Z final To fusion features, is the adjusted IMU weight coefficient, Z IMU is the IMU modal characteristic segment, Z mech is the mechanical modal characteristic segment; S3: Parameter optimization control: According to the fusion characteristics, the sampling frequency and the basic stiffness are adjusted.

2. The sports training analysis method based on multimodal sensor data joint modeling according to claim 1 is characterized in that: Perform data fusion, including: S1.1: Multimodal data preprocessing: IMU data, sEMG signal data, and pressure data are obtained through the set inertial measurement unit, surface electromyography sensor, and flexible pressure sensor array; S1.2: Determine motion state: Determine the patient's motion pattern based on the angular velocity; S1.3: Data fusion: According to the motion pattern, weight coefficients are set and fused data is obtained through the Kalman filter algorithm. Specifically: X fused =ω IMU ×X IMU +oh mech ×X mech Where: X fused is the primary fused three-dimensional vector, X mech is the motion joint angle data, X IMU is the preprocessed IMU data, ω IMU is the IMU weight coefficient, ω mech is the mechanical weight coefficient.

3. The sports training analysis method based on multimodal sensor data joint modeling according to claim 2 is characterized in that: Determine the patient's movement patterns, including: S1.2.1: Determine angular velocity: Set an anatomical coordinate system based on the constructed spatial sensor array, and obtain the X-axis angular velocity component, Y-axis angular velocity component, and Z-axis angular velocity component based on the anatomical coordinate system to determine the angular velocity; S1.2.2: Determine the motion state: Based on the angular velocity, obtain an output voltage, compare the output voltage signal with a preset voltage threshold range, and determine the patient's motion state based on the comparison result, specifically: When the output voltage signal is greater than the upper limit threshold of the preset voltage threshold range, the patient's movement state is a dynamic mode; when the output voltage signal is within the preset voltage threshold range, the patient's movement state is a mixed mode; when the output voltage signal is less than the lower limit threshold of the preset voltage threshold range, the patient's movement state is a static mode.

4. The sports training analysis method based on multimodal sensor data joint modeling according to claim 3 is characterized in that: The upper limit threshold of the preset voltage threshold range is set according to the temperature difference between the current ambient temperature and the reference temperature and the reference threshold voltage, specifically: in: is the upper threshold of the preset voltage threshold range, is the reference threshold voltage, k T is the temperature coefficient, T act is the current ambient temperature, T the is the reference temperature; According to the resistor divider network, the lower limit threshold of the preset voltage threshold range is set, specifically: in: is the lower threshold of the preset voltage threshold range, is the reference threshold voltage, R2 is the lower arm resistor in the resistor divider network, and R1 is the upper arm resistor in the resistor divider network.

5. The sports training analysis method based on multimodal sensor data joint modeling according to claim 2, characterized in that: The fused data obtained includes: S1.3.1: Determine the weight coefficient: Set the IMU weight coefficient and mechanical weight coefficient according to the angular velocity. Specifically: Where: IMU is the IMU weight coefficient, ω mech is the mechanical weight coefficient, e is the base of natural logarithm, and ||ω|| is the composite value of the angular velocity in three-dimensional space; S1.3.2: Determine the fused three-dimensional vector: Based on the IMU weight coefficient and the mechanical weight coefficient, fuse the IMU data, sEMG signal data, and pressure data to obtain a primary fused three-dimensional vector. Simultaneously, based on the motion joint angle data and the IMU data, obtain a data residual, and compare the data residual with a preset residual threshold. Based on the comparison result, determine the final fused three-dimensional vector, specifically: When the data residual is less than the preset residual threshold, the primary fused three-dimensional vector is the final fused three-dimensional vector; otherwise, the primary fused three-dimensional vector is corrected by the Kalman filtering algorithm to obtain a corrected fused three-dimensional vector, and the corrected fused three-dimensional vector is the final fused three-dimensional vector.

6. The sports training analysis method based on multimodal sensor data joint modeling according to claim 5 is characterized in that: The primary fused three-dimensional vector is corrected by the Kalman filter algorithm, including: S1.3.2.1: Determine the predicted state vector: Set a state transfer matrix based on the sampling time interval, and determine the predicted state vector at the current moment based on the state transfer matrix; S1.3.2.2: Determine observation vector: Construct an observation vector based on the primary fused three-dimensional vector; S1.3.2.3: Determine the gain matrix: Construct a prediction covariance matrix based on joint angles, angular velocities, and angular accelerations to determine the gain matrix. S1.3.2.4: Determine a modified fused three-dimensional vector: Modify the primary fused three-dimensional vector based on the predicted state vector, gain matrix, and observation vector to obtain a modified fused three-dimensional vector, specifically: Where: X final is the corrected fused 3D vector, is the predicted state vector of the current moment r, K r is the gain matrix, Z r is the observation vector, and H is the observation matrix.

7. The sports training analysis method based on multimodal sensor data joint modeling according to claim 1, characterized in that: Get the aligned timestamp and spatial coordinate system, including: S2.1.1: Time synchronization processing: Correct the sensor's timestamp through clock offset compensation to obtain the corrected timestamp. Specifically: t cor =t raw +(t mas -t sla )-t p-d Where: t cor is the timestamp after correction, t raw is the original timestamp, t mas is the PTP master clock reference time, t sla is the node local clock reading, t p-d It is the physical delay time of signal transmission; S2.1.2: Data elimination and compensation: Abnormal data points are determined through the calibration plate, and the abnormal data points are eliminated. At the same time, temperature compensation is performed on the remaining data points according to the constructed temperature field; S2.1.3: Determine spatial transformation parameters: Obtain the centered coordinates based on the temperature-compensated spatial coordinates and the single-point measurement spatial coordinates. Construct a covariance matrix based on the centered coordinates. Perform singular value decomposition on the covariance matrix to obtain the left singular vector, singular value matrix, and right singular vector, and determine the translation vector, specifically: T tra =mean(p i )-R rot ·mean(P i cor ) Where: T tra is the translation vector, mean(p i ) is the global coordinate mean, mean(P i cor ) is the mean value of sensor coordinates, R rot is the rotation matrix.

8. The sports training analysis method based on multimodal sensor data joint modeling according to claim 7 is characterized in that: Temperature compensation is performed on the remaining data points, including: S2.1.2.1: Abnormal data point removal: Obtain the mean of each coordinate component of each data point using the calibration plate, and determine the three-dimensional spatial standard deviation. Set an abnormality determination threshold based on the three-dimensional spatial standard deviation, and compare the Euclidean norm between the single-point measurement spatial coordinate and the coordinate component mean with the abnormality determination threshold. Based on the comparison result, remove the data point corresponding to the single-point measurement spatial coordinate. Specifically: When the Euclidean norm is greater than the abnormality determination threshold, the data point corresponding to the single-point measurement space coordinate is eliminated; otherwise, the data point corresponding to the single-point measurement space coordinate is retained; S2.1.2.2: Temperature compensation: Set the relative thermal expansion coefficient based on the thermal expansion coefficient of the calibration plate material and the current ambient temperature, and obtain the temperature-compensated spatial coordinates based on the relative thermal expansion coefficient, specifically: Where: P i cor is the spatial coordinate after temperature compensation, p i is the spatial coordinate of a single point measurement, and A is the relative thermal expansion coefficient.

9. The sports training analysis method based on multimodal sensor data joint modeling according to claim 1, characterized in that: When the patient's motion mode is a dynamic mode, the initial IMU weight coefficient is adjusted according to the angular velocity and the maximum setting value of the IMU weight coefficient to determine the adjusted IMU weight coefficient, specifically: in: is the IMU weight coefficient after adjustment in dynamic mode, ||ω|| is the composite value of the angular velocity in three-dimensional space, ω IMU is the IMU weight coefficient, The angular velocity corresponding to the upper threshold of the preset voltage threshold range; When the patient's motion mode is a static mode, the initial IMU weight coefficient is adjusted according to the angular velocity to determine the adjusted IMU weight coefficient, specifically: in: is the IMU weight coefficient after adjustment in static mode, ||ω|| is the composite value of the angular velocity in three-dimensional space, ω IMU is the IMU weight coefficient.

10. The sports training analysis method based on multimodal sensor data joint modeling according to claim 1, characterized in that: Adjust the sampling frequency and foundation stiffness, including: S3.1: Stiffness Adjustment: Set the maximum allowable stiffness based on the patient's weight. Dynamically adjust the initial stiffness based on the initial stiffness, sampling interval, and sampling acceleration. Specifically: S=S0+k s ·(Σa j ×Δt) Where: S is the adjusted stiffness, S0 is the initial stiffness, k s is the acceleration integral coefficient, a j is the acceleration of the jth sampling point, Δt is the sampling interval; S3.2: Sampling frequency adjustment: Set the sampling frequency according to the patient's movement pattern, specifically: When the patient's movement mode is a static mode, a fixed sampling frequency is set; When the patient's motion mode is a mixed mode, the fixed sampling frequency is adjusted according to the angular velocity, specifically: f adj =f0+k f ·||ω|| Where: f adj is the adjusted sampling frequency, f0 is the basic sampling frequency, k f is the adjustment coefficient, ||ω|| is the composite value of the angular velocity in three-dimensional space; When the patient's motion mode is a dynamic mode, the sampling frequency in the dynamic mode is determined according to the maximum sampling frequency and the adjusted sampling frequency, specifically: f=min(f max ,f adj ) Where: f is the sampling frequency in dynamic mode, f max is the maximum sampling frequency, f adj is the adjusted sampling frequency.

Images