Multimodal dynamic weighted action evaluation method, system and storage medium

Abstract

This invention discloses a multimodal dynamic weighted motion evaluation method, system, and storage medium. It acquires motion posture data from motion video streams; processes the acquired electrocardiogram (ECG) data to extract heart rate features, including average heart rate, heart rate variability, and respiratory periodicity; for each joint sequence, an improved DTW algorithm is used to calculate the similarity score with standard motion. This improved DTW algorithm incorporates a curved path optimization mechanism to reduce the computation of invalid paths; a dynamic weight model is constructed based on the random forest algorithm, dynamically allocating weights to each part according to motion complexity, part importance, and physiological indicators; and a comprehensive evaluation is performed by fusing motion posture data and heart rate data to classify evaluation levels. By allocating dynamic weights based on motion complexity and other factors, and fusing multimodal data to obtain a comprehensive evaluation, it achieves accurate and comprehensive motion evaluation.

Description

Technical Field

[0001] This invention belongs to the field of motion evaluation technology. It relates to a multimodal dynamic weighted motion evaluation method, system and storage medium. Background Technology

[0002] Baduanjin (Eight Pieces of Brocade) is widely promoted in national fitness activities due to its simple movements and significant health benefits. However, traditional training methods rely heavily on human guidance, resulting in significant problems such as strong subjectivity, delayed feedback, and a lack of quantitative data, making it difficult to meet the scientific and precise demands of modern fitness. With technological advancements, intelligent assessment technologies based on sensors and computer vision are gradually emerging, but existing solutions still face many challenges in terms of algorithm accuracy, data fusion, and personalized adaptation.

[0003] The Baduanjin training wristband monitoring device proposed in announcement number CN105797353B integrates a microcontroller, heart rate and blood oxygen sensing units, and position sensing units to record movement trajectory data and compare it with standard movements. However, the inertial sensor algorithm used in this device can only capture one-dimensional motion trajectory data such as acceleration curves, and lacks effective extraction of three-dimensional spatial posture information such as the arm lifting angle (accurate to ±1.5°) and body center of gravity offset in movements such as "Two Hands Supporting the Sky to Regulate the Three Burners". Its core algorithm does not introduce a joint point angle calculation model, resulting in an evaluation error of up to 15% for complex movements. For example, the detection deviation of the torso twisting angle in the "Five Labors and Seven Injuries Looking Back" movement can reach ±5°, seriously affecting the accuracy of the evaluation.

[0004] At the data fusion level, existing technologies generally lack multimodal correlation analysis capabilities. Although the aforementioned patent is equipped with a physiological sensing unit, it does not establish a dynamic correlation model between action features and physiological indicators. For example, in the action of "clenching fists and glaring to increase strength," when the heart rate exceeds the normal range (75-85 beats / minute), the similarity score weight of the action parts is not adaptively adjusted.

[0005] Furthermore, traditional assessment systems lack a dynamic weight allocation mechanism based on individual differences. For example, the fixed weight assessment mode of patent CN105797353B cannot adapt to the physical conditions of different practitioners. For instance, in the "Two Hands Climbing Feet to Strengthen Kidneys and Waist" exercise, the difference in waist flexibility between beginners and advanced practitioners requires different joint angle weight coefficients.

[0006] In summary, existing technologies, due to insufficient algorithm accuracy, lack of multimodal fusion, and weak dynamic adaptation capabilities, are unable to meet the evaluation requirements of Baduanjin training for "unity of form and spirit". Summary of the Invention

[0007] The purpose of this invention is to provide a multimodal dynamic weighted action evaluation method, system, and storage medium, which allocates dynamic weights based on action complexity and other factors, integrates multimodal data to obtain a comprehensive evaluation, and achieves accurate and comprehensive action evaluation.

[0008] The technical solution to achieve the purpose of this invention is as follows:

[0009] A multimodal dynamic weighted action evaluation method includes the following steps:

[0010] S01: Obtain motion pose data from the motion video stream;

[0011] S02: Process the collected electrocardiogram data to extract heart rate features, including average heart rate, heart rate variability, and respiratory periodicity;

[0012] S03: For the joint sequence of each part, the improved DTW algorithm is used to calculate the similarity score with the standard action. The improved DTW algorithm introduces a curved path optimization mechanism to reduce the amount of invalid path calculation.

[0013] S04: A dynamic weight model is constructed based on the random forest algorithm. The weights of each part are dynamically allocated according to the complexity of the movement, the importance of the part, and physiological indicators. The movement posture data and heart rate data are integrated for comprehensive evaluation, and the evaluation level is divided.

[0014] In the preferred technical solution, obtaining motion posture data in step S01 includes:

[0015] The Yolov8n_pose model was used to detect 21 key joints of the human body, calculate the three-dimensional spatial angles and the standard deviation of the joint spacing, and construct the motion time series feature vector, which includes 8 angles and three-dimensional coordinates.

[0016] In the preferred technical solution, a median filtering algorithm is used to remove outliers in the joint coordinates, and the coordinate data is normalized to map each dimension value to the [0,1] interval. ;

[0017] in, After median filtering, `median()` is the median filtering function, `x(t)` represents the original coordinate data of the key point at time `t`, and `t` is the time index in the time series. These are the normalized values ​​of the joint coordinates, where x is the original joint coordinate data. The minimum value of the corresponding dimension of a set of key point coordinate data. This represents the maximum value of the corresponding dimension of a set of keypoint coordinate data.

[0018] In the preferred technical solution, the acquired electrocardiogram data is processed by a three-level filtering algorithm chain, including IIR high-pass filtering to remove baseline drift, FIR band-pass filtering to suppress power frequency interference, and smoothing filtering to enhance R-wave characteristics;

[0019] A fourth-order Butterworth high-pass filter with a cutoff frequency of 0.5Hz is used to filter out low-frequency baseline fluctuations caused by respiratory movements. The transfer function is: s is a complex frequency variable;

[0020] A 51st-order FIR bandpass filter is used, with a passband range of 0.5-40Hz. The frequency response is optimized using the Hanning window function to eliminate 50Hz power frequency noise and electromyographic interference.

[0021] A 5-point moving average filtering algorithm is used. , This represents the output signal value at discrete time point n after 5-point moving average filtering. This represents the value of the original discrete input signal at time point n, where n is the discrete-time index and k is the index variable for the summation operation.

[0022] In the preferred technical solution, the improved DTW algorithm in step S03 includes:

[0023] The key optimization points for improving the DTW algorithm are to limit the path curvature range during time series alignment, allow the current frame to match with a certain number of previous frames, and calculate only the path jump between the current position and a certain number of previous row positions in the dynamic programming matrix.

[0024] In the dynamic programming matrix, a diagonal band is defined, allowing path searches only within this region; locations outside the region are ignored. That is, only paths satisfying the given conditions are computed. The position (i, j) is determined by the bandwidth w, which is dynamically adjusted based on the average inter-frame displacement of the action.

[0025] In the preferred technical solution, step S04 further includes:

[0026] The system trains on labeled data and automatically learns the weight coefficients of each body part under different actions.

[0027] Establish a motion-physiology correlation model: when the heart rate exceeds the normal range, the weight of the corresponding motion part is reduced by a certain value for each additional beat / minute; when the respiratory rate is abnormal, the weight of the motion coordination part is reduced by a certain value for each additional beat / minute.

[0028] In the preferred technical solution, step S04 uses a weighted summation method to calculate the comprehensive score:

[0029] The overall action score is obtained by calculating the weighted sum of the similarity scores of each part and the dynamic weights.

[0030] Heart rate assessment score was calculated using an SVM model, and respiratory rate assessment score was calculated using a linear regression model.

[0031] Multimodal assessment score = 0.6 × overall movement score + 0.3 × heart rate assessment score + 0.1 × respiratory rate assessment score.

[0032] In the preferred technical solution, after step S04, real-time voice and vibration reminders are also included. When the evaluation result is qualified or below, real-time feedback is provided on movement deviation, abnormal physiological state, and improvement suggestions. Multi-dimensional result display: The system's main interface displays historical evaluation records, movement deviation analysis charts, and physiological data waveforms.

[0033] This invention also discloses a multimodal dynamic weighted action evaluation system, comprising:

[0034] The motion posture data acquisition module acquires motion posture data from the motion video stream;

[0035] The ECG data acquisition module processes the collected ECG data and extracts heart rate features, including average heart rate, heart rate variability, and respiratory periodicity.

[0036] The action similarity calculation module uses an improved DTW algorithm to calculate the similarity score with the standard action for the joint sequence of each part. The improved DTW algorithm introduces a curved path optimization mechanism to reduce the amount of invalid path calculation.

[0037] The evaluation module constructs a dynamic weight model based on the random forest algorithm. It dynamically allocates the weight of each body part according to the complexity of the movement, the importance of the body part, and physiological indicators. It integrates movement posture data and heart rate data for comprehensive evaluation and classifies the evaluation level.

[0038] The present invention also discloses a computer storage medium storing a computer program, which, when executed, implements the above-described multimodal dynamic weighted action evaluation method.

[0039] Compared with the prior art, the significant advantages of this invention are:

[0040] The three-level filtering algorithm chain (IIR high-pass filtering → FIR band-pass filtering → smoothing filtering) used in this invention can improve the baseline drift suppression rate of exercise ECG signals to 92%. Combined with the Yolov8n_pose model for real-time detection of 21 joints (processing delay <30ms), it achieves millisecond-level synchronous fusion of motion angle (accuracy ±1.5°) and heart rate variability (HRV).

[0041] The random forest dynamic weight model constructed in this invention can automatically optimize weight allocation based on the complexity of the action (the amplitude / speed of the change of the joint), the importance of the body part (such as the weight of the arm accounting for 70% in "regulating the spleen and stomach by lifting one arm") and physiological indicators (the weight decreases by 2% for every 1 beat / minute increase in heart rate), thereby improving the reliability of personalized evaluation results by more than 50%. Attached Figure Description

[0042] Figure 1 This is a flowchart of the multimodal dynamic weighted action evaluation method in this embodiment;

[0043] Figure 2 This is a flowchart of the multimodal dynamic weighted action evaluation system in this embodiment;

[0044] Figure 3 This is the display interface of the multimodal dynamic weighted action evaluation system in this embodiment. Detailed Implementation

[0045] The principle of this invention is as follows: This invention breaks through the technical bottleneck of traditional single-modal assessment by combining the algorithm of Yolov8n_pose visual perception, three-level electrocardiogram filtering and dynamic weight allocation, and realizes high-precision and personalized motion assessment.

[0046] Example 1:

[0047] like Figure 1 As shown, a multimodal dynamic weighted action evaluation method includes the following steps:

[0048] S01: Obtain motion pose data from the motion video stream;

[0049] S02: Process the collected electrocardiogram data to extract heart rate features, including average heart rate, heart rate variability, and respiratory periodicity;

[0050] S03: For the joint sequence of each part, the improved DTW algorithm is used to calculate the similarity score with the standard action. The improved DTW algorithm introduces a curved path optimization mechanism to reduce the amount of invalid path calculation.

[0051] S04: A dynamic weight model is constructed based on the random forest algorithm. The weights of each part are dynamically allocated according to the complexity of the movement, the importance of the part, and physiological indicators. The movement posture data and heart rate data are integrated for comprehensive evaluation, and the evaluation level is divided.

[0052] Specifically, the Yolov8n_pose model (processing latency <30ms) is used to detect 21 key joints of the human body (head, neck, shoulder, elbow, wrist, hip, knee, ankle, etc.) in real time, calculate three-dimensional spatial angles (such as shoulder joint extension angle and trunk twisting angle) and standard deviation of joint spacing, and construct motion time series feature vectors (8 angles + 3 coordinate dimensions).

[0053] The median filtering algorithm is used to remove outliers in the keypoint coordinates. The formula is as follows: Normalize the coordinate data, mapping each dimension value to the [0,1] interval: ;

[0054] in, This represents the coordinates of the keypoint at time t after median filtering. `median()` is the median filtering function, `x(t)` is the original coordinate data of the keypoint at time t, and `t` is the time index in the time series. These are the normalized values ​​of the joint coordinates, where x is the original joint coordinate data. This represents the minimum value of the corresponding dimension of a set of keypoint coordinate data. This represents the maximum value of the corresponding dimension of a set of keypoint coordinate data.

[0055] Specifically, a self-developed three-lead ECG monitoring module is used to collect heart rate data at a sampling frequency of 1000 times / second. The data is then processed by a three-level filtering algorithm chain (IIR high-pass filtering → FIR band-pass filtering → smoothing filtering) to suppress 92% of baseline drift and electromyographic interference. Respiratory rate (20 times / second) is collected simultaneously, and features such as average heart rate, heart rate variability (HRV), and respiratory cycle regularity are extracted.

[0056] The regularity of the respiratory cycle can be determined by the following methods:

[0057] I. Data Collection

[0058] The system uses a self-developed electrocardiogram (ECG) monitoring module to synchronously collect respiratory rate data at a sampling frequency of 20 times per second. This module works in sync with ECG signal acquisition to ensure time alignment of respiratory data with multimodal data such as posture, heart rate, and other parameters.

[0059] II. Preprocessing and Feature Extraction

[0060] Outlier correction: Outlier processing is performed on the collected respiratory rate data to remove abrupt changes caused by poor sensor contact or motion interference.

[0061] Moving average smoothing: The respiratory rate data is smoothed using a moving average algorithm with a window size of 10, generating a continuous respiratory rate curve, eliminating random noise interference, and highlighting the regularity of the respiratory cycle.

[0062] III. Correlation Analysis with ECG Data

[0063] Although respiratory rate can be directly acquired by sensors, the "regularity of the respiratory cycle" mentioned in the document can also be indirectly derived from electrocardiogram signals:

[0064] Heart rate variability (HRV) in electrocardiogram (ECG) signals is correlated with the respiratory cycle. Analyzing the fluctuation period of HRV can help verify the regularity of respiratory rate. For example, during normal breathing, HRV will show periodic changes consistent with the respiratory rate. When the respiratory rate is abnormal (>16 breaths / minute or <12 breaths / minute), the regularity of HRV will also change accordingly.

[0065] IV. Specific Applications and Evaluation

[0066] When respiratory rate data is used for multimodal fusion assessment:

[0067] If the respiratory rate exceeds the normal range (e.g., >16 breaths / minute or <12 breaths / minute), the system will adjust the assessment weights according to the "motor-physiological association model": for every 1 breath / minute exceeding the normal range, the weight of the motor coordination part will be reduced by 1.5%.

[0068] The respiratory rate assessment score was calculated using a linear regression model and incorporated into the total multimodal assessment score with a weight of 10% (multimodal assessment score = 0.6 × comprehensive action score + 0.3 × heart rate assessment score + 0.1 × respiratory rate assessment score).

[0069] IIR high-pass filtering (to remove baseline drift):

[0070] A fourth-order Butterworth high-pass filter with a cutoff frequency of 0.5Hz is used to filter out low-frequency baseline fluctuations caused by respiratory movements. The transfer function is:

[0071] s is a complex frequency variable;

[0072] FIR bandpass filter (suppressing power frequency interference): A 51st-order FIR bandpass filter is designed with a passband range of 0.5-40Hz. The frequency response is optimized using the Hanning window function, which effectively eliminates 50Hz power frequency noise and electromyographic interference.

[0073] Smoothing filter (enhancing R-wave characteristics):

[0074] A 5-point moving average filtering algorithm is used: , This represents the output signal value at discrete time point n after 5-point moving average filtering. This represents the value of the original discrete input signal at time point n, where n is the discrete-time index and k is the index variable for the summation operation. Testing showed that this filter chain can improve the signal-to-noise ratio (SNR) of the ECG signal from 15dB to 28dB, with an R-wave detection accuracy of 99.2%.

[0075] The core improvement to the DTW algorithm lies in the introduction of a curved path optimization mechanism, which reduces the computational cost of invalid paths to improve efficiency and optimize accuracy. Specific details are as follows:

[0076] I. Limitations of the Traditional DTW Algorithm The traditional Dynamic Time Warping (DTW) algorithm searches for the optimal alignment by traversing the global path, resulting in a computational complexity of O(n log n). (where n is the sequence length), the following problems exist: Computational redundancy: For adjacent frames with small differences in the time series, all possible path jumps still need to be calculated, resulting in low efficiency. Excessive alignment bias: It may introduce "jumping" alignment that does not conform to the laws of human movement (such as matching non-continuous actions across frames), affecting the accuracy of similarity calculation.

[0077] II. Key Optimization Points for Improving the DTW Algorithm

[0078] 1. Curved Path Constraint: Limits the range of path curvature during time series alignment, allowing only the current frame to match with the previous 1-2 frames, avoiding invalid alignment across multiple frames. Implementation: In the dynamic programming matrix, only the path jump between the current position (i,j) and the previous 1-2 rows (i-1, i-2) is calculated.

[0079] 2. Band Constraint Principle: A diagonal band (bandwidth w=5) is defined in the dynamic programming matrix, allowing path searches only within this band; paths outside the band are ignored. Formula: Only calculate those satisfying... The position (i, j) is determined by w, which is dynamically adjusted based on the average inter-frame displacement of the Eight Pieces of Brocade movements (e.g., the average displacement of the "Two Hands Supporting the Sky and Regulating the Three Burners" move corresponds to w=3-5). Effect: Reduces computational complexity from... Down to When w=5, the computational load is reduced by about 80%, and errors caused by over-pruning are avoided by dynamically adapting action features.

[0080] III. Efficiency Improvement and Accuracy Verification

[0081] 1. Comparison of implementation logic with a 30% improvement in computational efficiency: Traditional DTW takes approximately 120ms to process a 50-frame motion sequence; the improved DTW (curved path + layered region) takes only 84ms, a 30% efficiency improvement. Key reason: The path search space is global. Points reduced to approximately The number of points (15n points when w=5) reduces the computational load by more than 75%; redundant calculations are avoided by pruning invalid paths in advance.

[0082] 2. Precision optimization: Improved action timing alignment.

[0083] Traditional algorithms might misalign a user's "raising hand" motion to the standard "flipping palm" phase. The improved algorithm, through curved path constraints, ensures that adjacent frames match only within a reasonable range, reducing the timing error from ±2 frames to ±1 frame. Part similarity accuracy is also improved.

[0084] Taking the "Look Backwards" move as an example, the similarity calculation error of the torso twisting angle after the improvement was reduced from ±5° to ±2.5°, and the mismatch rate caused by invalid paths decreased by 40%.

[0085] IV. Specific Application in the Assessment of Baduanjin Movements: Independent Calculation of Multiple Body Parts:

[0086] An improved DTW algorithm is applied separately to the six major body parts (head, torso, left arm, right arm, left leg, and right leg) to avoid interference caused by mixed calculations of whole-body movements. For example, when calculating the similarity of the right arm, only the joint sequence of the right arm in the standard movement is matched, improving part specificity. Dynamic difficulty adaptation: Beginner learners: The bandwidth is widened (w=8), allowing for a greater range of movement delays (such as a slower movement rhythm) to avoid overly strict evaluation due to strict alignment; Advanced learners: The bandwidth is tightened (w=3), strictly matching the temporal details of the standard movement to accurately capture minute deviations.

[0087] Output results: Each part outputs a similarity score from 0 to 100, and the aligned time series curve (e.g., ...). Figure 3 The image shows "Similarity: 92.39%, Maximum Deviation Joint: Right Shoulder," which visually indicates the specific frame position where the action is lagging or ahead.

[0088] A dynamic weight model is constructed based on the random forest algorithm. The model is trained with 1000 sets of labeled data and automatically learns the weight coefficients of each body part under different movements. Physiological index linkage regulation: A movement-physiological correlation model is established: When the heart rate exceeds the normal range (75-85 beats / minute), the weight of the corresponding movement part decreases by 2% for every 1 beat / minute exceeding the normal range; when the respiratory rate is abnormal (>16 or <12 breaths / minute), the weight of the movement coordination part decreases by 1.5% for every 1 breath / minute exceeding the abnormal respiratory rate. Multimodal fusion assessment: A weighted summation formula is used to calculate the comprehensive score: Multimodal assessment score = 0.6 × comprehensive movement score + 0.3 × heart rate assessment score + 0.1 × respiratory rate assessment score.

[0089] The overall action score is a weighted sum of the similarity scores of each body part and the dynamic weights. The heart rate / respiratory rate assessment scores are calculated using the SVM model and the linear regression model, respectively.

[0090] Real-time voice and vibration alerts: The heart rate monitor integrates a voice broadcast module (e.g., "Right arm raised angle is less than 15°, please increase the extension range") and a vibration alert module (different vibration frequencies correspond to different assessment levels). When the assessment result is qualified or below, it provides real-time feedback on movement deviations, abnormal physiological states, and improvement suggestions (e.g., "Current heart rate 88 beats / minute, it is recommended to slow down the pace of the movement"). Multi-dimensional result display: The system's main interface displays historical assessment records, movement deviation analysis charts (e.g., shoulder joint angle trend curve), and physiological data waveforms (heart rate variability graph). It supports exporting assessment reports to PDF format for easy review by users and coaches.

[0091] In another embodiment, a computer storage medium stores a computer program that, when executed, implements the aforementioned multimodal dynamic weighted action evaluation method. The action evaluation method described above will not be elaborated further here.

[0092] In another embodiment, a multimodal dynamic weighted action evaluation system includes:

[0093] The motion posture data acquisition module acquires motion posture data from the motion video stream;

[0094] The ECG data acquisition module processes the collected ECG data and extracts heart rate features, including average heart rate, heart rate variability, and respiratory periodicity.

[0095] The action similarity calculation module uses an improved DTW algorithm to calculate the similarity score with the standard action for the joint sequence of each part. The improved DTW algorithm introduces a curved path optimization mechanism to reduce the amount of invalid path calculation.

[0096] The evaluation module constructs a dynamic weight model based on the random forest algorithm. It dynamically allocates the weight of each body part according to the complexity of the movement, the importance of the body part, and physiological indicators. It integrates movement posture data and heart rate data for comprehensive evaluation and classifies the evaluation level.

[0097] The workflow of the multimodal dynamic weighted action evaluation system is illustrated below using a preferred embodiment as an example:

[0098] like Figure 2 As shown, it includes the following steps:

[0099] I. Equipment Deployment and Initialization

[0100] 1. Hardware Connection and Parameter Settings

[0101] Camera deployment:

[0102] Fix a 60fps HD camera 2-3 meters away from the user, with the center of the lens level with the user's chest, and adjust the viewing angle to cover the entire action space (75° field of view). Connect it to the evaluation system host via HDMI cable or wireless module to ensure stable video stream transmission.

[0103] Wearing physiological testing equipment:

[0104] Users wear the self-developed ECG monitor (with a built-in 6000mAh lithium battery and an 8-hour battery life) on their wrist or arm, ensuring the sensor fits snugly against the skin. Press and hold the power button for 3 seconds to pair with the main unit via Bluetooth. Once the screen displays "Connected," it will simultaneously collect heart rate (1000 beats / second) and respiratory rate (20 breaths / second) data.

[0105] System initialization:

[0106] Turn on the assessment system host, select the "Eight Pieces of Brocade Exercise Assessment" mode, and confirm that the camera and ECG monitor are "normally connected". Set the assessment parameters according to user needs, such as the difficulty level of the movements (beginner / intermediate / advanced) and the voice broadcast language.

[0107] II. Data Acquisition and Preprocessing

[0108] 1. Synchronous acquisition of multimodal data

[0109] When the user stands in the center of the camera's field of view and clicks "Start Evaluation" on the system interface, the following actions are triggered:

[0110] Motion data: The camera captures human motion images at a frame rate of 60fps. The Yolov8n_pose model detects 21 joints in real time, generating a three-dimensional coordinate time series (x, y, z), and calculating three-dimensional spatial angles (such as shoulder joint extension angle and trunk twist angle) and the standard deviation of joint spacing.

[0111] Physiological data: The electrocardiogram monitor synchronously collects heart rate waveform and respiratory rate data, which are preprocessed by the built-in three-level filtering algorithm (IIR→FIR→smoothing filter) to remove baseline drift and noise interference.

[0112] 2. Data cleaning and normalization

[0113] Motion data processing: Median filtering algorithm (window size 5) is used to remove outliers in joint coordinates. The formula is as follows: The coordinate data is normalized, mapping each dimension value to the [0, 1] interval: .

[0114] in, This represents the coordinates of the keypoint at time t after median filtering. `median()` is the median filtering function, `x(t)` is the original coordinate data of the keypoint at time t, and `t` is the time index in the time series. These are the normalized values ​​of the joint coordinates, where x is the original joint coordinate data. This represents the minimum value of the corresponding dimension of a set of keypoint coordinate data. This represents the maximum value of the corresponding dimension of a set of keypoint coordinate data.

[0115] Physiological data processing: Heart rate data was smoothed using a Butterworth low-pass filter (cutoff frequency 5Hz) to extract mean heart rate and HRV features. Respiratory rate data underwent outlier correction, and a moving average algorithm (window size 10) was used to generate a continuous curve.

[0116] III. Motion Evaluation and Dynamic Weight Calculation

[0117] 1. Action Feature Extraction and Similarity Analysis

[0118] Part division and angle calculation:

[0119] The human body is divided into 6 major parts (head, torso, left arm, right arm, left leg, and right leg). For the joint sequence of each part, the improved DTW algorithm is used to calculate the similarity score with the standard movement. In the "Two Hands Supporting the Sky to Regulate the Three Burners" movement, the angle of the right arm raised (standard 180°) and the perpendicularity to the torso are calculated, and the similarity score ranges from 0 to 100.

[0120] Dynamic weight allocation:

[0121] The 3D coordinates and angles of 21 joints of the human body were obtained using the Yolov8n_pose model. Three-level filtering was applied to the electrocardiogram (ECG) data to extract heart rate features. An improved DTW algorithm was used to calculate motion similarity, incorporating a curved path optimization mechanism to reduce invalid path calculations. A dynamic weight model was constructed based on the random forest algorithm, dynamically assigning weights according to motion complexity (e.g., joint change amplitude, movement speed) and site importance (e.g., core functional areas, vulnerable areas). The calculation formula is as follows:

[0122] Site weight i = RF model output (similarity i, complexity i, site importance i, heart rate bias, respiratory bias)

[0123] in:

[0124] Similarity i: The similarity score of the i-th part is calculated by the improved DTW algorithm.

[0125] Complexity i: Represents the degree of change of the i-th part in the current action. The degree of change in the current action is obtained based on the amplitude and speed.

[0126] Location importance i: Indicates the core function or vulnerability weight (e.g., standard weight percentage) of the i-th location in the current specific action (e.g., the arms in "holding up the sky with both hands to regulate the three jiaos", the torso in "looking back at the five strains and seven injuries").

[0127] Heart rate deviation: The deviation of the current heart rate from the expected normal range of the activity (e.g., 75-85 beats / minute).

[0128] Respiratory deviation: The deviation of the current respiratory rate from the normal range (e.g., 12-16 breaths / minute).

[0129] This model performs a comprehensive evaluation by fusing motion posture and heart rate data, classifying evaluation levels, and providing real-time voice and vibration alerts as well as multi-dimensional result display. The system includes modules for motion posture data acquisition and ECG data acquisition. The program stored on the storage medium can implement this method, and it represents a significant improvement over existing technologies in areas such as joint point detection accuracy.

[0130] The random forest model automatically assigns weights to different body parts based on motion complexity (e.g., joint displacement velocity), body part importance (e.g., the arm has a 70% weight in a lifting motion), and physiological indicators (weight decreases by 2% for every 1 beat / minute increase in heart rate). 3. Multimodal data fusion evaluation

[0131] Overall score calculation:

[0132] The study combined the overall movement score (60% weight), heart rate assessment score (SVM model, 30% weight), and respiratory rate score (linear regression, 10% weight).

[0133] Assessment level classification:

[0134] The scores are divided into 5 levels based on the overall score: Excellent (90-100 points): Standard movements, good physical condition; Good (80-89 points): Basic standard movements, minor adjustments needed; Average (70-79 points): Some deviations exist, normal physical condition; Pass (60-69 points): Obvious problems with movements, targeted improvement needed; Fail (<60 points): Seriously deviates from the standard, it is recommended to suspend training and consult a coach.

[0135] IV. Real-time feedback and personalized guidance

[0136] 1. Multi-channel result feedback

[0137] Voice broadcast:

[0138] After the assessment, the ECG monitor provided real-time voice feedback: Movement issue: "Right arm raised at an angle of less than 15°, it is recommended to increase the extension range to 180°"; Physiological status: "Current heart rate is 88 beats / minute, which is higher than the normal range, it is recommended to slow down the pace of the movement"; Overall recommendation: "The coordination of the movements in this exercise is insufficient, it is recommended to first perform decomposed movement training."

[0139] Vibration alert:

[0140] Different assessment levels correspond to different vibration modes: qualified and below: high frequency vibration (200Hz, lasting 3 seconds); moderate: medium frequency vibration (100Hz, lasting 2 seconds); good and above: low frequency vibration (50Hz, lasting 1 second).

[0141] 2. Historical Data Management and Analysis: Users can view historical evaluation records in the system interface, including: Movement Deviation Trend Chart: Curves showing the changes in joint angles over time; Physiological Indicator Reports: Correlation analysis of heart rate, respiratory rate, and overall score; PDF Report Export: Includes movement score details, improvement suggestions, and training plans, which can be shared with professional coaches.

[0142] System maintenance and model updates: Algorithm upgrade packages are pushed regularly through the cloud server to optimize the Yolov8n_pose model's ability to recognize subtle movements in Baduanjin; Equipment calibration: It is recommended to use a standard calibration board to correct camera distortion weekly to ensure the accuracy of joint coordinates; Data annotation: Users can actively submit typical movement data to the system backend to participate in expanding the annotation dataset and improving the model's generalization ability.

[0143] The following comparison of the present invention with existing technologies in terms of joint point detection accuracy, heart rate calculation error, multimodal fusion delay, and complex motion evaluation error:

[0144]

[0145] The comparison shows that the present invention significantly improves the accuracy of joint detection, heart rate calculation error, multimodal fusion delay, and complex motion evaluation error.

[0146] The present invention has the following advantages:

[0147] 1. The comprehensiveness and accuracy of the assessment have been significantly improved.

[0148] This invention integrates movement posture data with physiological indicators to construct a multimodal assessment system. Through real-time monitoring of 21 joint points' three-dimensional coordinates (angle accuracy ±1.5°) and heart rate (error ±1 beat / minute) and respiratory rate (error ±0.5 beats / minute), it achieves a three-dimensional analysis of the Baduanjin movements. Compared to traditional single-modal techniques, the assessment error for complex movements is reduced from 15% to ≤3%, and the assessment reliability is improved by approximately 50%, simultaneously reflecting the true correlation between movement standardization and physical condition.

[0149] 2. Dynamic weighting enables personalized and accurate assessment.

[0150] A dynamic weight allocation model based on the random forest algorithm can automatically adjust evaluation criteria according to the complexity of the movement, the importance of the body part, and the physiological state. For the "Two Hands Supporting the Sky to Regulate the Three Burners" movement, the model assigns 70% weight to the arms and dynamically calibrates based on the degree of heart rate deviation (weight decreases by 2% for every 1 beat / minute exceeding the deviation). This mechanism allows the evaluation to be adapted to the different physical conditions and practice stages of different users, increasing the rate of meeting users' personalized guidance needs from 40% to 92%.

[0151] 3. Real-time feedback and interactive experience optimization

[0152] The system provides multi-channel feedback through voice broadcasts, vibration alerts, and visual charts. When a movement deviation is detected (such as the right arm being raised at an angle of less than 15°), the voice module immediately provides corrective suggestions, while the vibration module reinforces the alert at different frequencies. This improves user comprehension of evaluation results by approximately 60% and increases movement correction speed by more than 3 times. Historical data reports and PDF reports support training effectiveness tracking and professional guidance integration.

[0153] 4. Enhance usability through collaborative innovation of software and hardware.

[0154] The independently developed ECG monitor uses medical-grade silicone material (IP67 waterproof), has a built-in 6000mAh lithium battery, and can work continuously for 8 hours. It supports high-speed Bluetooth data transmission. The Yolov8n_pose model is deployed in a lightweight manner, with a processing latency of <30ms. Combined with a three-level ECG filtering algorithm, it achieves millisecond-level synchronization between motion analysis and physiological monitoring. The overall system power consumption is reduced by 40%, meeting the needs of long-term use in multiple scenarios.

[0155] 5. Promote the scientific and widespread application of Baduanjin training.

[0156] This invention provides standardized assessment tools for scenarios such as national fitness, rehabilitation medicine, and sports teaching. In the rehabilitation field, it can be used to develop safe training plans through heart rate and movement amplitude monitoring; in teaching scenarios, it supports batch movement data analysis and adjustment of teaching focus. The system supports multilingual operation guidance, is suitable for different user groups, and helps promote the globalization and scientific inheritance of Baduanjin (Eight Pieces of Brocade).

[0157] 6. Technological scalability and industry reference value

[0158] The multimodal fusion framework and dynamic weighting mechanism can be reused in other exercise evaluation scenarios such as Tai Chi and Wu Qin Xi. The core algorithms (improved DTW, three-level filtering, and random forest model) provide a general technical solution for sports biomechanics and smart wearable devices. Tests have shown that they can shorten the development cycle of similar products by 30%, and reduce the technical threshold and development cost.

[0159] The above embodiments are preferred embodiments of the present invention, but the embodiments of the present invention are not limited to the above embodiments. Any changes, modifications, substitutions, combinations, or simplifications made without departing from the spirit and principle of the present invention shall be considered equivalent substitutions and shall be included within the protection scope of the present invention.

Claims

1. A multimodal dynamic weighted action evaluation method, characterized in that, Includes the following steps: S01: Obtain motion posture data from motion video stream and extract 6 major parts of the human body, including head, torso, left arm, right arm, left leg, and right leg; S02: Process the collected electrocardiogram data to extract heart rate features, including average heart rate, heart rate variability, and respiratory periodicity; S03: For the joint sequence of each human body part, the improved DTW algorithm is used to calculate the similarity score with the standard action. The improved DTW algorithm introduces a curved path optimization mechanism to reduce the amount of invalid path calculation. The improved DTW algorithm includes: The key optimization points for improving the DTW algorithm are to limit the path curvature range during time series alignment, allow the current frame to match with the previous 1-2 frames, and calculate only the path jump between the current position and the previous row position in the dynamic programming matrix. In the dynamic programming matrix, a diagonal band is defined, allowing path searches only within this region; locations outside the region are ignored. That is, only paths satisfying the given conditions are computed. Location The bandwidth w is dynamically adjusted based on the average inter-frame displacement of the Eight-Section Brocade movements. S04: A dynamic weight model is constructed based on the random forest algorithm to dynamically allocate weights to different parts of the human body according to the complexity of the action, the importance of the body part, and physiological indicators; including: Using labeled data for training, the system automatically learns the weight coefficients of each body part under different movements of the Eight Pieces of Brocade; body part weights. i = RF model output (similarity) i Complexity i Importance of location i (Heart rate deviation, respiratory deviation) Among them, similarity i The similarity score for the i-th part is calculated using the improved DTW algorithm, with a complexity of [missing information]. i Characterize the complexity of the i-th part's change in the current action; the complexity of the change in the current action is obtained based on amplitude and velocity; part importance. i This represents the core function or vulnerability weight of the i-th body part in the current specific action; heart rate deviation is the deviation of the current heart rate from the expected normal range of the action; respiratory deviation is the deviation of the current respiratory rate from the normal range; Establish a motion-physiology correlation model: When the heart rate exceeds the normal range, the weight of the corresponding movement part is reduced by a certain value for each additional beat / minute; when the respiratory rate is abnormal, the weight of the movement coordination part is reduced by a certain value for each additional beat / minute. The system integrates movement and posture data with heart rate data for comprehensive evaluation and classifies the evaluation levels.

2. The multimodal dynamic weighted action evaluation method according to claim 1, characterized in that, Step S01 involves acquiring motion posture data, including: The Yolov8n_pose model was used to detect 21 key joints of the human body, calculate the three-dimensional spatial angles and the standard deviation of the joint spacing, and construct the motion time series feature vector, which includes 8 angles and three-dimensional coordinates.

3. The multimodal dynamic weighted action evaluation method according to claim 2, characterized in that, The median filtering algorithm is used to remove outliers in the keypoint coordinates. The formula is as follows: Normalize the coordinate data, mapping each dimension value to the [0,1] interval: ; in, These are the coordinates of the joint at time t after median filtering. Let x(t) be the median filter function, x(t) be the original coordinate data of the key point at time t, and t be the time index in the time series. These are the normalized values ​​of the joint coordinates, where x is the original joint coordinate data. This represents the minimum value of the corresponding dimension of a set of keypoint coordinate data. This represents the maximum value of the corresponding dimension of a set of keypoint coordinate data.

4. The multimodal dynamic weighted action evaluation method according to claim 1, characterized in that, The acquired ECG data were processed by a three-stage filtering algorithm chain, including IIR high-pass filtering to remove baseline drift, FIR band-pass filtering to suppress power frequency interference, and smoothing filtering to enhance R-wave characteristics. A fourth-order Butterworth high-pass filter with a cutoff frequency of 0.5Hz is used to filter out low-frequency baseline fluctuations caused by respiratory movements. The transfer function is: ; s is a complex frequency variable; A 51st-order FIR bandpass filter is used, with a passband range of 0.5-40Hz. The frequency response is optimized using the Hanning window function to eliminate 50Hz power frequency noise and electromyographic interference. A 5-point moving average filtering algorithm is used. , This represents the output signal value at discrete time point n after 5-point moving average filtering. This represents the value of the original discrete input signal at time point n, where n is the discrete-time index and k is the index variable for the summation operation.

5. The multimodal dynamic weighted action evaluation method according to claim 1, characterized in that, In step S04, a weighted summation method is used to calculate the overall score: The overall action score is obtained by calculating the weighted sum of the similarity scores of each part and the dynamic weights. Heart rate assessment score was calculated using an SVM model, and respiratory rate assessment score was calculated using a linear regression model. Multimodal assessment score = 0.6 × overall movement score + 0.3 × heart rate assessment score + 0.1 × respiratory rate assessment score.

6. The multimodal dynamic weighted action evaluation method according to claim 1, characterized in that, Step S04 is followed by real-time voice and vibration alerts. When the evaluation result is qualified or below, real-time feedback is provided on movement deviations, abnormal physiological states, and improvement suggestions. Multi-dimensional result display: The system's main interface displays historical evaluation records, movement deviation analysis charts, and physiological data waveforms.

7. A multimodal dynamic weighted action evaluation system, characterized in that, include: The motion posture data acquisition module acquires motion posture data from the motion video stream and extracts six major parts of the human body, including the head, torso, left arm, right arm, left leg, and right leg. The ECG data acquisition module processes the collected ECG data and extracts heart rate features, including average heart rate, heart rate variability, and respiratory periodicity. The action similarity calculation module calculates the similarity score between the joint sequence of each human body part and the standard action using the improved DTW algorithm. The improved DTW algorithm introduces a curved path optimization mechanism to reduce the amount of invalid path calculation. The improved DTW algorithm includes: The key optimization points for improving the DTW algorithm are to limit the path curvature range during time series alignment, allow the current frame to match with the previous 1-2 frames, and calculate only the path jump between the current position and the previous row position in the dynamic programming matrix. In the dynamic programming matrix, a diagonal band is defined, allowing path searches only within this region; locations outside the region are ignored. That is, only paths satisfying the given conditions are computed. Location The bandwidth w is dynamically adjusted based on the average inter-frame displacement of the Eight-Section Brocade movements. The evaluation module, based on the random forest algorithm, constructs a dynamic weight model to dynamically allocate weights to different parts of the body according to action complexity, body part importance, and physiological indicators; it includes: Using labeled data for training, the system automatically learns the weight coefficients of each body part under different movements of the Eight Pieces of Brocade; body part weights. i = RF model output (similarity) i Complexity i Importance of location i (Heart rate deviation, respiratory deviation) Among them, similarity i The similarity score for the i-th part is calculated using the improved DTW algorithm, with a complexity of [missing information]. i Characterize the complexity of the i-th part's change in the current action; the complexity of the change in the current action is obtained based on amplitude and velocity; part importance. i This represents the core function or vulnerability weight of the i-th body part in the current specific action; heart rate deviation is the deviation of the current heart rate from the expected normal range of the action; respiratory deviation is the deviation of the current respiratory rate from the normal range; Establish a motion-physiology correlation model: When the heart rate exceeds the normal range, the weight of the corresponding movement part is reduced by a certain value for each additional beat / minute; when the respiratory rate is abnormal, the weight of the movement coordination part is reduced by a certain value for each additional beat / minute. The system integrates movement and posture data with heart rate data for comprehensive evaluation and classifies the evaluation levels.

8. A computer storage medium having a computer program stored thereon, characterized in that, When the computer executes the computer program, it implements the multimodal dynamic weighted action evaluation method according to any one of claims 1-6.

Images