A lower limb coordination and expression training method based on posture monitoring

Abstract

The application provides a lower limb coordination and expression training method based on posture monitoring, and belongs to the field of auxiliary rehabilitation training, and comprises the following steps: obtaining a dance time dynamic feature weighting factor of each posture feature dimension of each key joint according to dance time sensitivity and local feature volatility; the dance time dynamic feature weighting factor is used to represent the compensation significance of each posture feature dimension of each key joint at a current motion moment; the original feature matrix is enhanced by using the dance time dynamic feature weighting factor to obtain an enhanced feature matrix; and the enhanced feature matrix is input into a pre-trained neural network model GCN for compensation identification and early warning of user posture; and the application can significantly improve the posture identification precision of early and subtle compensation actions of the user and the accuracy of monitoring and early warning.

Description

Technical Field

[0001] This invention belongs to the field of assisted rehabilitation training, specifically relating to a method for lower limb coordination and expression training based on posture monitoring. Background Technology

[0002] In the fields of rehabilitation medicine and dance-based exercise training, the use of video pose monitoring technology to capture real-time human skeletal data and analyzing movement quality through deep learning models has become a mainstream trend when training users' lower limb coordination and expressive abilities. Among these methods, Graph Convolutional Networks (GCNs) based on Skeleton Graphs are widely used for spatiotemporal motion recognition tasks involving non-Euclidean data structures (i.e., human joint and skeletal connections) due to their inherent topological advantages. This approach effectively captures the anatomical spatial relationships between joints and the temporal dynamics of movements, thereby enabling the evaluation of the correctness of user movements.

[0003] However, in practical applications, the recognition of compensatory movements requires the model to have extremely high sensitivity to certain weak but crucial kinematic features (such as slight inversion of the ankle joint and minor lateral displacement of the pelvis). Existing GCN methods typically use uniform or preset weights for learning all joints and kinematic features (such as position, velocity, and angle). This leads to the feature signals carried by large-amplitude movements of the trunk or large joints (such as the hip joint) dominating the model learning process when analyzing complex or large-amplitude movements (such as squats and lunges), thus ignoring the weak feature signals of key small joints when compensation occurs. This deficiency seriously affects the recognition accuracy of subtle, early, or atypical compensatory movements and reduces the accuracy of posture monitoring and early warning. Summary of the Invention

[0004] To address the problem that existing GCN models in lower limb motion recognition lack adaptive weighting of key kinematic features of dancing movements, resulting in weak compensatory signals being overwhelmed by large-amplitude motion features, this invention provides a lower limb coordination and expression training method based on posture monitoring.

[0005] To achieve the above objectives, the present invention provides the following technical solution:

[0006] Acquire the skeletal posture data of the user during lower limb dance training, and extract the kinematic feature data of key joint movements to construct an original feature matrix; the original feature matrix includes time dimension, joint dimension and posture feature dimension.

[0007] Based on the fluctuation of each posture feature dimension of the key joint within the sliding time window during the dance, the local feature variability of each posture feature dimension of the key joint is obtained; based on the positional distribution of the key joints and the comparative analysis of the posture feature dimensions of the key joints with those of normal users, the dance-time compensation sensitivity of each posture feature dimension of the key joint is obtained.

[0008] Based on the compensatory sensitivity and local feature fluctuations during the dance, a dynamic feature weighting factor for each posture feature dimension of each key joint during the dance is obtained; the dynamic feature weighting factor during the dance is used to characterize the compensatory significance of each posture feature dimension of each key joint at the current action moment; the original feature matrix is ​​enhanced using the dynamic feature weighting factor during the dance to obtain an enhanced feature matrix;

[0009] The enhanced feature matrix is ​​input into the pre-trained neural network model GCN for compensatory recognition and early warning of user posture.

[0010] Preferably, the specific method for acquiring the user's posture skeleton data during lower limb dance training and extracting kinematic feature data from key joint movements to construct an original feature matrix includes:

[0011] The system acquires continuous video frames of a user performing lower limb dance training using a depth camera, and extracts the user's posture skeleton data from each frame of the user's lower limb training image using a posture estimation algorithm; the user's posture skeleton data includes the position coordinates of several key joints in each frame of the user's lower limb training image.

[0012] Based on the position coordinates of key joints, kinematic feature data of each key joint in the action of each key joint in each frame of user lower limb training image are calculated, and the original feature matrix is ​​constructed.

[0013] Preferably, the specific method for calculating the kinematic feature data of each key joint in the movement of each frame of the user's lower limb training image based on the position coordinates of key joints and constructing the original feature matrix includes:

[0014] The kinematic feature data of the action includes several posture feature dimensions: absolute position, relative velocity, and joint angle feature values;

[0015] The position coordinates of the i-th key joint in the t-th frame of the user's lower limb training image are denoted as the absolute position of the i-th key joint; the Euclidean distance between the position coordinates of the i-th key joint in the t-th frame of the user's lower limb training image and the position coordinates of the i-th key joint in the (t-1)-th frame of the user's lower limb training image is denoted as the relative velocity of the i-th key joint; the sine value of the angle between the femur and tibia in the t-th frame of the user's lower limb training image is denoted as the joint angle feature value of the i-th key joint.

[0016] Finally, all frames All key joints and all pose feature dimensions The data is concatenated to construct the original feature matrix. .

[0017] Preferably, the method for obtaining the local feature variability of each posture feature dimension of each key joint based on the variability of each posture feature dimension within the sliding time window during the dance includes:

[0018] A time window parameter 'a' is preset. During the continuous video frames when the user is performing lower limb dance training, the user's lower limb training images that are 'a' most recent before the user's lower limb training image in frame t and the user's lower limb training images that are 'a' most recent after the user's lower limb training image in frame t are combined to form an image sequence, which is denoted as the sliding time window sequence during the dance in frame t.

[0019] The standard deviation of the dth pose feature dimension of the i-th key joint within the k-th key joint within the sliding time window sequence during the t-th frame of the dance is denoted as the local fluctuation amplitude of the dth pose feature dimension of the i-th key joint within the sliding time window sequence during the k-th frame of the dance.

[0020] Based on the local fluctuation amplitude, obtain the local fluctuation change rate and enhanced fluctuation amplitude of the dth pose feature dimension of the i-th key joint within the sliding time window sequence during each frame of the dance;

[0021] The normalized value of the product of the mean of the local fluctuation change rate of the dth pose feature dimension of the i-th key joint in the sliding time window sequence of all frames in the t-th frame of the dance-time sequence and the enhanced fluctuation amplitude of the dth pose feature dimension of the i-th key joint in the sliding time window sequence of the t-th frame of the dance-time sequence is used as the local feature fluctuation of the dth pose feature dimension of the i-th key joint in the t-th frame of the user's lower limb training image.

[0022] Preferably, the specific method for obtaining the local fluctuation change rate and enhanced fluctuation amplitude of the d-th pose feature dimension of the i-th key joint within the sliding time window sequence during each frame of the dance based on the local fluctuation amplitude is as follows:

[0023] The absolute value of the difference between the local fluctuation amplitude of the dth pose feature dimension of the i-th key joint in the sliding time window sequence during the k-th frame of the dance and the local fluctuation amplitude of the dth pose feature dimension of the i-th key joint in the sliding time window sequence during the (k-1)-th frame of the dance is denoted as the local fluctuation change rate of the dth pose feature dimension of the i-th key joint in the sliding time window sequence during the k-th frame of the dance.

[0024] The maximum standard deviation of the dth pose feature dimension of the i-th key joint within the sliding time window sequence of all frames during the dance in frame t is denoted as the enhanced fluctuation amplitude of the dth pose feature dimension of the i-th key joint within the sliding time window sequence of the dance in frame t.

[0025] Preferably, the method for obtaining the dance-time compensation sensitivity of each posture feature dimension of each key joint based on the positional distribution of key joints and the comparative analysis of the posture feature dimensions of key joints with those of normal users includes:

[0026] Taking the hip joint as the target joint, in the t-th frame of the user's lower limb training image, the normalized value of the Euclidean distance between the position coordinates of the i-th joint and the position coordinates of the target joint is denoted as the sensitivity factor of the i-th joint.

[0027] By comparing and analyzing the posture features of key joints during dancing with those of normal users, the motion completion factor of each posture feature dimension of each key joint is obtained.

[0028] The product of the action completion factor of the dth pose feature dimension of the i-th key joint and the sensitivity factor of the i-th key joint is denoted as the compensatory anomaly of the dth pose feature dimension of the i-th key joint.

[0029] A static weight w is preset, and the normalized value of the product between the compensatory anomaly of the dth pose feature dimension of the i-th key joint and the preset static weight w is used as the dance compensatory sensitivity of the dth pose feature dimension of the i-th key joint in the t-th frame of the user's lower limb training image.

[0030] Preferably, the method for obtaining the action completion factor for each posture feature dimension of each key joint by comparing and analyzing the posture feature dimensions of key joints during dancing with those of normal users includes the following specific methods:

[0031] The sequence of all user lower limb training images preceding the t-th frame is denoted as the t-th frame historical reference image sequence.

[0032] In the video frames of a normal user's lower limb training, the local fluctuation amplitude of the dth posture feature dimension of the i-th key joint in the sliding time window sequence of all frames in the historical reference image sequence of the t-th frame during dancing is used to form a sequence, and the normal movement fluctuation amplitude sequence of the dth posture feature dimension of the i-th key joint in the user's lower limb training image of the t-th frame is used.

[0033] The sequence is formed by the local fluctuation amplitude of the dth posture feature dimension of the i-th key joint in the sliding time window sequence of all frames in the historical reference image sequence of the t-th frame during the user's lower limb training video frame, and the comparative motion fluctuation amplitude sequence of the dth posture feature dimension of the i-th key joint in the user's lower limb training image of the t-th frame.

[0034] The absolute value of the Pearson correlation coefficient between the normal motion fluctuation amplitude sequence of the dth posture feature dimension of the i-th key joint in the t-th frame of the user's lower limb training image and the comparative motion fluctuation amplitude sequence of the dth posture feature dimension of the i-th key joint in the t-th frame of the user's lower limb training image is denoted as the motion completion factor of the dth posture feature dimension of the i-th key joint.

[0035] Preferably, the method for obtaining the dynamic feature weighting factor for each posture feature dimension of each key joint during dance based on dance-time compensation sensitivity and local feature fluctuation includes:

[0036] In the user's lower limb training image in frame t, the product of the dance-time compensation sensitivity and local feature variability of the dth pose feature dimension of the i-th key joint is denoted as the weighted value of the dth pose feature dimension of the i-th key joint; the product of the mean of the dance-time compensation sensitivity and the mean of the local feature variability of all pose feature dimensions of the i-th key joint is denoted as the feature weighted mean of the i-th key joint.

[0037] The normalized value of the ratio between the weighted value of the dth pose feature dimension of the i-th key joint and the weighted mean value of the features of the i-th key joint is used as the dynamic feature weighting factor of the dth pose feature dimension of the i-th key joint in the dance-time in the t-th frame of the user's lower limb training image.

[0038] Preferably, the step of enhancing the original feature matrix using the dynamic feature weighting factor during the dance to obtain the enhanced feature matrix includes the following specific formula:

[0039]

[0040] In the formula, Represents the enhanced feature matrix; Represents the original feature matrix; This represents the dynamic feature weighting factor for the d-th pose feature dimension of the i-th key joint in the t-th frame of the user's lower limb training image during dance.

[0041] Preferably, the method for inputting the enhanced feature matrix into the pre-trained neural network model GCN for user pose compensation recognition and early warning includes:

[0042] Enhance the feature matrix The input is fed into the trained neural network ST-GCN for feature extraction and action classification, and the output is the compensation mode and compensation risk score P of the current action.

[0043] A threshold parameter T1 is preset. If the compensation risk score P is greater than or equal to the threshold parameter T1, an early warning mechanism is immediately triggered, and a targeted correction instruction is generated.

[0044] The lower limb coordination and expression training method based on posture monitoring provided by this invention has the following beneficial effects:

[0045] This invention obtains a dynamic feature weighting factor for each posture feature dimension of each key joint during dance based on the sensitivity to compensation during dance and the volatility of local features. This dynamic feature weighting factor characterizes the compensatory significance of each posture feature dimension of each key joint at the current moment of the action. The original feature matrix is ​​enhanced using this dynamic feature weighting factor to obtain an enhanced feature matrix. The enhanced feature matrix is ​​then input into a pre-trained neural network model (GCN) for user posture compensation recognition and early warning. By adaptively strengthening the attention weights of the joint features most sensitive to compensation, the original feature matrix is ​​enhanced and input into the ST-GCN model for compensation recognition, thereby effectively preventing the loss of weak compensation signals and significantly improving the recognition accuracy and early warning accuracy of early and subtle compensation movements. Attached Figure Description

[0046] To more clearly illustrate the embodiments and design schemes of the present invention, the accompanying drawings required for this embodiment will be briefly described below. The drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0047] Figure 1 This is a flowchart illustrating the steps of a lower limb coordination and expression training method based on posture monitoring, according to an exemplary embodiment of the present invention.

[0048] Figure 2 It shows the kinematic characteristic curve of the left knee joint angle in a lower limb coordination and expression training method based on posture monitoring;

[0049] Figure 3 It shows a local characteristic fluctuation heatmap of the left knee joint angle in a lower limb coordination and expression training method based on posture monitoring;

[0050] Figure 4 It shows a heatmap of the compensatory sensitivity of the left knee joint angle in a lower limb coordination and expression training method based on posture monitoring;

[0051] Figure 5 It shows a real-world comparison chart of compensatory risk scores for a lower limb coordination and expression training method based on posture monitoring. Detailed Implementation

[0052] To enable those skilled in the art to better understand and implement the technical solutions of the present invention, the present invention will be described in detail below with reference to the accompanying drawings and specific embodiments. The following embodiments are only used to more clearly illustrate the technical solutions of the present invention and should not be construed as limiting the scope of protection of the present invention.

[0053] The technical solutions provided by the various embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0054] First, this invention provides a lower limb coordination and expression training method based on posture monitoring, specifically as follows: Figure 1 As shown, it includes the following steps:

[0055] Step S001: Obtain the posture skeleton data of the user during lower limb dance training, and extract the kinematic feature data of key joint movements to construct the original feature matrix; the original feature matrix includes time dimension, joint dimension and posture feature dimension.

[0056] The system acquires continuous video frames of users performing lower limb dance exercises (such as squatting) using a depth camera, and extracts the user's posture skeleton data from each frame of the user's lower limb training image using a pose estimation algorithm.

[0057] The pose estimation algorithm is an existing technology and will not be described in detail here. The user's pose skeleton data includes the position coordinates of several key joints (such as hip joint, knee joint, and ankle joint) in each frame of the user's lower limb training image.

[0058] This example illustrates the use of a user performing a squatting exercise for lower limb training, with the right knee joint being the i-th key joint:

[0059] Preferably, in one embodiment of the present invention, the specific method for calculating the kinematic feature data of each key joint in the movement of each key joint in each frame of user lower limb training image based on the position coordinates of key joints is as follows:

[0060] The kinematic feature data during the dance includes several posture feature dimensions: absolute position, relative velocity, and joint angle feature values;

[0061] The position coordinates of the i-th key joint in the t-th frame of the user's lower limb training image are denoted as the absolute position of the i-th key joint; the Euclidean distance between the position coordinates of the i-th key joint in the t-th frame of the user's lower limb training image and the position coordinates of the i-th key joint in the (t-1)-th frame of the user's lower limb training image is denoted as the relative velocity of the i-th key joint; the sine value of the angle between the femur and tibia in the t-th frame of the user's lower limb training image is denoted as the joint angle feature value of the i-th key joint.

[0062] Finally, all frames All key joints and all pose feature dimensions The data is concatenated to construct the original feature matrix. .

[0063] It should be noted that constructing the original feature matrix is ​​an existing technology, and will not be elaborated on in this embodiment. By constructing the original feature matrix containing multi-dimensional physical information, a comprehensive and digital description of human motion is provided for subsequent analysis, which is the data foundation for intelligent recognition.

[0064] Thus, the user's posture skeleton data during lower limb dance training is obtained, and the kinematic features of key joints are extracted to construct the original feature matrix; the original feature matrix includes time dimension, joint dimension and posture feature dimension.

[0065] Step S002: Based on the fluctuation of each posture feature dimension of the key joint within the sliding time window during the dance, obtain the local feature variability of each posture feature dimension of each key joint; based on the position distribution of the key joints and a comparative analysis of the posture feature dimensions of the key joints with those of normal users, obtain the dance-time compensation sensitivity of each posture feature dimension of each key joint.

[0066] It should be noted that when users perform lower limb training movements, such as squats, there are usually different phases (such as the descent phase, the hold phase, and the ascent phase). When compensatory movements occur, joint kinematic characteristics (such as joint angle characteristic values) will show abnormal local fluctuations during key movement phases (such as the bottom of a squat). That is, when the human body compensates (such as when the knees tremble), its characteristic data will fluctuate drastically in a short period of time. Therefore, by capturing and quantifying this local fluctuation through a sliding window, potential abnormal kinematic characteristics can be preliminarily identified.

[0067] Please see Figure 2The figure shows the kinematic characteristic curve of the left knee joint angle in a lower limb coordination and expression training method based on posture monitoring. The green dashed line in the figure clearly marks the moment when the compensatory movement begins (around frame 30). After this point, the curve shows continuous local fluctuations, that is, the user exhibits the compensatory movement of knee valgus during lower limb training.

[0068] Preferably, in one embodiment of the present invention, the specific method for obtaining the local feature variability of each posture feature dimension of each key joint based on the variability of each posture feature dimension of the key joint within the sliding time window during the dance is as follows:

[0069] A time window parameter 'a' is preset. In this embodiment, a=5 is used as an example. This embodiment does not impose specific limitations. The value of 'a' depends on the specific implementation.

[0070] In the continuous video frames when the user is performing lower limb dance training, the user's lower limb training images that are most recent a frames before the user's lower limb training image in frame t and the user's lower limb training images that are most recent a frames after the user's lower limb training image in frame t are combined to form an image sequence, which is denoted as the sliding time window sequence during the dance in frame t.

[0071] The standard deviation of the dth pose feature dimension of the i-th key joint within the k-th key joint within the sliding time window sequence during the t-th frame of the dance is denoted as the local fluctuation amplitude of the dth pose feature dimension of the i-th key joint within the sliding time window sequence during the k-th frame of the dance.

[0072] The absolute value of the difference between the local fluctuation amplitude of the dth pose feature dimension of the i-th key joint in the sliding time window sequence during the k-th frame of the dance and the local fluctuation amplitude of the dth pose feature dimension of the i-th key joint in the sliding time window sequence during the (k-1)-th frame of the dance is denoted as the local fluctuation change rate of the dth pose feature dimension of the i-th key joint in the sliding time window sequence during the k-th frame of the dance.

[0073] The maximum standard deviation of the dth pose feature dimension of the i-th key joint within the sliding time window sequence of all frames during the dance in frame t is denoted as the enhanced fluctuation amplitude of the dth pose feature dimension of the i-th key joint within the sliding time window sequence of the dance in frame t.

[0074] The normalized value of the product between the mean of the local fluctuation change rate of the dth pose feature dimension of the i-th key joint in the sliding time window sequence of all frames in the t-th frame of the dance and the enhanced fluctuation amplitude of the dth pose feature dimension of the i-th key joint in the sliding time window sequence of the t-th frame of the dance is used as the local feature fluctuation of the dth pose feature dimension of the i-th key joint in the t-th frame of the user's lower limb training image.

[0075] The specific formula is as follows:

[0076]

[0077] In the formula, This represents the local feature variability of the d-th pose feature dimension of the i-th key joint in the t-th frame of the user's lower limb training image; It represents the standard deviation of the d-th pose feature dimension of the i-th key joint within the sliding time window sequence of all frames during the dance in the t-th frame. This represents the number of sliding time window sequences in the sliding time window sequence during the dance of frame t; The standard deviation of the d-th pose feature dimension of the i-th key joint in the k-th frame of the sliding time window sequence during the t-th frame of the dance sequence; This represents the standard deviation of the d-th pose feature dimension of the i-th key joint in the (k-1)-th frame of the sliding time window sequence during the t-th frame of the dance sequence; Indicates taking the absolute value; This represents the function that takes the maximum value. This represents the linear normalization function.

[0078] It should be noted that when the volatility of a certain motion posture feature dimension changes drastically in a short period of time (e.g., from stable holding to sudden shaking), or when the overall volatility is large, the local feature volatility will increase, indicating that the posture feature dimension has local anomalies or instability in the current action phase; therefore, the local feature volatility is positively correlated with the sum of the local volatility change rates of the sliding time window sequence during adjacent frame dancing, and is also positively correlated with the maximum local volatility amplitude.

[0079] Please see Figure 3 The figure shows a heatmap of the local feature fluctuation of the left knee joint angle in a lower limb coordination and expression training method based on posture monitoring. The figure shows the local feature fluctuation of the posture feature dimensions of all key joints in the form of a heatmap. The figure shows that the posture feature dimensions of the knee joint, especially the values ​​of the angle and angular velocity features, are significantly higher than those of other key joints because compensatory movements lead to increased local fluctuations.

[0080] It should be noted that not all local fluctuations indicate compensation; for example, fluctuations in knee joint angle characteristics are a normal part of squatting; only when such fluctuations occur in joints of high concern in rehabilitation goals and their fluctuation patterns deviate from the preset functional constraints are they indicated as compensation.

[0081] Preferably, in one embodiment of the present invention, the specific method for obtaining the dance-time compensation sensitivity of each posture feature dimension of each key joint based on the positional distribution of key joints and a comparative analysis of the posture feature dimensions of key joints with those of normal users is as follows:

[0082] Since the knee joint is the core monitoring indicator in the squatting movement, a static weight w is preset for the i-th key joint. In this embodiment, w=0.9 is used as an example for description. This embodiment does not make specific limitations. The weight w depends on the specific implementation.

[0083] Taking the hip joint as the target joint, in the t-th frame of the user's lower limb training image, the normalized value of the Euclidean distance between the position coordinates of the i-th joint and the position coordinates of the target joint is denoted as the sensitivity factor of the i-th joint.

[0084] The sequence of all user lower limb training images preceding the t-th frame is denoted as the t-th frame historical reference image sequence.

[0085] In the video frames of a normal user's lower limb training, the local fluctuation amplitude of the dth posture feature dimension of the i-th key joint in the sliding time window sequence of all frames in the historical reference image sequence of the t-th frame during dancing is used to form a sequence, and the normal movement fluctuation amplitude sequence of the dth posture feature dimension of the i-th key joint in the user's lower limb training image of the t-th frame is used.

[0086] The sequence is formed by the local fluctuation amplitude of the dth posture feature dimension of the i-th key joint in the sliding time window sequence of all frames in the historical reference image sequence of the t-th frame during the user's lower limb training video frame, and the comparative motion fluctuation amplitude sequence of the dth posture feature dimension of the i-th key joint in the user's lower limb training image of the t-th frame.

[0087] The absolute value of the Pearson correlation coefficient between the normal motion fluctuation amplitude sequence of the dth posture feature dimension of the i-th key joint in the t-th frame user lower limb training image and the comparative motion fluctuation amplitude sequence of the dth posture feature dimension of the i-th key joint in the t-th frame user lower limb training image is denoted as the motion completion factor of the dth posture feature dimension of the i-th key joint.

[0088] The product of the action completion factor of the dth pose feature dimension of the i-th key joint and the sensitivity factor of the i-th key joint is denoted as the compensatory anomaly of the dth pose feature dimension of the i-th key joint.

[0089] The normalized value of the product between the compensatory anomaly of the dth pose feature dimension of the i-th key joint and the preset static weight w is used as the dance compensatory sensitivity of the dth pose feature dimension of the i-th key joint in the t-th frame of the user's lower limb training image.

[0090] The specific formula is as follows:

[0091]

[0092] In the formula, The dance-time compensation sensitivity of the dth pose feature dimension of the i-th key joint in the t-th frame of the user's lower limb training image is represented. This represents the preset static weight of the i-th key joint; This represents the sensitivity factor of the i-th key joint in the t-th frame of the user's lower limb training image; This represents the compensatory anomaly of the d-th pose feature dimension of the i-th key joint in the t-th frame of the user's lower limb training image; This represents the linear normalization function.

[0093] It should be noted that the compensatory sensitivity is directly proportional to the preset static weight of each posture feature dimension. That is, in a specific action, the most critical posture feature dimension will be used as the sensitivity factor. Since a posture feature dimension that is highly positively correlated with the action target under normal circumstances will have a more significant compensatory indication once it fluctuates drastically, the compensatory anomaly of each posture feature dimension is inversely proportional to the compensatory sensitivity, aiming to amplify the sensitivity of features that deviate significantly from the normal pattern. At the same time, the closer the critical joint is to the core target joint, the higher its compensatory sensitivity. Therefore, the sensitivity factor and compensatory sensitivity are related. The Pearson correlation coefficient is existing technology and will not be described in detail here.

[0094] Please see Figure 4 The paper presents a heatmap of the compensatory sensitivity of the left knee joint angle in a lower limb coordination and expression training method based on posture monitoring. The heatmap shows the compensatory sensitivity of the posture feature dimensions of all key joints. The paper should intuitively show that the area corresponding to the knee joint has the highest compensatory sensitivity, while the compensatory sensitivity of areas such as the ankle joint is lower.

[0095] At this point, the local feature variability and compensatory sensitivity of each posture feature dimension of each key joint are obtained.

[0096] Step S003: Based on the compensatory sensitivity and local feature fluctuation of the dance, obtain the dynamic feature weighting factor of each posture feature dimension of each key joint during the dance; the dynamic feature weighting factor during the dance is used to characterize the compensatory significance of each posture feature dimension of each key joint at the current action moment; the original feature matrix is ​​enhanced using the dynamic feature weighting factor during the dance to obtain the enhanced feature matrix.

[0097] It should be noted that when a certain posture feature dimension of a key joint not only exhibits severe local feature fluctuations in its current action phase, but also has high compensatory sensitivity (i.e., the posture feature dimension is highly critical to the compensatory behavior), the final dynamic feature weighting factor value will increase significantly; this will ensure that the "key posture feature dimensions" of those "key moments" and "key joints" can be highlighted and enhanced in real time.

[0098] Preferably, in one embodiment of the present invention, the specific method for obtaining the dynamic feature weighting factor of each posture feature dimension of each key joint during dance based on the dance-time compensation sensitivity and local feature fluctuation is as follows:

[0099] In the user's lower limb training image in frame t, the product of the dance-time compensation sensitivity and local feature variability of the dth pose feature dimension of the i-th key joint is denoted as the weighted value of the dth pose feature dimension of the i-th key joint; the product of the mean of the dance-time compensation sensitivity and the mean of the local feature variability of all pose feature dimensions of the i-th key joint is denoted as the feature weighted mean of the i-th key joint.

[0100] The normalized value of the ratio between the weighted value of the dth pose feature dimension of the i-th key joint and the weighted mean value of the i-th key joint is used as the dynamic feature weighting factor of the dth pose feature dimension of the i-th key joint in the dance-time in the t-th frame of the user's lower limb training image.

[0101] The specific formula is as follows:

[0102]

[0103] In the formula, represents the dynamic feature weighting factor of the dth pose feature dimension of the i-th key joint in the t-th frame of the user's lower limb training image during dance; The dance-time compensation sensitivity of the dth pose feature dimension of the i-th key joint in the t-th frame of the user's lower limb training image is represented. This represents the local feature variability of the d-th pose feature dimension of the i-th key joint in the t-th frame of the user's lower limb training image; This represents the mean of the dance-time compensation sensitivity of all pose feature dimensions of the i-th key joint in the t-th frame of the user's lower limb training image; This represents the mean of the local feature variability of all pose feature dimensions of the i-th key joint in the t-th frame of the user's lower limb training image; This represents the hyperbolic tangent function, used for normalization.

[0104] It should be noted that the dynamic feature weighting factor comprehensively considers the local feature volatility and compensatory sensitivity of each posture feature dimension and ensures its value range is stable, so as to serve as the input enhancement term of GCN. Since the dynamic feature weighting factor is positively correlated with the local feature volatility and compensatory sensitivity, the local feature volatility and compensatory sensitivity of lower limb training are multiplied sequentially to construct the dynamic feature weighting factor.

[0105] Preferably, in one embodiment of the present invention, the specific method for obtaining the enhanced feature matrix by weighting the original feature matrix using a dynamic feature weighting factor is as follows:

[0106]

[0107] In the formula, Represents the enhanced feature matrix; Represents the original feature matrix; This represents the dynamic feature weighting factor for the d-th pose feature dimension of the i-th key joint in the t-th frame of the user's lower limb training image during dance.

[0108] Among them, weighting factor This is used to smoothly amplify the feature values, giving them greater attention in the next layer of the GCN input.

[0109] Thus, the enhanced feature matrix is ​​obtained.

[0110] Step S004: Input the enhanced feature matrix into the pre-trained neural network model GCN to perform compensatory recognition and early warning of user posture.

[0111] Preferably, in one embodiment of the present invention, the specific method for inputting the enhanced feature matrix into the pre-trained neural network model GCN for compensatory recognition and early warning of user pose is as follows:

[0112] Enhance the feature matrix The input is fed into the trained neural network ST-GCN for feature extraction and action classification, and the output is the compensation mode and compensation risk score P of the current action.

[0113] A threshold parameter T1 is preset. In this embodiment, T1=0.7 is used as an example. This embodiment does not impose specific limitations. T1 is determined according to the specific implementation.

[0114] If the compensation risk score P is greater than or equal to the threshold parameter T1, an early warning mechanism is immediately triggered, and a targeted correction instruction is generated. For example, if the neural network outputs that the current action has knee valgus compensation and its compensation risk score is 0.85, the user is given real-time feedback through the smart sign, which slows down the speed and abducts the knee.

[0115] The feature extraction and action classification of the ST-GCN neural network, as well as the training process of the ST-GCN neural network, are well-known aspects of neural networks, and will not be elaborated upon in this embodiment.

[0116] Please see Figure 5 The figure shows a comparison of real-time compensatory risk scores for a lower limb coordination and expression training method based on posture monitoring; the figure is a curve comparison of the real-time compensatory risk scores calculated in this embodiment and the prior art (fixed weight) throughout the lower limb training process.

[0117] In the figure, the green dashed line clearly marks the start time of the compensation action (approximately frame 30). For the existing technology curve (fixed weight), the increase in the real-time compensation risk score is relatively gradual before and after the compensation begins, indicating a lack of sensitivity to compensation and delayed or low early warning scores. For the curve of this embodiment (dynamic weight), the real-time compensation risk score rises sharply after the compensation action begins, significantly higher than the existing technology curve, clearly demonstrating the higher sensitivity and earlier early warning capability of this embodiment.

[0118] In summary, after real-time dynamic feature weighting enhancement, this embodiment can identify compensatory behavior earlier and more significantly, overcoming the limitation of traditional GCN models in insufficient sensitivity to subtle compensatory behavior.

[0119] This concludes the embodiment.

[0120] It should be noted that the specific embodiments described above enable those skilled in the art to more fully understand the present invention, but do not limit the present invention in any way. Therefore, although the present invention has been described in detail in this specification, those skilled in the art should understand that modifications or equivalent substitutions can still be made to the present invention; and all technical solutions and improvements that do not depart from the spirit and scope of the present invention are covered within the protection scope of the patent of the present invention. No reference numerals in the claims should be construed as limiting the scope of the claims.

Claims

1. A method for lower limb coordination and expression training based on posture monitoring, characterized in that, The method includes: Acquire the skeletal data of the user's posture during lower limb dance training, and extract the kinematic feature data of key joint movements to construct the original feature matrix; The kinematic feature data of the action includes several posture feature dimensions: absolute position, relative velocity, and joint angle feature values; The position coordinates of the i-th key joint in the t-th frame of the user's lower limb training image are denoted as the absolute position of the i-th key joint; the Euclidean distance between the position coordinates of the i-th key joint in the t-th frame of the user's lower limb training image and the position coordinates of the i-th key joint in the (t-1)-th frame of the user's lower limb training image is denoted as the relative velocity of the i-th key joint; the sine value of the angle between the femur and tibia in the t-th frame of the user's lower limb training image is denoted as the joint angle feature value of the i-th key joint. The original feature matrix includes a time dimension, a joint dimension, and a pose feature dimension; Based on the fluctuations of each posture feature dimension of the key joint within the sliding time window during the dance, the local feature variability of each posture feature dimension of each key joint is obtained, including: A time window parameter 'a' is preset. During the continuous video frames when the user is performing lower limb dance training, the user's lower limb training images that are 'a' most recent before the user's lower limb training image in frame t and the user's lower limb training images that are 'a' most recent after the user's lower limb training image in frame t are combined to form an image sequence, which is denoted as the sliding time window sequence during the dance in frame t. The standard deviation of the dth pose feature dimension of the i-th key joint within the k-th key joint within the sliding time window sequence during the t-th frame of the dance is denoted as the local fluctuation amplitude of the dth pose feature dimension of the i-th key joint within the sliding time window sequence during the k-th frame of the dance. Based on the local fluctuation amplitude, obtain the local fluctuation change rate and enhanced fluctuation amplitude of the dth pose feature dimension of the i-th key joint within the sliding time window sequence during each frame of the dance; The normalized value of the product between the mean of the local fluctuation change rate of the dth pose feature dimension of the i-th key joint in the sliding time window sequence of all frames in the t-th frame of the dance and the enhanced fluctuation amplitude of the dth pose feature dimension of the i-th key joint in the sliding time window sequence of the t-th frame of the dance is used as the local feature fluctuation of the dth pose feature dimension of the i-th key joint in the t-th frame of the user's lower limb training image. Based on the location distribution of key joints and a comparative analysis of the posture feature dimensions of key joints with those of normal users, the dance-time compensation sensitivity of each posture feature dimension of each key joint is obtained, including: Taking the hip joint as the target joint, in the t-th frame of the user's lower limb training image, the normalized value of the Euclidean distance between the position coordinates of the i-th joint and the position coordinates of the target joint is denoted as the sensitivity factor of the i-th joint. By comparing and analyzing the posture features of key joints with those of normal users during dance, the motion completion factor of each posture feature dimension of each key joint is obtained. The product of the action completion factor of the dth pose feature dimension of the i-th key joint and the sensitivity factor of the i-th key joint is denoted as the compensatory anomaly of the dth pose feature dimension of the i-th key joint. A static weight w is preset, and the normalized value of the product between the compensatory anomaly of the dth pose feature dimension of the i-th key joint and the preset static weight w is used as the dance compensatory sensitivity of the dth pose feature dimension of the i-th key joint in the t-th frame of the user's lower limb training image. Based on the compensatory sensitivity and local feature fluctuations during the dance, a dynamic feature weighting factor for each posture feature dimension of each key joint during the dance is obtained; the dynamic feature weighting factor during the dance is used to characterize the compensatory significance of each posture feature dimension of each key joint at the current action moment; the original feature matrix is ​​enhanced using the dynamic feature weighting factor during the dance to obtain an enhanced feature matrix; The enhanced feature matrix is ​​input into the pre-trained neural network model GCN for compensatory recognition and early warning of user posture.

2. The method for lower limb coordination and expression training based on posture monitoring according to claim 1, characterized in that, The specific methods for obtaining the user's posture skeleton data during lower limb dance training and extracting kinematic feature data from key joint movements to construct the original feature matrix are as follows: The system acquires continuous video frames of a user performing lower limb dance training using a depth camera, and extracts the user's posture skeleton data from each frame of the user's lower limb training image using a posture estimation algorithm; the user's posture skeleton data includes the position coordinates of several key joints in each frame of the user's lower limb training image. Based on the position coordinates of key joints, kinematic feature data of each key joint in the action of each key joint in each frame of user lower limb training image are calculated, and the original feature matrix is ​​constructed.

3. The method for lower limb coordination and expression training based on posture monitoring according to claim 2, characterized in that, The method for calculating the kinematic feature data of each key joint in the movement of each frame of the user's lower limb training image based on the position coordinates of key joints and constructing the original feature matrix includes the following specific methods: All frames All key joints and all pose feature dimensions The data is concatenated to construct the original feature matrix. .

4. The lower limb coordination and expression training method based on posture monitoring according to claim 1, characterized in that, The specific method for obtaining the local fluctuation change rate and enhanced fluctuation amplitude of the d-th pose feature dimension of the i-th key joint within the sliding time window sequence of each frame of the dance movement based on the local fluctuation amplitude is as follows: The absolute value of the difference between the local fluctuation amplitude of the dth pose feature dimension of the i-th key joint in the sliding time window sequence during the k-th frame of the dance and the local fluctuation amplitude of the dth pose feature dimension of the i-th key joint in the sliding time window sequence during the (k-1)-th frame of the dance is denoted as the local fluctuation change rate of the dth pose feature dimension of the i-th key joint in the sliding time window sequence during the k-th frame of the dance. The maximum standard deviation of the dth pose feature dimension of the i-th key joint within the sliding time window sequence of all frames during the dance in frame t is denoted as the enhanced fluctuation amplitude of the dth pose feature dimension of the i-th key joint within the sliding time window sequence of the dance in frame t.

5. The lower limb coordination and expression training method based on posture monitoring according to claim 1, characterized in that, The method for obtaining the action completion factor for each posture feature dimension of each key joint by comparing and analyzing the posture features of key joints with those of normal users during dance is as follows: The sequence of all user lower limb training images preceding the t-th frame is denoted as the t-th frame historical reference image sequence. In the video frames of a normal user's lower limb training, the local fluctuation amplitude of the dth posture feature dimension of the i-th key joint in the sliding time window sequence of all frames in the historical reference image sequence of the t-th frame during dancing is used to form a sequence, and the normal movement fluctuation amplitude sequence of the dth posture feature dimension of the i-th key joint in the user's lower limb training image of the t-th frame is used. The sequence is formed by the local fluctuation amplitude of the dth posture feature dimension of the i-th key joint in the sliding time window sequence of all frames in the historical reference image sequence of the t-th frame during the user's lower limb training video frame, and the comparative motion fluctuation amplitude sequence of the dth posture feature dimension of the i-th key joint in the user's lower limb training image of the t-th frame. The absolute value of the Pearson correlation coefficient between the normal motion fluctuation amplitude sequence of the dth posture feature dimension of the i-th key joint in the t-th frame of the user's lower limb training image and the comparative motion fluctuation amplitude sequence of the dth posture feature dimension of the i-th key joint in the t-th frame of the user's lower limb training image is denoted as the motion completion factor of the dth posture feature dimension of the i-th key joint.

6. The lower limb coordination and expression training method based on posture monitoring according to claim 1, characterized in that, The method for obtaining the dynamic feature weighting factor for each posture feature dimension of each key joint during dance, based on the dance-time compensation sensitivity and local feature fluctuation, includes the following specific methods: In the user's lower limb training image in frame t, the product of the dance-time compensation sensitivity and local feature variability of the dth pose feature dimension of the i-th key joint is denoted as the weighted value of the dth pose feature dimension of the i-th key joint; the product of the mean of the dance-time compensation sensitivity and the mean of the local feature variability of all pose feature dimensions of the i-th key joint is denoted as the feature weighted mean of the i-th key joint. The normalized value of the ratio between the weighted value of the dth pose feature dimension of the i-th key joint and the weighted mean value of the features of the i-th key joint is used as the dynamic feature weighting factor of the dth pose feature dimension of the i-th key joint in the dance-time in the t-th frame of the user's lower limb training image.

7. The lower limb coordination and expression training method based on posture monitoring according to claim 1, characterized in that, The process of enhancing the original feature matrix using the dynamic feature weighting factor during the dance to obtain the enhanced feature matrix includes the following specific formula: In the formula, Represents the enhanced feature matrix; Represents the original feature matrix; This represents the dynamic feature weighting factor for the d-th pose feature dimension of the i-th key joint in the t-th frame of the user's lower limb training image during dance.

8. The lower limb coordination and expression training method based on posture monitoring according to claim 1, characterized in that, The method for inputting the enhanced feature matrix into the pre-trained neural network model GCN for user pose compensation recognition and early warning includes the following specific methods: Enhance the feature matrix The input is fed into the trained neural network ST-GCN for feature extraction and action classification, and the output is the compensation mode and compensation risk score P of the current action. A threshold parameter T1 is preset. If the compensation risk score P is greater than or equal to the threshold parameter T1, an early warning mechanism is immediately triggered, and a targeted correction instruction is generated.

Images