Accurate aerobics motion analysis system based on multi-modal data fusion

Abstract

The invention relates to the technical field of motion analysis, in particular to an aerobics motion accurate analysis system based on multi-modal data fusion, which comprises a multi-modal data acquisition module, a data preprocessing module, a multi-modal data fusion module, a motion evaluation module and a feedback module. Acquiring and processing data to obtain an Euler angle, an action time sequence and an acceleration mean value, and splicing a sensor feature vector and an image feature vector to form a joint feature vector; the relative motion trajectory similarity, the relative angle similarity, the motion rhythm error and the trajectory smoothness are calculated through the combination of the feature vector, the Euler angle, the motion time sequence and the acceleration mean value, the motion standard is evaluated according to the relative motion trajectory similarity and the relative angle similarity, the rhythm coordination is evaluated according to the motion rhythm error, and the motion rhythm error is evaluated according to the motion rhythm error. And evaluating action fluency according to the track smoothness, reminding the user in real time, and generating a training report containing action standard, rhythm coordination and action fluency after the aerobics exercise is finished.

Description

Technical Field

[0001] The present invention relates to the technical field of motion analysis, and in particular to a precise aerobics motion analysis system based on multimodal data fusion. Background Art

[0002] Aerobics is a form of exercise that combines gymnastics, dance and music. It not only requires standard movements, but also focuses on rhythmic coordination and smoothness of movements. Movement standardization refers to the degree of matching between the performer's movements and standard movements in spatial trajectory and joint angles, reflecting the consistency of the movements with standard movements in spatial form. Rhythmic coordination refers to the degree of matching between the performer's movement time and standard movement time, reflecting the consistency of the movements with standard movements in time dimension. Movement smoothness refers to the continuity of the trajectory and the smoothness of movement changes during the execution of the movement. The specific representation of movement smoothness is that there is no discontinuity in the continuous movement process. , jamming and a relatively stable rate of change of movements during the movement. Performers need to express artistic expression through a coherent combination of standard movements coordinated with the rhythm of the music. Therefore, in aerobics, movement standardization, rhythm coordination and movement fluency are key indicators for judging the quality of exercise, and are also the core elements for improving artistic expression. For beginners, in the learning process, they need to correct non-standard movements in time, and also pay attention to rhythm coordination and movement fluency. Insufficient rhythm coordination will affect the overall performance effect, and insufficient movement fluency will make the movement transition appear abrupt, affecting the beauty and may also increase the risk of injury.

[0003] With the continuous development of science and technology, the application of technologies such as motion capture, computer vision and deep learning in the field of motion analysis is becoming increasingly mature. Aerobics analysis systems based on these technologies have gradually replaced traditional aerobics learning methods through video learning or manual guidance. However, the aerobics analysis systems in the existing technology mostly focus on the quantitative evaluation of the standardization of movements, and often ignore the evaluation of rhythm coordination and movement fluency. For example, the Chinese patent with the authorization announcement number CN118781658B and the patent name is Aerobics Training Movement Analysis Method, System and Storage Medium, which uses the multi-angle camera of the camera unit to collect movement videos, monitors the heart rate data through the monitoring equipment, and the collected movement videos are processed by the processing unit into sub-movement frames and generate Joint trajectory, then in the comparison unit, the user's joint trajectory is compared with the standard joint trajectory formed by the reference video to determine whether the movement is standard. In addition, the current training intensity is judged by the changes in heart rate data. Finally, the analysis unit obtains the analysis results and suggestions for the user. This solution extracts joint trajectories through visual images to compare with the standard joint trajectory, completing the evaluation of the standardization of the movement. However, in scenes where the user's movement is occluded or the color of the clothing is highly similar to the background, inaccurate trajectory extraction may occur, which may distort the evaluation results. In addition, this solution only compares joint trajectories, and does not introduce comparison in the time dimension and comparison of the smoothness of the movement change rate, and cannot make an effective evaluation of rhythm coordination and movement fluency.

[0004] Another example is the Chinese patent with publication number CN119015676A, and the patent name is "An electronic assistance system for aerobics learning". The sensor modules installed on the user's limbs, head and back collect acceleration and angular velocity data, and then calculate the accumulated value of angle change and motion trajectory, extract the motion trajectory within the interval between two angle zeroing times, compare the extracted motion trajectory with the standard motion trajectory and calculate the trajectory similarity. When the trajectory similarity is higher than the set threshold, the angular displacement value within the interval is further calculated to obtain the angular velocity of the part within the interval, and compare the obtained angular velocity with the standard angular velocity to determine whether the movement is standard. This solution uses a sensor module to capture the user's motion data while performing aerobics in real time, and then uses the trajectory analysis module and the angle analysis module to conduct a detailed analysis of the user's motion trajectory and angular velocity. The standardization of the movement is evaluated through angular velocity integral calculation and trajectory similarity comparison. However, sensor position offset and other situations may occur during aerobics exercise, resulting in deviations in the accumulated value of the angle change, which in turn affects the evaluation results of the movement standardization. In addition, this solution also does not introduce comparisons in the time dimension and the smoothness of the movement change rate, and cannot effectively evaluate rhythm coordination and movement fluency.

[0005] In summary, the existing technology has the problem that the evaluation results of movement standardization are not accurate enough, and the evaluation results are not comprehensive due to the lack of evaluation of rhythm coordination and movement fluency, which may lead to poor learning effects for trainees. Summary of the Invention

[0006] In response to the shortcomings of the existing technology, the present invention aims to provide a precise aerobics analysis system based on multimodal data fusion to solve the problem that the evaluation results of the existing technology on the standardization of movements are not accurate enough, and the evaluation results are not comprehensive due to the lack of evaluation of rhythm coordination and movement fluency, which may lead to poor learning effects for trainees.

[0007] The basic solution provided by the present invention is a precise analysis system for aerobics based on multimodal data fusion, which includes a multimodal data acquisition module, a data preprocessing module, a multimodal data fusion module, a motion evaluation module and a feedback module;

[0008] The multimodal data acquisition module includes an inertial sensor group and a visual acquisition unit; the inertial sensor group is set at the main parts of the human body to collect angular velocity and acceleration data of each part; the visual acquisition unit is used to collect action images;

[0009] The data preprocessing module includes a sensor data processing submodule and a visual image data processing submodule; the sensor data processing submodule is used to calculate the Euler angle based on the collected angular velocity and acceleration data, convert the angular velocity and acceleration data from local coordinates to global coordinates, and calculate the acceleration mean; the visual image data processing submodule is used to align the timestamps of the visual acquisition unit and the inertial sensor group, extract the three-dimensional coordinates of the main parts of the human body in the collected action images based on the OpenPose algorithm and multi-viewpoint fusion technology, calculate the coordinate change of each part and the angle difference between adjacent parts, and construct an action time sequence based on the start time and end time of each action. The specific steps of constructing the action time sequence are as follows:

[0010] S1, the visual acquisition unit acquires movement images of professional aerobics athletes and current practitioners;

[0011] S2. Manually mark the start and end time points of each action in the collected action images of professional aerobics athletes to form a standard action time sequence, and use a dynamic time warping algorithm to mark the start and end time points of each action in the user action images to form a user action time sequence;

[0012] The multimodal data fusion module includes a sensor feature extraction unit, an image feature extraction unit, and a joint feature construction unit; the sensor feature extraction unit is used to extract a sensor feature vector; the image feature extraction unit is used to extract an image feature vector; and the joint feature construction unit is used to splice the sensor feature vector and the image feature vector to form a joint feature vector.

[0013] The motion evaluation module includes a data set establishment unit, a motion standardization evaluation unit, a rhythm coordination evaluation unit, and a motion fluency evaluation unit; the data set establishment unit is used to construct a standard data set and a user data set containing the joint feature vector, Euler angle, motion time series, and acceleration mean; the motion standardization evaluation unit is used to calculate the relative motion trajectory similarity through the joint feature vector, calculate the relative angle similarity through the Euler angle, and evaluate the motion standardization according to the relative motion trajectory similarity and the relative angle similarity; the rhythm coordination evaluation unit is used to calculate the motion rhythm error through the motion time series, and evaluate the rhythm coordination according to the motion rhythm error; the motion fluency evaluation unit is used to calculate the trajectory smoothness through the acceleration mean, and evaluate the motion fluency according to the trajectory smoothness;

[0014] The feedback module is used to provide real-time prompts of relative motion trajectory similarity, relative angle similarity, movement rhythm error and trajectory smoothness, and to generate a training report including movement standardization, rhythm coordination and movement smoothness after the aerobics exercise is completed.

[0015] The principle of the present invention is as follows: the inertial sensor group captures the dynamic data of the main parts of the human body in real time, the visual acquisition unit collects the user's action images in real time, the OpenPose algorithm and multi-viewpoint fusion technology are used to obtain the spatial three-dimensional coordinates of the main parts of the human body in the action images, the angular velocity and acceleration data collected by the inertial sensor group are used to calculate the Euler angle and convert the collected data from local coordinates to global coordinates, the timestamp alignment algorithm is used to establish cross-modal data mapping, the action time series is constructed according to the start time and end time of each action, the sensor feature vector and the image feature vector are extracted and spliced ​​together to form a joint feature vector, and the action evaluation module constructs a standard data containing the joint feature vector, Euler angle, action time series and acceleration mean. The system uses the joint feature vector and the user data set to calculate the relative motion trajectory similarity, the relative angle similarity, the action rhythm error, and the acceleration mean. It also calculates the trajectory smoothness based on the joint feature vector, the relative angle similarity, the action rhythm error, and the acceleration mean. It evaluates the action standardization based on the relative motion trajectory similarity and the relative angle similarity. The greater the motion trajectory similarity and the relative angle similarity, the more standard the action. It evaluates the rhythm coordination based on the action rhythm error. The smaller the action rhythm error, the more coordinated the action. It evaluates the action fluency based on the trajectory smoothness. The smaller the trajectory smoothness, the smoother the action. The feedback module provides real-time prompts for the relative motion trajectory similarity, relative angle similarity, action rhythm error, and trajectory smoothness, thereby achieving quantitative evaluation of action standardization, rhythm coordination, and action fluency and generating a training report.

[0016] The beneficial effects of the present invention are:

[0017] 1. By cross-modally fusing the data collected by the inertial sensor group and the visual acquisition unit to form a multi-dimensional joint feature vector, the joint feature vector is used to calculate the relative motion trajectory similarity, and the Euler angle is used to calculate the relative angle similarity. The movement standardization is evaluated based on the relative motion trajectory similarity and relative angle similarity. The greater the motion trajectory similarity and relative angle similarity, the more standard the movement. Compared with the evaluation method of single modal data, the robustness and accuracy of the movement standardization evaluation can be significantly improved. When the visual acquisition unit causes distortion in the extraction of part coordinates due to occlusion or the high similarity between the clothing color and the background, the real-time dynamic data collected by the inertial sensor group can correct the relative motion trajectory similarity. Similarly, during aerobics exercises, when the sensor position is offset due to large movement amplitude, resulting in deviation in the accumulated value of its angle change, the part trajectory collected by the visual unit can correct the relative motion trajectory similarity, thereby completing an accurate movement standardization evaluation.

[0018] 2. The action time series is introduced into the aerobics motion analysis, and the action rhythm error is calculated through the action time series. The rhythm coordination is evaluated based on the action rhythm error. The smaller the action rhythm error, the more coordinated it is. The acceleration mean is also introduced to judge whether there is discontinuity and jamming in the continuous action process, and whether the action change rate is unstable. The trajectory smoothness is calculated through the acceleration mean data, and the movement fluency is evaluated based on the trajectory smoothness. The smaller the trajectory smoothness, the smoother it is. This fills the gaps in the rhythm coordination and movement fluency of the aerobics motion analysis system in the existing technology, and makes the evaluation of the aerobics motion analysis system change from a single action standard evaluation to a more comprehensive action standard, rhythm coordination and movement fluency evaluation system. When using it, users can not only correct wrong actions in time, but also improve their grasp of rhythm coordination and movement fluency. The action rhythm error can be used to remind users to adjust their own action rhythm, and the trajectory smoothness can be used to remind users to optimize the strength of their own action connection, improve the continuity between actions, achieve a comprehensive improvement in action quality, and effectively improve their own artistic expression of aerobics.

[0019] Furthermore, in the sensor data processing submodule, the specific calculation steps of the Euler angle are:

[0020] S1. Calculate the Euler angle of the collected acceleration data through gravity vector projection and perform low-pass filtering;

[0021] S2, calculating the Euler angle of the collected angular velocity data through gyroscope integration and performing high-pass filtering;

[0022] S3, performing weighted averaging on the Euler angles obtained by low-pass filtering in step S1 and the Euler angles obtained by high-pass filtering in step S2 to obtain final Euler angle data;

[0023] In the sensor data processing submodule, the collected angular velocity and acceleration data are converted from local coordinates to global coordinates through a rotation matrix, and the rotation matrix is ​​constructed by the Euler angles.

[0024] Low-pass filtering suppresses high-frequency vibration noise in the Euler angles calculated from acceleration, while high-pass filtering eliminates low-frequency drift in the Euler angles calculated from the integral of angular velocity. A weighted average of the two is used to obtain the final Euler angles, combining the advantages of both and improving their accuracy. A rotation matrix is ​​constructed based on the Euler angles, converting the collected angular velocity and acceleration data from local coordinates to global coordinates, providing a unified benchmark for subsequent cross-modal data alignment and fusion.

[0025] Furthermore, in the visual image data processing submodule, the time stamps of the visual acquisition unit and the inertial sensor group are aligned using linear interpolation; the coordinate change Δd of each part in adjacent frames is calculated using the following formula: i,k :

[0026]

[0027] Where k represents the frame count, i represents the part count, and Δd i,k Indicates the coordinate change of the i-th part in the k-th frame;

[0028] The angle difference Δθ between adjacent parts is calculated by the following formula i,j,k :

[0029]

[0030] Where k represents the frame count, i and j represent the counts of two adjacent parts, Δθ i,j,k It represents the angular difference between the i-th part and the j-th part in the k-th frame.

[0031] Furthermore, the multimodal data fusion module includes a sensor feature extraction unit, an image feature extraction unit and a joint feature construction unit; the sensor feature extraction unit is used to extract the acceleration mean a μ , angular velocity variance σ ω , forming the sensor feature vector F s =[a μ ,σ ω ]; The image feature extraction unit is used to extract the part coordinate change Δd and the adjacent part angle difference Δθ to form the image feature vector F v =[Δd, Δθ]; a joint feature construction unit, configured to combine the sensor feature vector with the image feature vector to form a joint feature vector F = [F s ,F v ].

[0032] The acceleration mean and angular velocity variance are extracted as sensor feature vectors, and the part coordinate change and the angle difference between adjacent parts are extracted as image feature vectors. The acceleration mean reflects the overall stability during the movement change, and the angular velocity variance represents the degree of movement jitter; the part coordinate change describes the motion trajectory, and the angle difference between adjacent parts reflects the fixed relationship between adjacent parts during the movement change. The two are spliced ​​together to form a joint feature vector. The complementary features enhance the representation ability.

[0033] Furthermore, the action standardization evaluation unit calculates the relative motion trajectory similarity S by using the standard joint feature vector and the user joint feature vector. D , calculate the relative angle similarity S by using the standard Euler angle and the user Euler angle ω :

[0034]

[0035] Where k is the frame count, N is the total number of frames, i is the part count, M is the total number of parts, and Fuser,i,k and F std,i,k denote the user and standard joint feature vectors respectively, ω is the three-dimensional vector [α, β, γ] representing the Euler angle, ω user,i,k and ω std,i,k represent user and standard Euler angles respectively;

[0036] The rhythm coordination evaluation unit is based on the standard action beat time sequence T std =[t1,t2,…,t n ] and user action time series T user =[t1′,t2′,…,t′ N ], the action rhythm error R is calculated using the following formula:

[0037]

[0038] Where i represents the site count and M is the total number of sites;

[0039] The movement fluency evaluation unit is based on the standard acceleration mean a μ,user (t) and the user's average acceleration a μ,std (t) Calculate trajectory smoothness S m :

[0040]

[0041] According to the relative motion trajectory similarity S D and relative angle similarity S θ Evaluate the standardization of the action and the similarity of the relative motion trajectory S D and relative angle similarity S ω The larger the error, the more standard the rhythm coordination is. The smaller the error, the more coordinated the rhythm coordination is. The smaller the error, the more coordinated the rhythm coordination is. m Evaluate movement smoothness, trajectory smoothness S m The smaller, the smoother.

[0042] Furthermore, the feedback module includes a real-time reminder unit and a training report generation unit; the real-time reminder unit is used to generate a training report when the relative motion trajectory similarity S D Or the relative angle similarity S ω Below the preset threshold, or the action rhythm error R or trajectory smoothness S mWhen it is higher than a preset threshold, the deviation information of the specific part is output in real time through voice interaction or a visual interface. The deviation information includes the part name, and one or more of the relative motion trajectory similarity, relative angle similarity, movement rhythm error value and trajectory smoothness; a training report generation unit is used to generate a training report containing movement standardization, rhythm coordination and movement fluency, and provide targeted training suggestions for movement standardization, rhythm coordination and movement fluency.

[0043] By setting preset thresholds, users can be promptly reminded of the parts of the body that need to be adjusted, the movement rhythm, and the movement smoothness issues. Based on the quantified relative motion trajectory similarity, relative angle similarity, movement rhythm error, and trajectory smoothness, targeted training suggestions are generated. For example, when the user's movement rhythm error is large, the user can be advised to perform segmented training with a metronome, and make adjustments based on real-time reminders and new training reports.

[0044] Furthermore, the inertial sensor group is arranged at 11 positions of the user, namely, the head, shoulders, elbows, wrists, knees and ankles.

[0045] In this scheme, the sensor group layout covers the main motion nodes, ensuring the complete monitoring of aerobics movements and avoiding data redundancy caused by setting up more sensors.

[0046] Furthermore, the visual acquisition unit includes three high-definition cameras, located in the front, left front, and right front, for capturing user motion images. The visual image data processing submodule extracts the two-dimensional coordinates of 11 parts, namely the head, shoulders, elbows, wrists, knees, and ankles, based on the OpenPose algorithm, and calculates the three-dimensional coordinates of the parts using the following formula of the dual-camera triangulation method:

[0047]

[0048] Where f is the focal length, B is the camera baseline distance, (u l ,v l ) and (u r ,v r ) represent the pixel coordinates of the left and right cameras respectively, and (u0, v0) represent the coordinates of the image center.

[0049] The arrangement of three cameras can ensure that the extraction of coordinates of each part will not be affected by limb occlusion during movement. The accuracy of coordinate extraction can be improved by using a two-camera triangulation method. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] Figure 1 This is a system module diagram of Example 1 of the aerobics exercise precision analysis system based on multimodal data fusion of the present invention. DETAILED DESCRIPTION

[0051] The following is further described in detail through specific implementation methods:

[0052] Example 1 is basically as shown in the attached Figure 1 As shown:

[0053] The aerobics exercise precision analysis system based on multimodal data fusion includes a multimodal data acquisition module, a data preprocessing module, a multimodal data fusion module, a motion evaluation module, and a feedback module;

[0054] The multimodal data acquisition module includes an inertial sensor group and a visual acquisition unit. The inertial sensor group uses a high-precision six-axis IMU and is installed at 11 parts of the user's head, shoulders, elbows, wrists, knees, and ankles, with a sampling frequency of 100Hz, for collecting angular velocity and acceleration data of each part; the visual acquisition unit uses three high-definition cameras with a frame rate of 60fps, with F placed in the front, 45° to the left, and 45° to the right, respectively, forming a triangular layout for collecting motion images, and the collected motion images are transmitted via synchronization signal lines;

[0055] The data preprocessing module includes a sensor data processing submodule and a visual image data processing submodule;

[0056] The sensor data processing submodule is used to calculate the Euler angles based on the collected angular velocity and acceleration data and construct a rotation matrix using the Euler angles. The angular velocity and acceleration data are converted from local coordinates to global coordinates using the rotation matrix, and then the acceleration mean is calculated. The acceleration mean is the geometric mean of the three-axis acceleration within a sampling period. The accelerator mean is calculated using the following formula:

[0057]

[0058] Where n is the sampling period, respectively a x,k 、a y,k 、a z,k are the x-, y-, and z-axis acceleration values ​​of the k-th sampling respectively;

[0059] Specific calculation steps of Euler angles:

[0060] S1. Calculate the Euler angles of the collected acceleration data by projecting the gravity vector and perform low-pass filtering with a cutoff frequency of 2 Hz.

[0061] S2. Calculate the Euler angles from the collected angular velocity data by integrating the gyroscope and perform high-pass filtering with a cutoff frequency of 2 Hz.

[0062] S3, performing a weighted average of the Euler angles after low-pass filtering in step S1 and the Euler angles after high-pass filtering in step S2, wherein the weight of the Euler angles calculated based on the acceleration data is 0.05, and the weight of the Euler angles calculated based on the angular velocity data is 0.95, and the final Euler angle data is obtained after weighted averaging;

[0063] The visual image data processing submodule is used to align the timestamps of the visual acquisition unit and the inertial sensor group, extract the three-dimensional coordinates of the main parts of the human body in the collected action images based on the OpenPose algorithm and multi-viewpoint fusion technology, calculate the coordinate change of each part and the angular difference between adjacent parts, and construct an action time sequence based on the start and end time of each action;

[0064] The timestamps of the visual acquisition unit and the inertial sensor group are unified to 100 Hz using linear interpolation;

[0065] The OpenPose algorithm extracts the 2D coordinates of 11 body parts (head, shoulders, elbows, wrists, knees, and ankles). The 3D coordinates of these parts are calculated using the following formula using the dual-camera triangulation method:

[0066]

[0067] Where f is the focal length, B is the camera baseline distance, (u l ,v l ) and (u r ,v r ) represent the pixel coordinates of the left and right cameras respectively, and (u0, v0) represent the coordinates of the image center;

[0068] The coordinate change Δd of each part in adjacent frames is calculated by the following formula: i,k :

[0069]

[0070] Where k represents the frame count, i represents the part count, and Δd i,k Indicates the coordinate change of the i-th part in the k-th frame;

[0071] The angle difference Δθ between adjacent parts is calculated by the following formula i,j,k :

[0072]

[0073] Where k represents the frame count, i and j represent the counts of two adjacent parts, Δθ i,j,k represents the angle difference between the i-th part and the j-th part in the k-th frame;

[0074] The specific steps to construct the action time series are as follows:

[0075] S1, the visual acquisition unit acquires movement images of professional aerobics athletes and current practitioners;

[0076] S2. Manually mark the start and end time points of each action in the collected action images of professional aerobics athletes to form a standard action time sequence, and use a dynamic time warping algorithm to mark the start and end time points of each action in the user action images to form a user action time sequence;

[0077] The multimodal data fusion module includes a sensor feature extraction unit, an image feature extraction unit and a joint feature construction unit; the sensor feature extraction unit is used to extract the acceleration mean a in each action cycle. μ , angular velocity variance σ ω , forming the sensor feature vector f s =[a μ ,σ ω ]; The image feature extraction unit is used to extract the part coordinate change Δd and the adjacent part angle difference Δθ to form the image feature vector F v =[Δd, Δθ]; a joint feature construction unit, configured to combine the sensor feature vector with the image feature vector to form a joint feature vector F = [F s ,F v ];

[0078] The action evaluation module includes a data set establishment unit, an action standardization evaluation unit, a rhythm coordination evaluation unit and an action fluency evaluation unit;

[0079] The data set establishment unit is used to establish a standard data set and a user data set including the joint feature vector, Euler angle, action time series and acceleration mean;

[0080] The action standardization evaluation unit calculates the relative motion trajectory similarity S by using the standard joint feature vector and the user joint feature vector. D , calculate the relative angle similarity S by using the standard Euler angle and the user Euler angle ω , the formula is as follows:

[0081]

[0082] Where k is the frame count, N is the total number of frames, i is the part count, M is the total number of parts, and F user,i,k and F std,i,k denote the user and standard joint feature vectors respectively, ω is the three-dimensional vector [α, β, γ] representing the Euler angle, ω user,i,k and ω std,i,k represent user and standard Euler angles respectively;

[0083] The rhythm coordination evaluation unit is based on the standard action beat time sequence T std =[t1,t2,…,t N ] and user action time series T user =[t′1,t′2,…,t′ N ], the action rhythm error R is calculated using the following formula:

[0084]

[0085] Where i represents the site count and M is the total number of sites;

[0086] The movement fluency evaluation unit, according to the standard acceleration mean a μ,user (t) and the user's average acceleration a μ,std (t) Calculate trajectory smoothness S m :

[0087]

[0088] According to the relative motion trajectory similarity S D and relative angle similarity S θ Evaluate the standardization of the action and the similarity of the relative motion trajectory S d and relative angle similarity S ω The larger the error, the more standard the rhythm coordination is. The smaller the error, the more coordinated the rhythm coordination is. The smaller the error, the more coordinated the rhythm coordination is. m Evaluate movement smoothness, trajectory smoothness S m The smaller, the smoother;

[0089] The feedback module includes a real-time reminder unit and a training report generation unit; the real-time reminder unit is used to generate a training report when the relative motion trajectory similarity S D Lower than the preset threshold 0.8 or the relative angle similarity S ω Below the preset threshold of 0.65, or the action rhythm error R is higher than 0.15 or the trajectory smoothness S m When it is higher than the preset threshold of 0.2, the deviation information of the specific part is output in real time through voice interaction or visual interface. The deviation information includes the part name, and one or more of the relative motion trajectory similarity, relative angle similarity, movement rhythm error value and trajectory smoothness; the training report generation unit is used to generate a training report including movement standardization, rhythm coordination and movement fluency, and provide targeted training suggestions for movement standardization, rhythm coordination and movement fluency.

[0090] The above are only embodiments of the present invention. Common knowledge such as the known specific structures and characteristics in the scheme are not described in detail here. Ordinary technicians in the field are aware of all common technical knowledge in the technical field of the invention before the application date or priority date, can obtain all existing technologies in the field, and have the ability to apply conventional experimental means before that date. Ordinary technicians in the field can improve and implement this scheme in combination with their own abilities under the inspiration given by this application. Some typical known structures or known methods should not become obstacles for ordinary technicians in the field to implement this application. It should be pointed out that for those skilled in the art, without departing from the structure of the present invention, several variations and improvements can be made, which should also be regarded as the scope of protection of the present invention. These will not affect the effect of the implementation of the present invention and the practicality of the patent. The scope of protection required by this application shall be based on the content of its claims, and the specific implementation methods and other records in the specification can be used to interpret the content of the claims.

Claims

1. A precise aerobics analysis system based on multimodal data fusion, characterized by: It includes multimodal data acquisition module, data preprocessing module, multimodal data fusion module, action evaluation module and feedback module; The multimodal data acquisition module includes an inertial sensor group and a visual acquisition unit; the inertial sensor group is set at the main parts of the human body to collect angular velocity and acceleration data of each part; the visual acquisition unit is used to collect action images; The data preprocessing module includes a sensor data processing submodule and a visual image data processing submodule; the sensor data processing submodule is used to calculate the Euler angle through the collected angular velocity and acceleration data, convert the angular velocity and acceleration data from local coordinates to global coordinates and calculate the acceleration mean; the visual image data processing submodule is used to align the timestamps of the visual acquisition unit and the inertial sensor group, extract the three-dimensional coordinates of the main parts of the human body in the collected action images based on the OpenPose algorithm and multi-viewpoint fusion technology, calculate the coordinate change of each part and the angle difference between adjacent parts, and construct an action time sequence according to the start time and end time of each action. The specific steps of constructing the action time sequence are as follows: S1, the visual acquisition unit acquires movement images of professional aerobics athletes and current practitioners; S2. Manually mark the start and end time points of each action in the collected action images of professional aerobics athletes to form a standard action time sequence, and use a dynamic time warping algorithm to mark the start and end time points of each action in the user action images to form a user action time sequence; The multimodal data fusion module includes a sensor feature extraction unit, an image feature extraction unit and a joint feature construction unit; The sensor feature extraction unit is used to extract the sensor feature vector; the image feature extraction unit is used to extract the image feature vector; the joint feature construction unit is used to concatenate the sensor feature vector and the image feature vector to form a joint feature vector; The motion evaluation module includes a data set establishment unit, a motion standardization evaluation unit, a rhythm coordination evaluation unit, and a motion fluency evaluation unit; the data set establishment unit is used to construct a standard data set and a user data set containing the joint feature vector, Euler angle, motion time series, and acceleration mean; the motion standardization evaluation unit is used to calculate the relative motion trajectory similarity through the joint feature vector, calculate the relative angle similarity through the Euler angle, and evaluate the motion standardization based on the relative motion trajectory similarity and the relative angle similarity; the rhythm coordination evaluation unit is used to calculate the motion rhythm error through the motion time series, and evaluate the rhythm coordination based on the motion rhythm error; The motion smoothness evaluation unit is used to calculate the trajectory smoothness through the acceleration mean and evaluate the motion smoothness according to the trajectory smoothness; The feedback module is used to provide real-time prompts of relative motion trajectory similarity, relative angle similarity, movement rhythm error and trajectory smoothness, and to generate a training report including movement standardization, rhythm coordination and movement smoothness after the aerobics exercise is completed.

2. The aerobics exercise precision analysis system based on multimodal data fusion according to claim 1 is characterized by: In the sensor data processing submodule, the specific calculation steps of the Euler angle are: S1. Calculate the Euler angle of the collected acceleration data through gravity vector projection and perform low-pass filtering; S2, calculating the Euler angle of the collected angular velocity data through gyroscope integration and performing high-pass filtering; S3, performing weighted averaging on the Euler angles obtained by low-pass filtering in step S1 and the Euler angles obtained by high-pass filtering in step S2 to obtain final Euler angle data; In the sensor data processing submodule, the collected angular velocity and acceleration data are converted from local coordinates to global coordinates through a rotation matrix, and the rotation matrix is ​​constructed by the Euler angles.

3. The aerobics exercise precision analysis system based on multimodal data fusion according to claim 2 is characterized by: In the visual image data processing submodule, a linear interpolation method is used to align the timestamps of the visual acquisition unit and the inertial sensor group; The coordinate change Δd of each part in adjacent frames is calculated by the following formula: i,k : Where k represents the frame count, i represents the part count, and Δd i,k Indicates the coordinate change of the i-th part in the k-th frame; The angle difference Δθ between adjacent parts is calculated by the following formula i,j,k : Where k represents the frame count, i and j represent the counts of two adjacent parts, Δθ i,j,k It represents the angular difference between the i-th part and the j-th part in the k-th frame.

4. The aerobics exercise precision analysis system based on multimodal data fusion according to claim 3 is characterized by: The multimodal data fusion module includes a sensor feature extraction unit, an image feature extraction unit and a joint feature construction unit; The sensor feature extraction unit is used to extract the acceleration mean a μ , angular velocity variance σ ω , forming the sensor feature vector F s =[a μ ,σ ω ]; The image feature extraction unit is used to extract the part coordinate change Δd and the adjacent part angle difference Δθ to form the image feature vector F v =[Δd,Δθ]; a joint feature construction unit, configured to combine the sensor feature vector with the image feature vector to form a joint feature vector F = [F s ,F v ].

5. The aerobics exercise precision analysis system based on multimodal data fusion according to claim 4 is characterized in that: The action standardization evaluation unit calculates the relative motion trajectory similarity S by using the standard joint feature vector and the user joint feature vector. D , calculate the relative angle similarity S by using the standard Euler angle and the user Euler angle ω : Where k is the frame count, N is the total number of frames, i is the part count, M is the total number of parts, and F user,i,k and F std,i,k denote the user and standard joint feature vectors respectively, ω is the three-dimensional vector [α, β, γ] representing the Euler angle, ω user,i,k and ω std,i,k represent user and standard Euler angles respectively; The rhythm coordination evaluation unit is based on the standard action beat time sequence r std =[t1,t2,…,t N ] and user action time series T user =[t′1,t′2,…,t′ N ], the action rhythm error R is calculated using the following formula: Where i represents the site count and M is the total number of sites; The movement fluency evaluation unit is based on the standard acceleration mean a μ,user (t) and the user's average acceleration a μ,std (t) Calculate trajectory smoothness S m : According to the relative motion trajectory similarity S D and relative angle similarity S θ Evaluate the standardization of the action and the similarity of the relative motion trajectory S D and relative angle similarity S ω The larger the error, the more standard the rhythm coordination is. The smaller the error, the more coordinated the rhythm coordination is. The smaller the error, the more coordinated the rhythm coordination is. m Evaluate movement smoothness, trajectory smoothness S m The smaller, the smoother.

6. The aerobics exercise precision analysis system based on multimodal data fusion according to claim 5 is characterized by: The feedback module includes a real-time reminder unit and a training report generation unit; A real-time reminder unit is used to remind the user when the relative motion trajectory similarity S D Or the relative angle similarity S ω Below the preset threshold, or the action rhythm error R or trajectory smoothness S m When it is higher than a preset threshold, the deviation information of the specific part is output in real time through voice interaction or a visual interface. The deviation information includes the part name, and one or more of the relative motion trajectory similarity, relative angle similarity, movement rhythm error value and trajectory smoothness; a training report generation unit is used to generate a training report containing movement standardization, rhythm coordination and movement fluency, and provide targeted training suggestions for movement standardization, rhythm coordination and movement fluency.

7. The aerobics exercise precision analysis system based on multimodal data fusion according to claim 6 is characterized by: The inertial sensor group is arranged on the user's head, shoulders, elbows, wrists, knees and ankles, a total of 11 parts.

8. The aerobics exercise precision analysis system based on multimodal data fusion according to claim 7 is characterized by: The visual acquisition unit includes three high-definition cameras, located in the front, left, and right front, for capturing user motion images. The visual image data processing submodule extracts the two-dimensional coordinates of 11 body parts (head, shoulders, elbows, wrists, knees, and ankles) based on the OpenPose algorithm, and calculates the three-dimensional coordinates of the parts using the following formula of the dual-camera triangulation method: Where f is the focal length, B is the camera baseline distance, (u l ,v l ) and (u r ,v r ) represent the pixel coordinates of the left and right cameras respectively, and (u0, v0) represent the coordinates of the image center.

Images