Motion evaluation method, device and equipment based on multi-modal perception and medium

Abstract

The invention relates to the field of artificial intelligence, can be applied to service system platforms such as medical treatment and health, and discloses a motion evaluation method, device and equipment based on multi-modal perception and a medium, and the method comprises the steps: collecting multi-modal sensing data through different types of sensors in the motion process of a user; performing data fusion processing on the multi-modal sensing data to generate fused sensing data; key point identification is carried out according to the multi-modal sensing data, and action posture information of the user is acquired according to a key point identification result; inputting the fused sensing data and the motion posture information into a motion evaluation model, evaluating the motion completion quality of the user, and outputting a motion evaluation result; and according to the motion evaluation result, displaying corresponding motion guidance information in the interactive interface through the digital human image. Through multi-modal action perception and intelligent action evaluation, the action of the user is accurately monitored and evaluated in real time, targeted guidance is provided, the user is helped to accurately master action essentials, and the teaching quality is improved.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a motion assessment method, device, equipment and medium based on multimodal perception. Background Art

[0002] With the rapid development of technology, people's lifestyles have undergone significant changes. The widespread use of convenient tools, such as cars, elevators, and smart home devices, has greatly improved life efficiency, but it has also reduced people's daily activities to a certain extent. Modern people's increasing reliance on these tools has led to a decrease in physical activity, which in turn has caused a variety of health problems. This change in lifestyle has led to a growing number of people suffering from sub-health. Sub-health refers to a state of being on the edge between health and disease. Long-term sub-health can further lead to chronic diseases.

[0003] To address this health crisis, more and more people are beginning to prioritize fitness. For example, in recent years, the integration of fitness and wellness has gained increasing attention. People are pursuing not only physical fitness but also physical and mental harmony and well-being. Against this backdrop, the integration of traditional fitness methods with modern wellness concepts is gaining popularity. Ba Duan Jin, a traditional Chinese Qigong practice, is gaining popularity due to its ease of learning, gentle movements, and minimal space requirements.

[0004] However, currently, the teaching of various fitness exercises such as Ba Duan Jin, yoga, Pilates, etc. mostly relies on video or text tutorials. It is difficult for users to accurately grasp the details and key points of the movements during exercise, and they are prone to problems such as movement deviation and irregular movements. This not only affects the fitness effect, but may also cause sports injuries, further weakening the user's fitness enthusiasm, and making it difficult to achieve intelligent and accurate personalized exercise guidance. Summary of the Invention

[0005] In view of the above-mentioned deficiencies in the prior art, the purpose of the present invention is to provide a motion assessment method, device, equipment and medium based on multimodal perception that can be applied to the medical health field or other related fields. Its main purpose is to achieve real-time and accurate assessment during the motion teaching process and realize intelligent and accurate personalized motion guidance.

[0006] The technical solutions of the present invention are as follows:

[0007] A first aspect of the present invention provides a motion assessment method based on multimodal perception, comprising:

[0008] Different types of sensors are used to collect corresponding data during the user's movement to obtain multimodal sensing data;

[0009] Performing data fusion processing on the multimodal sensor data to generate fused perception data;

[0010] Performing key point recognition based on the multimodal sensor data, and obtaining user's motion posture information based on the key point recognition results;

[0011] Inputting the fused perception data and the motion posture information into a pre-trained motion evaluation model, evaluating the quality of the user's motion completion, and outputting corresponding motion evaluation results;

[0012] According to the motion evaluation results, corresponding motion guidance information is displayed through a digital human image on the interactive interface.

[0013] A second aspect of the present invention provides a motion assessment device based on multimodal perception, comprising:

[0014] The acquisition module is used to collect corresponding data during the user's movement through different types of sensors to obtain multimodal sensor data;

[0015] Data fusion processing is used to perform data fusion processing on the multimodal sensor data to generate fused perception data;

[0016] A gesture recognition module, configured to perform key point recognition based on the multimodal sensor data and obtain user's motion gesture information based on the key point recognition results;

[0017] An action evaluation module is used to input the fused perception data and the action posture information into a pre-trained action evaluation model, evaluate the quality of the user's action completion, and output a corresponding motion evaluation result;

[0018] The display module is used to display corresponding action guidance information through a digital human image on the interactive interface according to the movement evaluation result.

[0019] A third aspect of the present invention provides a computer device comprising at least one processor; and

[0020] a memory communicatively connected to the at least one processor; wherein,

[0021] The memory stores instructions that can be executed by the at least one processor. The instructions are executed by the at least one processor to enable the at least one processor to perform the above-mentioned motion assessment method based on multimodal perception.

[0022] A fourth aspect of the present invention provides a non-volatile computer-readable storage medium, which stores computer-executable instructions. When the computer-executable instructions are executed by one or more processors, the one or more processors can execute the above-mentioned motion assessment method based on multimodal perception.

[0023] Beneficial effects: The present invention discloses a motion evaluation method, device, equipment and medium based on multimodal perception. Compared with the existing technology, the embodiment of the present invention collects corresponding data during the user's motion through different types of sensors to obtain multimodal sensing data; performs data fusion processing on the multimodal sensing data to generate fused sensing data; performs key point recognition based on the multimodal sensing data, and obtains the user's motion posture information based on the key point recognition result; inputs the fused sensing data and the motion posture information into a pre-trained motion evaluation model, evaluates the quality of the user's motion completion, and outputs the corresponding motion evaluation result; and displays the corresponding motion guidance information through a digital human image on the interactive interface based on the motion evaluation result. Through multimodal motion perception and intelligent motion evaluation, the user's motion can be accurately monitored and evaluated in real time during the motion teaching process, providing targeted guidance to help users accurately master the key points of the motion and improve the quality of teaching. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] In order to more clearly illustrate the solutions in the present invention, a brief introduction is given below to the drawings required for use in describing the embodiments of the present invention. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0025] Figure 1 A schematic diagram of an application environment for a motion assessment method based on multimodal perception provided by an embodiment of the present invention;

[0026] Figure 2 A flowchart of a motion assessment method based on multimodal perception provided by an embodiment of the present invention;

[0027] Figure 3 A flowchart of step S202 in the motion assessment method based on multimodal perception provided by an embodiment of the present invention;

[0028] Figure 4 A flowchart of step S203 in the motion assessment method based on multimodal perception provided by an embodiment of the present invention;

[0029] Figure 5 A flowchart of step S204 in the motion assessment method based on multimodal perception provided by an embodiment of the present invention;

[0030] Figure 6 A schematic diagram of the functional modules of a motion assessment device based on multimodal perception provided by an embodiment of the present invention;

[0031] Figure 7A schematic diagram of the hardware structure of a computer device provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0032] To make the objectives, technical solutions, and effects of the present invention more clear and distinct, the present invention is further described in detail below. It should be understood that the specific embodiments described herein are merely for the purpose of explaining the present invention and are not intended to limit the present invention. The embodiments of the present invention are described below with reference to the accompanying drawings.

[0033] The motion assessment method based on multimodal perception provided by the embodiment of the present invention can be applied in the following fields: Figure 1 In an application environment, the system includes a first terminal device 101, a second terminal device 102, a third terminal device 103, a network 104, and a server 105. The network 104 is a medium for providing a communication link between the first terminal device 101, the second terminal device 102, the third terminal device 103, and the server 105. The network 104 may include various connection types, such as wired and / or wireless communication links, etc.

[0034] The user may use the first terminal device 101, the second terminal device 102, and the third terminal device 103 to interact with the server 105 via the network 104 to receive or send messages, etc. Various communication client applications may be installed on the first terminal device 101, the second terminal device 102, and the third terminal device 103, such as knowledge reading applications, web browser applications, search applications, instant messaging tools, email clients, and / or social platform software (for example only).

[0035] The first terminal device 101 , the second terminal device 102 , and the third terminal device 103 may be various electronic devices having display screens and supporting web browsing, including but not limited to smart phones, tablet computers, laptop computers, desktop computers, and the like.

[0036] The server 105 may be a server that provides various services, such as a backend server that provides support for the content browsed by the user using the first terminal device 101, the second terminal device 102, and the third terminal device 103 (for example only). The backend server may analyze and process the received user requests and other data, and feed back the processing results (such as web pages, information, or data obtained or generated according to the user request) to the terminal device. The server 105 may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services ("Virtual Private Server", or "VPS" for short). The server 105 may also be a server for a distributed system, or a server combined with a blockchain.

[0037] It should be noted that the motion assessment method based on multimodal perception provided in the embodiments of the present application can generally be executed by the first terminal device 101, the second terminal device 102, or the third terminal device 103. Accordingly, the motion assessment apparatus based on multimodal perception provided in the embodiments of the present invention can also be provided in the first terminal device 101, the second terminal device 102, or the third terminal device 103. Alternatively, the motion assessment method based on multimodal perception provided in the embodiments of the present invention can generally be executed by the server 105. Accordingly, the motion assessment apparatus based on multimodal perception provided in the embodiments of the present invention can generally be provided in the server 105.

[0038] It should be understood that the numbers of the above terminal devices, networks and servers are merely illustrative and any number of terminal devices, networks and servers may be provided as required.

[0039] like Figure 2 As shown, the motion assessment method based on multimodal perception provided by the embodiment of the present invention specifically includes the following steps:

[0040] S201 : Collect corresponding data during the user's movement through different types of sensors to obtain multimodal sensing data.

[0041] In this embodiment, when a user is exercising, such as practicing Ba Duan Jin, yoga, or Pilates, multiple types of sensors are used to collect data related to the user's exercise. Different types of sensors are installed in designated positions to collect the user's motion data during exercise in all directions, thereby obtaining multimodal sensor data.

[0042] Specific sensors include inertial measurement units (IMUs), depth cameras, and pressure sensors. IMUs can measure the acceleration and angular velocity of objects, capturing the body's motion and movement. Specifically, IMUs can be installed on the user's head, shoulders, elbows, wrists, waist, hips, knees, ankles, and other areas. By being installed on the user's limbs or torso, they can capture the body's motion in real time. For example, if installed on the head, it can collect movement information such as rotation and tilt of the head, and help determine whether the position and direction of the practitioner's head is correct during the movement. Taking the movements of Ba Duan Jin as an example, it can accurately monitor movements involving head rotation, such as "Looking back for the five labors and seven injuries"; if installed on the shoulder, for example, one is placed on each shoulder, it can obtain shoulder movement information, and effectively monitor the movement amplitude and angle of the shoulder for movements such as "Hands supporting the sky to regulate the three burners" in Ba Duan Jin that involve lifting and stretching the shoulders; if installed on the elbow, for example, it can help understand the bending and extension of the arm during movement, and plays an important role in judging the degree of standardization of movements such as "To regulate the spleen and stomach, you must lift one arm"; if installed on the wrist, for example, it can monitor wrist rotation, flexion and extension, and other movements, In Ba Duan Jin, collecting details of various hand movements is more critical; installing it on the waist can measure the twisting, bending and other movements of the waist. For moves with larger waist movements such as "shaking your head and tail to relieve heart fire", the movement data of the waist can be accurately obtained; installing it on the hip, for example, one on each left and right hip, is used to monitor the movement of the hip, and assist in judging the coordination of lower limb movements and upper body movements, as well as the force and rotation of the hip during movement conversion; installing it on the knee, for example, arranging it on the left and right knees, can collect information such as the bending angle of the knee. For moves involving lower limb movements such as "clenching fists and glaring to increase energy and strength", it can effectively monitor whether the knee movements are standard; installing it on the ankle, for example, placing it on both ankles, is used to monitor the movement of the ankle, which is helpful for maintaining body balance and judging the details of some lower limb movements.

[0043] Depth cameras can obtain depth information of a scene and capture the three-dimensional outline and movement posture of the human body in real time, making them suitable for monitoring human movement as a whole. Specifically, a depth camera can be installed in front of the user. For example, a depth camera can be installed at an appropriate distance and height directly in front of the practitioner so that it can fully capture the practitioner's frontal movements, thereby monitoring the body's overall posture, movement amplitude, and symmetry from a frontal perspective, such as observing the overall situation of movements such as raising hands or squatting. It can also be installed to the side of the user, such as installing one or more depth cameras to the side of the practitioner to obtain body movement information from the side. This is more accurate for judging the angle and amplitude of movements such as forward and backward tilt and twisting of the body, such as side posture monitoring of movements such as "shooting a bird with a bow to the left and right."

[0044] Pressure sensors are primarily used to detect the pressure distribution of the user on the ground during exercise. By measuring the pressure distribution between the body and the ground or support surface, the changes in the body's center of gravity and the stress conditions of various parts of the body can be understood. Pressure sensors can be specifically installed in insoles. For example, multiple pressure sensors are distributed inside the insoles to monitor the pressure distribution of the feet in different stages of movement in real time, and determine whether the force on the feet is uniform during standing, moving, exerting force, etc. This plays an important role in analyzing movements in Ba Duan Jin that require steady standing and balance (such as "Seven Shakes Behind the Back to Eliminate All Diseases"). Alternatively, they can be installed in seats for sitting movements. For example, pressure sensors can be placed on the seat surface to monitor the pressure distribution of the buttocks and legs on the seat, assisting in determining whether the sitting posture is correct and the changes in the body's center of gravity during transitions from sitting posture to sitting posture.

[0045] Data collected by multiple different types of sensors reflects the characteristics of the user's movements from different perspectives, including acceleration, posture, pressure, and muscle activity. This allows for comprehensive and accurate capture of the details of the user's movements, providing a rich data foundation for subsequent motion assessment. This multimodal data collection approach overcomes the limitations of single-sensor data and improves the accuracy and reliability of motion assessment.

[0046] S202: Perform data fusion processing on the multimodal sensor data to generate fused perception data.

[0047] In this embodiment, multimodal data collected from different sensors is fused. Specifically, the data collected by different sensors can be aligned in time to ensure data synchronization. The features of the different sensor data are then extracted and integrated through a fusion algorithm to generate fused perception data. Specifically, fusion algorithms such as Kalman filtering, Bayesian fusion, and deep learning fusion can be used to obtain fused perception data. Data fusion processing can eliminate noise and errors in single sensor data and improve data accuracy and stability. The fused perception data can more comprehensively reflect the multidimensional characteristics of the user's movement, providing more reliable input for subsequent action evaluation.

[0048] S203: Perform key point recognition based on the multimodal sensor data, and obtain user's motion posture information based on the key point recognition result.

[0049] In this embodiment, multimodal sensor data is processed for key point recognition using a deep learning-based key point recognition algorithm. Specifically, the algorithm identifies key skeletal points and joint positions of the human body from image or video data, thereby obtaining information about the position, angle, and trajectory of various body parts during the user's movement, reflecting the user's movement posture and state. Specifically, the modal sensor data can be normalized and filtered to improve data quality. Then, using a deep learning-based algorithm such as OpenPose, the image data collected by the depth camera is analyzed to identify key skeletal points of the user's body, such as the head, shoulders, elbows, wrists, hips, knees, and ankles. Based on the identified key point positions, the angles and positional relationships of various body parts are calculated to obtain the user's movement posture information. For example, the user's movement posture and state can be determined by calculating the angle between the arm and the body and the bending angle of the leg. By identifying key skeletal points and joint positions, the user's movement posture can be analyzed more intuitively, providing a data foundation for subsequent movement evaluation and guidance, and improving the standardization and accuracy of movement judgment.

[0050] S204: Input the fused perception data and the motion posture information into a pre-trained motion evaluation model, evaluate the quality of the user's motion completion, and output a corresponding motion evaluation result.

[0051] In this embodiment, a pre-trained motion evaluation model is used to evaluate the fusion of sensory data and motion posture information. That is, in the training phase, a large amount of standard motion data and the user's actual practice data are first collected, and a machine learning algorithm (such as support vector machine (SVM), random forest, deep learning algorithm, etc.) is used to train the motion evaluation model. The goal of the model is to evaluate the accuracy, standardization, and fluency of the user's motion based on the input multimodal data and motion posture information. The fusion of sensory data and motion posture information is input into the motion evaluation model. The model performs inference calculations based on the input data and outputs an evaluation result of the user's motion completion quality. The specific evaluation results may include the accuracy score, standardization score, fluency score, or grade (such as excellent, good, medium, poor), etc. of the motion, and specific improvement suggestions can also be provided. For example, for the "shaking head and tail to relieve heart heat" motion, the evaluation model can comprehensively evaluate the completion quality of the motion based on the user's body torsion amplitude, head swing trajectory, and breathing coordination. Based on the multimodal data and motion posture information, the user's motion completion quality is comprehensively and accurately evaluated and a specific motion evaluation result is output. The user can intuitively understand their own motion situation, adjust and improve their motion in a timely manner, and improve the exercise effect.

[0052] S205 : Displaying corresponding action guidance information through a digital human image on an interactive interface according to the motion evaluation result.

[0053] In this embodiment, users interact with the motion assessment and guidance system through a visual interactive interface. A digital human avatar is created on this interface through methods such as 3D modeling and motion capture. This virtual human avatar allows for more natural and fluid interaction with the user. Based on the motion assessment results, corresponding motion guidance information is generated and displayed through the digital human avatar. This information can include voice prompts, text instructions, and visual annotations. Specifically, the digital human dynamically adjusts the motion display based on the user's real-time motion performance, providing real-time encouragement and corrective guidance. For example, when the user's motion is correct, the digital human provides positive feedback; when the user's motion deviates, the digital human promptly corrects and provides improvement suggestions. For example, if the user's arm extension angle is insufficient, the digital human can provide a voice prompt, "Arm extension angle is insufficient, please raise it another 5 degrees." Visual annotations are also displayed on the user's motion screen, indicating the correct motion trajectory and deviations from the current motion, helping the user to make timely adjustments. Displaying motion guidance information through the digital human avatar provides intuitive and vivid feedback and corrective suggestions to the user.

[0054] Furthermore, the interactive interface enables diverse interaction methods by receiving voice and user gesture images. This means users can interact naturally with the digital human through voice and gestures. For example, users can ask voice questions, such as "How do I coordinate breathing with this movement?" and the digital human can answer the user's question in real time. Users can also control the teaching process through gestures, such as pausing, replaying, and fast-forwarding. Furthermore, the system supports interactive exercises with the digital human. For example, if users imitate the digital human's movements, the digital human will provide real-time encouragement and guidance, enhancing the user's learning experience and sense of participation.

[0055] In the above embodiment, the present invention discloses a motion assessment method based on multimodal perception. Different types of sensors are used to collect corresponding data during a user's motion to obtain multimodal sensor data. The multimodal sensor data is subjected to data fusion processing to generate fused sensor data. Key points are identified based on the multimodal sensor data, and the user's motion posture information is obtained based on the key point identification results. The fused sensor data and the motion posture information are input into a pre-trained motion assessment model to evaluate the quality of the user's motion completion and output a corresponding motion assessment result. Based on the motion assessment result, corresponding motion guidance information is displayed on an interactive interface through a digital human image. Through multimodal motion perception and intelligent motion assessment, the user's motion can be accurately monitored and evaluated in real time during the motion teaching process, providing targeted guidance to help users accurately master the key points of the motion and improve the quality of teaching.

[0056] In one embodiment, the multimodal sensor data includes IMU data, depth image data, and pressure sensor data, such as Figure 3 As shown, step S202 includes:

[0057] S301, performing data preprocessing on the IMU data, depth image data, and pressure sensor data respectively;

[0058] S302, performing temporal registration and spatial registration on the pre-processed IMU data, depth image data, and pressure sensor data to obtain IMU data, depth image data, and pressure sensor data that are aligned in time and space;

[0059] S303: Perform data fusion processing on the registered IMU data, depth image data, and pressure sensor data through a preset fusion algorithm to generate fused perception data.

[0060] In this embodiment, when data fusion is performed on multiple sensor data, data preprocessing is first performed on the IMU data, depth image data, and pressure sensor data. The IMU data includes data from the accelerometer, gyroscope, and magnetometer. During data preprocessing, the accelerometer and gyroscope data can be first denoised, such as using a Kalman filter or a complementary filter, to eliminate measurement noise and drift. When preprocessing the depth image data, median filtering or bilateral filtering is first used to remove noise from the depth image. Then, background subtraction or a depth threshold-based method is used to segment the user from the background. Finally, image normalization is performed to normalize the pixel values of the depth image to a specific range (e.g., 0 to 1) for ease of subsequent processing. When preprocessing the pressure sensor data, a low-pass filter can be first used to remove high-frequency noise from the pressure sensor data, implementing signal filtering to remove noise, and detecting and removing outliers in the pressure sensor data, such as sudden peaks or negative values. Data preprocessing can effectively remove noise and outliers in the sensor data, improving the quality and stability of the data.

[0061] Because data from different sensors may differ in time and space, data registration is required to align the preprocessed data in time and space. Temporal registration can be performed by synchronizing clocks or timestamps to align data from different sensors in time, ensuring that the data is at the same time point. Alternatively, interpolation can be performed, i.e., for sensor data with different sampling rates, interpolation algorithms (such as linear interpolation or spline interpolation) can be used to align the data to a unified time series. Spatial registration can be performed by converting data from different sensors to the same coordinate system. For example, IMU data and depth camera data can be converted to the world coordinate system for subsequent fusion processing. Data from different sensors can also be spatially aligned through feature point matching or bundle adjustment algorithms. Temporal and spatial registration can eliminate temporal and spatial differences between data from different sensors, ensuring data consistency and accuracy.

[0062] The registered data is then fused using a preset fusion algorithm. Specifically, feature extraction can be performed first, such as extracting statistical features such as the maximum value, mean, and variance of acceleration and angular velocity from the IMU data, extracting features such as edges, contours, and key points from the depth image, and extracting features such as the center of gravity position and pressure change rate of the pressure distribution. Data fusion is performed using methods such as Bayesian fusion and the Extended Kalman Filter (EKF). The Extended Kalman Filter (EKF) is preferably used as the fusion algorithm, as it can handle nonlinear systems and has good performance in multi-sensor fusion. Through data fusion processing, multi-source data is combined to generate more comprehensive and accurate perception data, effectively eliminating the limitations of single sensor data, improving data reliability and availability, and providing more accurate input for subsequent action evaluation.

[0063] In one embodiment, Figure 4 As shown, step S203 includes:

[0064] S401, extracting depth image data from the multimodal sensing data;

[0065] S402: Determine a reference plane of the user's body, and perform key point recognition on the depth image data to obtain positions of several key skeletal points;

[0066] S403, calculating the rotation angle of the head relative to the reference plane based on the positions of the key bone points of the head as the user's head rotation angle;

[0067] S404, constructing an upper body plane and a lower body plane respectively according to the positions of key skeletal points of the body;

[0068] S405: Calculate the angle between the normal vectors of the upper body plane and the lower body plane as the degree of body twisting of the user.

[0069] In this embodiment, when identifying and obtaining the user's motion posture information, the depth image data in the multimodal sensor data is extracted to perform skeletal key point recognition processing, and then the angle of the user's head turning and the degree of body twisting are analyzed as the motion posture information. Among them, when analyzing the angle of the user's head turning, the reference plane of the user's body is first determined. For example, the user's chest plane (determined by the skeletal key points of the shoulders and chest) can be used as the reference plane. Then, based on the key point recognition results, the positions of several key skeletal points are obtained, such as the nose tip point, the center point of the back of the head, the uppermost point (near the head) and the lowermost point (near the shoulder) of the cervical vertebra of the neck, etc. According to the three-dimensional coordinates of the key points, the rotation vector of the head relative to the reference plane is calculated. Vector operations can be used, such as calculating the angle between the vector from the nose tip point to the center point of the back of the head and the normal vector of the reference plane. Then, trigonometric functions (such as the law of cosines) are used to calculate the angle between the two vectors as the user's head turning angle. Alternatively, you can use computer vision libraries (such as OpenCV or Python's NumPy library combined with vector operations) to more accurately calculate the head rotation angle using rotation matrices or quaternions. For example, you can first convert the key point coordinates to homogeneous coordinates, construct a rotation matrix, and then extract the rotation angle information from the rotation matrix.

[0070] Preferably, the calculated angle can be corrected in consideration of the differences in body structure and initial posture of different users, for example, by setting some default standard postures (such as the head angle is 0 when standing facing forward), comparing and adjusting the calculated angle with the standard posture, and standardizing the head turning angle value to a reasonable range.

[0071] When analyzing the degree of twisting of the user's body, the upper body plane and the lower body plane are first constructed based on the positions of the key bone points of the body. That is, the key bone points on the body are selected, such as the left and right acromion points of the shoulders, the center point of the chest, the left and right iliac crest points of the waist (the protruding points on both sides of the pelvis), etc., and the planes of the upper body (shoulders and chest) and the lower body (waist and pelvis) are constructed according to the selected key bone points. The upper body plane can be determined by the key points of the shoulders and chest, and the lower body plane is determined by the key points of the waist. Then, the normal vectors of the upper body plane and the lower body plane are calculated respectively. The normal vector can be obtained by vector cross multiplication, that is, two vectors on the plane are selected, and their cross product is calculated to obtain the normal vector. The angle between the two normal vectors is calculated using vector dot product and trigonometric functions (such as cosine function). This angle reflects the degree of twisting of the upper body relative to the lower body, which is used as the degree of twisting of the user's body.

[0072] Preferably, when analyzing the degree of body twisting, not only static angle calculations can be considered, but also dynamic changes during the movement process should be noted. For example, the rate of change of the angle during the twisting process is observed to see if it meets the specifications of the "look back for injuries" movement. At the same time, the pressure sensor data (if any) is combined to evaluate the stability of the body during the twisting process and determine whether there is any imbalance in the body. For example, a pressure sensor can detect changes in the pressure distribution between the feet or the body and the support surface, thereby assisting in determining the body's state of balance and providing a reliable data basis for subsequent movement evaluation and guidance.

[0073] In one embodiment, Figure 5 As shown, step S204 includes:

[0074] S501, inputting the fused perception data and the motion posture information into a pre-trained motion evaluation model, extracting the user's motion features and breathing features from the fused perception data, and extracting posture features from the motion posture information;

[0075] S502, performing fusion recognition based on the motion features, breathing features, and posture features to obtain the user's current motion pattern;

[0076] S503: Match the user's current motion pattern with a preset standard motion pattern, and obtain a motion evaluation result based on the matching degree.

[0077] In this embodiment, a pre-built and trained motion assessment model comprehensively evaluates the quality of motion completion based on fused sensory data collected by multimodal sensors, combined with information about the user's body torsional amplitude, head swing, and other motion posture information, as well as breathing coordination. Specifically, the multimodal sensory data also includes respiratory data collected using respiratory sensors (such as respiratory belts and piezoelectric sensors), including respiratory rate, respiratory depth, inhalation time, and exhalation time. The fused sensory data (such as acceleration, angular velocity, pressure distribution, and breathing information) and motion posture information (such as key point locations and joint angles) are input into a pre-trained motion assessment model for feature extraction. Motion features related to the user's movements are extracted from the fused sensory data. For example, the trajectory of the user's limb movement, such as the arm swing path and the range of motion of the legs, is calculated; the speed of the user's limb movement, such as the arm swing speed and the body movement speed, is calculated; the acceleration of the user's limb movement, such as the body's take-off acceleration and the arm swing acceleration, is calculated; features related to breathing coordination are extracted from the breathing data, such as the ratio of the breathing cycle to the movement cycle, and the correspondence between the inhalation and exhalation phases and the body twisting movement. Furthermore, features related to the user's posture are extracted from the motion posture information, such as the angles of each joint of the user's body, such as the flexion angle of the knee joint and the rotation angle of the shoulder joint; and the overall body posture of the user is analyzed, such as whether the user maintains balance and is upright. By extracting multiple features from the fusion perception data and motion posture information, it can comprehensively describe the user's motion state and provide rich information for motion quality assessment.

[0078] Based on the extracted motion features, breathing features and posture features, a comprehensive feature vector is formed. Specifically, fusion methods such as simple splicing, weighted fusion, deep learning fusion, etc. can be used, and the present embodiment does not limit this. By analyzing the fused feature vector, the user's current motion pattern is identified. For example, when the motion evaluation model adopts machine learning algorithms such as support vector machine (SVM) and random forest, the feature vector can be classified, and the user's current motion pattern is identified, including action type (such as "two hands supporting the sky to regulate the three burners", "looking back due to five kinds of fatigue and seven kinds of injuries", etc.) and motion state (including action, posture, breathing coordination, etc.). The user's current motion pattern is matched with a preset standard motion pattern, i.e., a series of standard motion patterns are predefined for each training action, and these patterns are obtained by expert demonstration or a large amount of sample data training. Each standard motion pattern includes standard motion features, standard breathing features and standard posture features, etc.

[0079] The user's current motion pattern is matched with a preset standard motion pattern, and the degree of matching is calculated to determine whether the user's movement posture is standard, whether the coordination between movement and breathing is standard, etc. Specifically, the similarity between the user's motion pattern and the standard motion pattern can be calculated as the degree of matching through Euclidean distance, cosine similarity, etc., and the user's motion pattern is mapped to the corresponding motion evaluation result according to the degree of matching. For example, a mapping table of similarity and score or grade is pre-stored, and the user's motion score or motion grade is given from the mapping table based on the similarity obtained by matching. Based on the matching result, specific improvement suggestions are generated, such as "the arm extension angle is insufficient, it is recommended to raise it another 5 degrees", or to guide the user to improve the coordination between breathing and movement, etc., so as to accurately and efficiently evaluate the quality of the user's movement completion through the motion evaluation model, and generate specific scores and improvement suggestions, providing intuitive feedback to the user, helping the user to adjust his breathing and improve his movement in time, and improve the exercise effect.

[0080] In one embodiment, the method further comprises:

[0081] Performing motion anomaly recognition based on the fused perception data and the motion posture information to confirm whether there is any abnormality when the user completes the current action;

[0082] If there is an abnormality, obtain the abnormal action information and the user's historical evaluation results, and generate a decomposition video of the current action based on the abnormal action information and the historical evaluation results;

[0083] The decomposition action in the decomposition video is demonstrated through the digital human image of the interactive interface.

[0084] In this embodiment, during the user's exercise, motion anomalies can be identified by fusing sensory data with the motion posture information. This means that abnormal analysis of the motion data can be used to automatically identify whether the user is having significant difficulty completing the current action. Specifically, this can be done through analysis of motion data anomalies, analysis of action completion time, analysis of the number of action repetitions, and so on. Among them, the abnormal analysis of motion data can specifically detect whether the speed of the user when completing the action is significantly lower than the standard speed, whether the amplitude of the user's action is significantly smaller than the standard amplitude, whether the user's breathing rate is significantly accelerated when completing the action, and whether the user's body is obviously shaking or unbalanced when completing the action. The action completion time analysis can record the time it takes for the user to complete the current action and compare it with the standard action time. If the user takes too long to complete the action, exceeding the normal time range (for example, the standard action completion time is 8-10 seconds, but the user takes more than 15 seconds each time), it may mean that the user encountered difficulties during the execution of the action, resulting in unsmooth action or requiring multiple attempts to complete. The action repetition number analysis can count the number of times the user repeats the current action within a period of practice. If the user frequently repeats the same action, it may be because they are trying to meet the required standards of the action but failed due to difficulties. For example, during a 10-minute practice, the user repeated the "Look Back" action significantly more times than other actions, and the quality of each action was not high. The system can determine that the user has difficulty with this action. Alternatively, in other embodiments, it is also possible to receive user-initiated feedback on abnormal movements, that is, to provide users with feedback channels during or after practice. For example, a feedback button or dialog box is set in the digital human guided teaching system, and users can actively input the difficulties they encounter when completing the "Five Labors and Seven Injuries Look Back" movement, such as "My neck hurts when I turn my head and I can't reach the required angle", "My waist feels very stiff when I twist my body and it is difficult to turn", etc. Users can also provide feedback through voice input, and the system will convert the voice into text for analysis.

[0085] When a user's movement abnormality is confirmed, detailed information related to the abnormality is obtained. For example, abnormal feature values include specific values for movement speed, amplitude, and breathing rate; abnormality types include insufficient movement amplitude, rapid breathing, and unstable posture. The system also obtains the user's historical evaluation results, such as the user's movement scores from previous exercises and previous improvement suggestions. Based on this movement abnormality information and historical evaluation results, adaptive adjustments are made to generate a breakdown video of the current movement that is appropriate for the user's athletic ability. This breakdown video, generated using digital human avatars or animation technology, demonstrates the correct posture and movement trajectory for each key step. For example, a standard movement can be broken down into multiple key steps, each corresponding to a key movement or posture. Based on the user's historical evaluation results, the content and difficulty of the breakdown video are adjusted to better suit the user's current level. By generating breakdown videos tailored to the user's athletic level, personalized and targeted guidance can be provided to the user, helping them gradually overcome difficulties in the movement and improving the efficiency and relevance of exercise guidance.

[0086] In one embodiment, the method further comprises:

[0087] Collecting multi-dimensional information of the user, including basic information, exercise information, action progress information and action feedback information;

[0088] Construct a user profile based on the multi-dimensional information to obtain a corresponding user profile;

[0089] Adaptive matching is performed in a preset action library according to the user portrait, and an exercise teaching plan that matches the user is generated according to the matching result.

[0090] In this embodiment, different exercise teaching plans are adaptively generated based on the characteristics of different users to achieve personalized exercise guidance. First, multi-dimensional information about the user is collected, including basic information such as age, gender, physical fitness, exercise habits, etc., as well as the user's movement data, learning progress, and feedback during exercise. Using big data analysis and machine learning techniques, a user profile is constructed. Through analysis of this user profile, the user's individual characteristics and learning needs are understood, providing a basis for the generation of personalized teaching plans. For example, for older users with less physical flexibility, the teaching plan can appropriately reduce the difficulty and intensity of the movements and increase the segmentation of movements and slow motion demonstrations.

[0091] Adaptive teaching strategies are employed to dynamically adjust teaching content and progress by adaptively matching a pre-set action library based on the user profile. Specifically, a library of various exercise movements is pre-built, each including its name, description, video, difficulty level, target audience, and exercise effect. Based on user profile characteristics (such as ability, preferences, and needs), adaptive matching is performed within the library. For example, rule-based matching based on predefined rules and thresholds is used to select appropriate movements, or machine learning algorithms such as collaborative filtering and decision trees are used to match appropriate movements. Based on the matching results, a user-specific exercise teaching plan is generated. This plan includes an action sequence (i.e., an order of movements based on user ability), as well as a recommended number of repetitions for each movement and a recommended duration for each exercise session. The plan also records user progress and feedback, allowing for dynamic adjustments to the teaching plan. Through adaptive matching and the generation of personalized teaching plans, users can be provided with exercise guidance tailored to their individual characteristics and needs. This personalized teaching plan can improve the user's exercise effectiveness and experience, and enhance their interest and motivation in learning.

[0092] In one embodiment, the method further comprises:

[0093] Obtain the exercise plan currently selected by the user, and retrieve corresponding action data from the preset action library according to the exercise plan;

[0094] Loading the motion data onto the digital human image so that the digital human image displays corresponding standard motions based on the motion data;

[0095] Performing rhythm analysis and force analysis based on the movement data to obtain the rhythm and force of the standard movement;

[0096] Confirming the expression information and breathing information of the digital human image based on a pre-established first mapping relationship between rhythm and force and expression, and a pre-established second mapping relationship between rhythm and force and breathing;

[0097] The expression and breathing state of the digital human image on the interactive interface when performing the standard action are dynamically adjusted according to the expression information and breathing information.

[0098] In this embodiment, 3D modeling, motion capture, and deep learning technologies are used to create a highly realistic teaching digital human. When starting an exercise, the user selects the desired exercise plan through an interactive interface, such as "Ba Duan Jin Basic Course" or "Yoga Relaxation Course." Based on the user's selected exercise plan, the system searches and retrieves the corresponding motion data from a preset motion library. The retrieved motion data is analyzed and converted into a format that the digital human can execute. For example, images from an action video are extracted as skeletal animation data for a 3D model. This data is then applied to the digital human, enabling it to accurately and smoothly demonstrate the corresponding standard movements. This provides users with intuitive and vivid demonstrations of standard movements, helping them better understand and imitate the movements.

[0099] At the same time, the digital human's expression and breathing are adjusted based on the rhythm and strength of the standard movements, making the digital human's guided teaching more vivid and realistic, and enhancing the user's practice experience. Specifically, rhythm analysis and strength analysis are performed based on the movement data to obtain the rhythm and strength of the standard movement. Rhythm analysis extracts the time series information of the movement from the collected movement data. For example, it calculates the duration of each movement phase (such as head turning, body twisting, etc.) to determine the rhythm of the entire movement. It can also judge the speed of the movement rhythm by analyzing the time interval between adjacent key movement points. Strength analysis can analyze indicators such as movement amplitude and movement acceleration to evaluate the strength of the movement. For example, a larger acceleration value and a significant body twist amplitude indicate a stronger movement strength; conversely, a smaller strength.

[0100] Based on the rhythm and strength of the analyzed standard movements, and the pre-established first mapping relationship between rhythm and strength and expression, as well as the second mapping relationship between rhythm and strength and breathing, the digital human's facial expression and breathing information are confirmed. Regarding the mapping of facial expressions, when the movement rhythm is slow and the strength is low, the digital human can present a calm and soothing expression, such as slightly relaxed eyebrows and a natural mouth corner, for example, during the preparation phase or the brief pause after the movement. As the movement rhythm speeds up and the strength increases, the digital human's expression can become more focused and powerful, such as a slight furrowing of the eyebrows, a more focused gaze, and a slightly raised or tightened mouth corner to reflect the strength of the movement. For certain specific movements, such as the rapid head turn and body twist in the "Five Labors and Seven Injuries, Look Back" movement, the digital human can display a momentary expression of exertion, such as widening the eyes and briefly tightening the facial muscles. Regarding the mapping of breathing, when the movement rhythm is slow and the force is low, the digital human's breathing should be steady and slow, with a low breathing frequency, and relatively long and even inhalation and exhalation times. When the movement rhythm is accelerated and the force is increased, the digital human's breathing frequency should increase accordingly, and the breathing depth can also be appropriately increased. For example, during continuous and rapid movements, the digital human can show a relatively rapid breathing state. In combination with the breathing requirements of standard movements (such as the coordination of breathing and movement), breathing adjustments are made at specific movement stages. For example, in "Looking Back to Suffer from Five Fatigues and Seven Injuries", the digital human's breathing should be synchronized with the body's inhalation when twisting and exhalation when turning back to the right position.

[0101] Based on this mapping relationship, combined with the real-time analysis of movement rhythm and force data, the digital human's facial expressions and breathing state are driven to achieve dynamic adjustment of the digital human's information. Digital human expressions are typically achieved through technologies such as facial skeleton animation and texture deformation. This dynamic adjustment of facial expressions can adjust the position and angle of the facial bones, as well as the degree of texture deformation, according to changes in movement rhythm and force, thereby achieving real-time changes in expression. Breathing control can be simulated through chest and abdominal fluctuation animation. Based on the breathing rate and depth determined by the movement rhythm and force, the chest and abdomen models' scaling animation is adjusted to reflect the corresponding breathing state. Audio effects can also be added, with appropriate breathing sound effects selected and played according to changes in breathing rate and depth to enhance the realism of the breathing performance. Dynamically adjusting the digital human's expression and breathing based on the rhythm and force of standard movements provides users with a more realistic and effective guided teaching experience.

[0102] It should be noted that there is not necessarily a certain order between the above steps. A person skilled in the art can understand, based on the description of the embodiments of the present invention, that in different embodiments, the above steps may have different execution orders, that is, they may be executed in parallel, or may be executed interchangeably, etc.

[0103] Further references Figure 6 , as a response to the above Figure 2 The present invention provides an embodiment of a motion evaluation device based on multimodal perception. Figure 2 Corresponding to the method embodiment shown, the device can be specifically applied to various electronic devices.

[0104] like Figure 6 As shown, the motion assessment device 60 based on multimodal perception in this embodiment includes:

[0105] The acquisition module 601 is used to collect corresponding data during the user's exercise through different types of sensors to obtain multi-modal sensor data;

[0106] Data fusion processing 602, for performing data fusion processing on the multimodal sensor data to generate fused perception data;

[0107] A gesture recognition module 603 is configured to perform key point recognition based on the multimodal sensor data and obtain user's motion gesture information based on the key point recognition results;

[0108] The motion evaluation module 604 is configured to input the fused perception data and the motion posture information into a pre-trained motion evaluation model, evaluate the quality of the user's motion completion, and output a corresponding motion evaluation result;

[0109] The display module 605 is used to display corresponding action guidance information through a digital human image on the interactive interface according to the movement evaluation result.

[0110] The module referred to in the present invention refers to a series of computer program instruction segments that can perform specific functions. It is more suitable for describing the execution process of motion assessment based on multimodal perception than a program. For the specific implementation of each module, please refer to the corresponding method embodiment above, which will not be repeated here.

[0111] In one embodiment, the multimodal sensor data includes IMU data, depth image data, and pressure sensor data, and the data fusion processing 602 includes:

[0112] A preprocessing unit, configured to perform data preprocessing on the IMU data, the depth image data, and the pressure sensing data respectively;

[0113] a registration unit, configured to perform temporal and spatial registration on the pre-processed IMU data, depth image data, and pressure sensor data to obtain IMU data, depth image data, and pressure sensor data that are aligned in time and space;

[0114] The fusion unit is used to perform data fusion processing on the aligned IMU data, depth image data and pressure sensor data through a preset fusion algorithm to generate fused perception data.

[0115] In one embodiment, the gesture recognition module 603 includes:

[0116] a data extraction unit, configured to extract depth image data from the multimodal sensing data;

[0117] A key point recognition unit, configured to determine a reference plane of the user's body and perform key point recognition on the depth image data to obtain positions of a plurality of key skeletal points;

[0118] A head rotation angle calculation unit, used to calculate the rotation angle of the head relative to the reference plane according to the positions of the key bone points of the head as the user's head rotation angle;

[0119] A plane construction unit, used to construct an upper body plane and a lower body plane respectively according to the positions of key bone points of the body;

[0120] The body twist calculation unit is used to calculate the angle between the normal vectors of the upper body plane and the lower body plane as the user's body twist degree.

[0121] In one embodiment, the action evaluation module 604 includes:

[0122] a feature extraction unit, configured to input the fused perception data and the motion posture information into a pre-trained motion evaluation model, extract the user's motion features and breathing features from the fused perception data, and extract posture features from the motion posture information;

[0123] a fusion recognition unit, configured to perform fusion recognition based on the motion features, breathing features, and posture features to obtain the user's current motion pattern;

[0124] The matching mapping unit is used to match the user's current motion pattern with a preset standard motion pattern, and obtain a motion evaluation result according to the matching degree.

[0125] In one embodiment, the apparatus 60 further includes:

[0126] An anomaly recognition module, used to identify motion anomalies based on the fused perception data and the motion posture information, and confirm whether there is an anomaly when the user completes the current action;

[0127] An action decomposition module is used to obtain action abnormality information and the user's historical evaluation results if an abnormality exists, and generate a decomposition video of the current action based on the action abnormality information and the historical evaluation results;

[0128] The decomposition display module is used to demonstrate the decomposition action in the decomposition video through the digital human image of the interactive interface.

[0129] In one embodiment, the apparatus 60 further includes:

[0130] An information collection module is used to collect multi-dimensional information of the user, including basic information, movement information, movement progress information and movement feedback information;

[0131] A portrait construction module is used to construct a user portrait based on the multi-dimensional information to obtain a corresponding user portrait;

[0132] The adaptive matching module is used to perform adaptive matching in a preset action library according to the user portrait, and generate an exercise teaching plan that matches the user according to the matching results.

[0133] In one embodiment, the apparatus 60 further includes:

[0134] A data acquisition module is used to obtain the exercise plan currently selected by the user and retrieve corresponding action data from the preset action library according to the exercise plan;

[0135] A data loading module, configured to load the motion data onto the digital human image, so that the digital human image can display corresponding standard motions based on the motion data;

[0136] A data analysis module, configured to perform rhythm analysis and force analysis based on the movement data to obtain the rhythm and force of the standard movement;

[0137] An expression and breathing mapping module, configured to determine the expression information and breathing information of the digital human image based on a pre-established first mapping relationship between rhythm and force and expression, and a pre-established second mapping relationship between rhythm and force and breathing;

[0138] The expression and breathing adjustment module is used to dynamically adjust the expression and breathing state of the digital human image on the interactive interface when performing the standard action according to the expression information and breathing information.

[0139] In the above embodiment, the present invention discloses a motion assessment device based on multimodal perception. Different types of sensors are used to collect corresponding data during a user's motion to obtain multimodal sensor data. The multimodal sensor data is subjected to data fusion processing to generate fused sensor data. Key points are identified based on the multimodal sensor data, and the user's motion posture information is obtained based on the key point identification results. The fused sensor data and the motion posture information are input into a pre-trained motion assessment model to evaluate the quality of the user's motion completion and output a corresponding motion assessment result. Based on the motion assessment result, corresponding motion guidance information is displayed on an interactive interface through a digital human image. Through multimodal motion perception and intelligent motion assessment, the user's motion can be accurately monitored and evaluated in real time during the motion teaching process, providing targeted guidance to help users accurately master the key points of the motion and improve the quality of teaching.

[0140] Another embodiment of the present invention provides a computer device, such as Figure 7 As shown, the computer device 70 includes:

[0141] One or more processors 701 and memory 702, Figure 7 In the description, a processor 701 is used as an example. The processor 701 and the memory 702 can be connected via a bus or other means. Figure 7 The bus connection is taken as an example.

[0142] The processor 701 is used to complete various control logics of the computer device 70. It can be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA), a single-chip microcomputer, an ARM (Acorn RISC Machine) or other programmable logic device, discrete gate or transistor logic, discrete hardware components or any combination of these components. In addition, the processor 701 can also be any traditional processor, microprocessor or state machine. The processor 701 can also be implemented as a combination of computing devices, for example, a combination of a DSP and a microprocessor, multiple microprocessors, one or more microprocessors combined with a DSP and / or any other such configuration.

[0143] Memory 702, as a non-volatile computer-readable storage medium, can be used to store non-volatile software programs, non-volatile computer-executable programs, and modules, such as program instructions corresponding to the multimodal perception-based motion assessment method in the embodiments of the present invention. Processor 701 executes the non-volatile software programs, instructions, and modules stored in memory 702 to execute various functional applications and data processing of computer device 70, thereby implementing the multimodal perception-based motion assessment method in the above-mentioned method embodiments.

[0144] The memory 702 may include a program storage area and a data storage area, wherein the program storage area may store an operating system and application programs required for at least one function; the data storage area may store data created according to the use of the computer device 70, etc. In addition, the memory 702 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage device. In some embodiments, the memory 702 may optionally include a memory remotely located relative to the processor 701, and these remote memories may be connected to the computer device 70 via a network. Examples of the above-mentioned network include but are not limited to the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof. One or more units are stored in the memory 702, and when executed by one or more processors 701, the steps of the motion assessment method based on multimodal perception in any of the above-mentioned method embodiments are executed.

[0145] In the above-mentioned embodiment, the present invention discloses a computer device that collects corresponding data during a user's movement through different types of sensors to obtain multimodal sensor data; performs data fusion processing on the multimodal sensor data to generate fused perception data; performs key point recognition based on the multimodal sensor data, and obtains the user's motion posture information based on the key point recognition results; inputs the fused perception data and the motion posture information into a pre-trained motion evaluation model to evaluate the quality of the user's motion completion and output a corresponding motion evaluation result; and displays corresponding motion guidance information through a digital human image on an interactive interface based on the motion evaluation result. Through multimodal motion perception and intelligent motion evaluation, the user's motion can be accurately monitored and evaluated in real time during the motion teaching process, providing targeted guidance to help users accurately master the key points of the movement and improve the quality of teaching.

[0146] An embodiment of the present invention provides a non-volatile computer-readable storage medium, which stores computer-executable instructions. When the computer-executable instructions are executed by one or more processors, the steps of the motion assessment method based on multimodal perception in any of the above method embodiments are performed.

[0147] In the above-mentioned embodiment, the present invention discloses a non-volatile computer-readable storage medium that collects corresponding data during a user's movement through different types of sensors to obtain multimodal sensory data; performs data fusion processing on the multimodal sensory data to generate fused perception data; performs key point recognition based on the multimodal sensory data, and obtains the user's motion posture information based on the key point recognition results; inputs the fused perception data and the motion posture information into a pre-trained motion evaluation model to evaluate the quality of the user's motion completion and output a corresponding motion evaluation result; and displays corresponding motion guidance information through a digital human image on an interactive interface based on the motion evaluation result. Through multimodal motion perception and intelligent motion evaluation, the user's motion can be accurately monitored and evaluated in real time during the motion teaching process, providing targeted guidance to help users accurately master the key points of the movement and improve the quality of teaching.

[0148] Through the description of the above embodiments, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus the necessary general hardware platform, and of course can also be implemented by hardware, but in many cases the former is a better embodiment. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for enabling a terminal device (which can be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in each embodiment of the present invention.

[0149] The present invention can be used in a wide variety of general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, and the like. The present invention can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present invention can also be practiced in distributed computing environments in which tasks are performed by remote processing devices connected via a communications network. In a distributed computing environment, program modules can be located in local and remote computer storage media, including storage devices.

[0150] In summary, the present invention discloses a motion assessment method, device, equipment, and medium based on multimodal perception, wherein the method includes: collecting corresponding data during the user's motion process through different types of sensors to obtain multimodal sensor data; performing data fusion processing on the multimodal sensor data to generate fused perception data; performing key point recognition based on the multimodal sensor data, and obtaining the user's motion posture information based on the key point recognition results; inputting the fused perception data and the motion posture information into a pre-trained motion assessment model, evaluating the quality of the user's motion completion, and outputting corresponding motion assessment results; and displaying corresponding motion guidance information through a digital human image on an interactive interface based on the motion assessment results. Through multimodal motion perception and intelligent motion assessment, the user's motion can be accurately monitored and evaluated in real time during the motion teaching process, providing targeted guidance, helping users accurately master the key points of the motion, and improving the quality of teaching.

[0151] Of course, those skilled in the art will appreciate that all or part of the processes in the above-described method embodiments can be implemented by instructing related hardware (such as a processor, controller, etc.) through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes in the above-described method embodiments. The storage medium can be a memory, a magnetic disk, a floppy disk, a flash memory, an optical storage device, etc.

[0152] It should be noted that if any software tools or components not developed by our company appear in the examples of this application, they are for illustration purposes only and do not represent actual use. It should be understood that the application of the present invention is not limited to the examples above. Those skilled in the art can make improvements or modifications based on the above description, and all such improvements and modifications shall fall within the scope of protection of the appended claims.

Claims

1. A motion assessment method based on multimodal perception, characterized in that: include: Different types of sensors are used to collect corresponding data during the user's movement to obtain multimodal sensing data; Performing data fusion processing on the multimodal sensor data to generate fused perception data; Performing key point recognition based on the multimodal sensor data, and obtaining user's motion posture information based on the key point recognition results; Inputting the fused perception data and the motion posture information into a pre-trained motion evaluation model, evaluating the quality of the user's motion completion, and outputting corresponding motion evaluation results; According to the motion evaluation results, corresponding motion guidance information is displayed through a digital human image on the interactive interface.

2. The motion assessment method based on multimodal perception according to claim 1, characterized in that: The multimodal sensor data includes IMU data, depth image data, and pressure sensor data. The performing data fusion processing on the multimodal sensor data to generate fused perception data includes: Performing data preprocessing on the IMU data, depth image data, and pressure sensor data respectively; Performing temporal and spatial registration on the pre-processed IMU data, depth image data, and pressure sensor data to obtain IMU data, depth image data, and pressure sensor data that are aligned in time and space; The registered IMU data, depth image data and pressure sensor data are fused using a preset fusion algorithm to generate fused perception data.

3. The motion assessment method based on multimodal perception according to claim 1, characterized in that: The step of identifying key points based on the multimodal sensor data and obtaining user's motion posture information based on the key point identification results includes: Extracting depth image data from the multimodal sensor data; Determining a reference plane of the user's body and performing key point recognition on the depth image data to obtain positions of several key skeletal points; The rotation angle of the head relative to the reference plane is calculated based on the positions of the key bone points of the head as the user's head rotation angle; Construct the upper body plane and lower body plane respectively according to the positions of the key bone points of the body; The angle between the normal vectors of the upper body plane and the lower body plane is calculated as the degree of body twist of the user.

4. The motion assessment method based on multimodal perception according to claim 1, characterized in that: The step of inputting the fused perception data and the motion posture information into a pre-trained motion evaluation model, evaluating the quality of the user's motion completion, and outputting corresponding motion evaluation results includes: Inputting the fused perception data and the motion posture information into a pre-trained motion evaluation model, extracting the user's motion features and breathing features from the fused perception data, and extracting posture features from the motion posture information; The user's current movement mode is obtained by fusing and identifying the movement characteristics, breathing characteristics and posture characteristics; The user's current motion pattern is matched with the preset standard motion pattern, and the motion evaluation result is obtained according to the matching degree.

5. The motion assessment method based on multimodal perception according to claim 1, characterized in that: The method also includes: Performing motion anomaly recognition based on the fused perception data and the motion posture information to confirm whether there is any abnormality when the user completes the current action; If there is an abnormality, obtain the abnormal action information and the user's historical evaluation results, and generate a decomposition video of the current action based on the abnormal action information and the historical evaluation results; The decomposition action in the decomposition video is demonstrated through the digital human image of the interactive interface.

6. The motion assessment method based on multimodal perception according to claim 1, characterized in that: The method also includes: Collecting multi-dimensional information of the user, including basic information, exercise information, action progress information and action feedback information; Construct a user profile based on the multi-dimensional information to obtain a corresponding user profile; Adaptive matching is performed in a preset action library according to the user portrait, and an exercise teaching plan that matches the user is generated according to the matching result.

7. The motion assessment method based on multimodal perception according to claim 1, characterized in that: The method also includes: Obtain the exercise plan currently selected by the user, and retrieve corresponding action data from the preset action library according to the exercise plan; Loading the motion data onto the digital human image so that the digital human image displays corresponding standard motions based on the motion data; Performing rhythm analysis and force analysis based on the movement data to obtain the rhythm and force of the standard movement; Confirming the expression information and breathing information of the digital human image based on a pre-established first mapping relationship between rhythm and force and expression, and a pre-established second mapping relationship between rhythm and force and breathing; The expression and breathing state of the digital human image on the interactive interface when performing the standard action are dynamically adjusted according to the expression information and breathing information.

8. A motion assessment device based on multimodal perception, characterized in that: include: The acquisition module is used to collect corresponding data during the user's movement through different types of sensors to obtain multimodal sensor data; Data fusion processing is used to perform data fusion processing on the multimodal sensor data to generate fused perception data; A gesture recognition module, configured to perform key point recognition based on the multimodal sensor data and obtain user's motion gesture information based on the key point recognition results; An action evaluation module is used to input the fused perception data and the action posture information into a pre-trained action evaluation model, evaluate the quality of the user's action completion, and output a corresponding motion evaluation result; The display module is used to display corresponding action guidance information through a digital human image on the interactive interface according to the movement evaluation result.

9. A computer device, characterized in that: comprising at least one processor; and, a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the motion assessment method based on multimodal perception according to any one of claims 1 to 7.

10. A non-volatile computer-readable storage medium, characterized in that: The non-volatile computer-readable storage medium stores computer-executable instructions, which, when executed by one or more processors, enable the one or more processors to execute the motion assessment method based on multimodal perception according to any one of claims 1 to 7.

Images