Multi-Modal Perception Collaborative Training System and Multi-Modal Perception Collaborative Training Method

Abstract

The present disclosure relates to a multi-modal perception collaborative training system and a multi-modal perception collaborative training method. The multi-modal perception collaborative training system includes a body posture sensor, a sensing module, an operation module, and a display module. The body posture sensor senses body posture data and electromyogram data related to the training actions of a user. The sensing module outputs limb torsion data according to the body posture data and outputs muscle group activation time data according to the electromyogram data. The operation module converts the limb torsion data and the muscle group activation time data into a torque-skeleton coordinate system and a muscle strength eigenvalue-skeleton coordinate system respectively according to a skeleton coordinate system, performs a fusion calculation on the torque-skeleton coordinate system and the muscle strength eigenvalue-skeleton coordinate system, calculates evaluation data for the training actions according to the result of the fusion calculation, and determines that the training action corresponds to one of the known motion actions according to the evaluation data. The display module displays the evaluation data and the known motion actions.

Description

Technical Field

[0001] The present disclosure relates to a training system, and particularly to a multi-modal perception collaborative training system and a multi-modal perception collaborative training method. Background Art

[0002] Currently, the number of people engaging in regular exercise is increasing, and gyms of various sizes can be found almost everywhere. There are various fitness equipment in the gym. After many users arrive at the gym, they simply start using the fitness equipment according to the simple instructions on the equipment without the guidance of a coach. As a result, cases of sports injuries caused by improper use of fitness equipment are endless.

[0003] Or even if there is guidance from a coach, when performing independent training without the coach present, sports injuries may also occur due to incorrect postures, mismatches between the muscle groups used and the training actions, or incorrect force application sequences of the muscle groups in the training actions.

[0004] In addition, when athletes are undergoing sports training, coaches or bystanders can only preliminarily judge whether there is a risk of sports injury from the athletes' movement postures. However, there is no way to quantify the sports performance into an index to provide coaches and athletes with a way to discuss improvement methods. Summary of the Invention

[0005] The multi-modal perception collaborative training system provided by the present disclosure includes a body posture sensor, a sensing module, an operation module, and a display module. The body posture sensor includes an inertial sensor and an electromyography sensor. The inertial sensor is used to sense a plurality of body posture data related to the training actions of the user, and the electromyography sensor is used to sense a plurality of electromyography data related to the training actions of the user. The sensing module is used to output limb torsion data according to the body posture data and output muscle group activation time data according to the electromyography data. The operation module is used to convert the limb torsion data and the muscle group activation time data into torque-skeleton coordinates and muscle strength eigenvalue-skeleton coordinates respectively according to the skeleton coordinates, perform a fusion calculation on the torque-skeleton coordinates and the muscle strength eigenvalue-skeleton coordinates, calculate evaluation data for the training actions according to the results of the fusion calculation, and determine that the training action corresponds to one of a plurality of known sports actions according to the evaluation data. The display module is used to display the evaluation data and the known sports actions.

[0006] The multi-modal perception collaborative training method provided by the present disclosure includes: sensing a plurality of body postures data related to the training actions of the user through the inertial sensor of the body posture sensor, and sensing a plurality of electromyogram data related to the training actions of the user through the electromyogram sensor of the body posture sensor; outputting a plurality of limb torsion data according to the body posture data, and outputting a plurality of muscle group activation time data according to the electromyogram data; converting the limb torsion data into a torque-skeleton coordinate system according to the skeleton coordinate system; converting the muscle group activation time data into a muscle strength eigenvalue-skeleton coordinate system according to the skeleton coordinate system; performing a fusion calculation on the torque-skeleton coordinate system and the muscle strength eigenvalue-skeleton coordinate system; calculating evaluation data for the training actions according to the result of the fusion calculation; determining that the training action corresponds to one of a plurality of known motion actions according to the evaluation data; and displaying the evaluation data and the known motion actions. BRIEF DESCRIPTION OF THE DRAWINGS

[0007] The accompanying drawings are included to provide a further understanding of the present disclosure, and are incorporated into and constitute a part of this specification. The drawings illustrate embodiments of the present disclosure and, together with the embodiments, are used to explain the principles of the present disclosure.

[0008] Figure 1 is an architecture diagram of a multi-modal perception collaborative training system shown according to an embodiment of the present disclosure;

[0009] Figure 2 is a schematic diagram of a fusion calculation performed by an operation module in a multi-modal perception collaborative training system shown according to an embodiment of the present disclosure;

[0010] Figure 3 is a block diagram of a multi-modal perception collaborative training system shown according to an embodiment of the present disclosure;

[0011] Figure 4 is a block diagram of a multi-modal perception collaborative training system shown according to another embodiment of the present disclosure;

[0012] Figure 5 is a block diagram of a multi-modal perception collaborative training system for updating training data and error data shown according to an embodiment of the present disclosure;

[0013] Figure 6 is a schematic diagram of using an inertial sensor to determine the deviation degree in a multi-modal perception collaborative training system shown according to an embodiment of the present disclosure;

[0014] Figure 7 is a block diagram of using an inertial sensor to determine the deviation degree in a multi-modal perception collaborative training system shown according to an embodiment of the present disclosure;

[0015] Figure 8AIt is a schematic diagram of calculating the left - right balance degree by using body posture sensors and mechanical sensors in a multi - mode perception collaborative training system according to an embodiment of the present disclosure;

[0016] Figure 8B It is another schematic diagram of calculating the left - right balance degree by using body posture sensors and mechanical sensors in a multi - mode perception collaborative training system according to an embodiment of the present disclosure;

[0017] Figure 9 It is a block diagram of calculating the left - right balance degree by using body posture sensors and mechanical sensors in a multi - mode perception collaborative training system according to an embodiment of the present disclosure;

[0018] Figure 10 It is a schematic diagram of action recognition by using body inertial sensors and electromyography sensors in a multi - mode perception collaborative training system according to an embodiment of the present disclosure;

[0019] Figure 11 It is a flowchart of a multi - mode perception collaborative training method according to an embodiment of the present disclosure;

[0020] Figure 12 It is a flowchart of calculating evaluation data in a multi - mode perception collaborative training method according to an embodiment of the present disclosure.

[0021] Explanation of reference numerals in the drawings

[0022] 1: Multi - mode perception collaborative training system;

[0023] 10: Body posture sensor;

[0024] 110: Inertial sensor;

[0025] 110a: Body inertial sensor;

[0026] 110b: Equipment inertial sensor;

[0027] 111: Offset sensing unit;

[0028] 120a, 120b, 120c, 120d: Electromyography sensors;

[0029] 151, 152, 153, 154: Sensors;

[0030] 16: Sensing vehicle;

[0031] 2: Multi - mode perception collaborative training method;

[0032] 20: Sensing module;

[0033] 21: Spatial coordinate transformation;

[0034] 22: Dynamic EMG processing;

[0035] 30: Operation module;

[0036] 301: Coordinate system conversion;

[0037] 302, 303: Conversion;

[0038] 304: Fusion calculation;

[0039] 31: Skeleton coordinate system;

[0040] 31a: Human body skeleton coordinate system;

[0041] 31b: Human body / equipment skeleton coordinate system;

[0042] 32: Camera device;

[0043] 361: Another fusion calculation;

[0044] 40: Display module;

[0045] 50: Motion simulation model module;

[0046] 60: Training data database;

[0047] 70: Motion model database;

[0048] 80: Mechanical sensor;

[0049] 90: Physiological information sensor;

[0050] A1, A2: Acceleration;

[0051] d: Relative offset;

[0052] e: Error;

[0053] S310~S380, S321, S330~S331, S340~S341: Steps. Detailed implementation manners

[0054] Some embodiments of the present disclosure will be described in detail with reference to the accompanying drawings hereinafter. For the component symbols cited in the following description, when the same component symbols appear in different drawings, they will be regarded as the same or similar components. These embodiments are only a part of the present disclosure and do not disclose all the implementable manners of the present disclosure.

[0055] Figure 1 It is an architecture diagram of a multi-modal perception collaborative training system 1 shown according to an embodiment of the present disclosure. Please refer to Figure 1, the multi-modal perception collaborative training system 1 includes a body posture sensor 10, a sensing module 20, an operation module 30, and a display module 40. The multi-modal perception collaborative training system 1 senses data related to the user's training actions through the body posture sensor 10. After processing the sensed data, it determines which of the pre-built movement actions in the multi-modal perception collaborative training system 1 the user's current training action belongs to, and displays in real-time the data that can be used as a reference for the user during the training action. The feedback of the multi-modal perception collaborative training system 1 can help the user judge whether the posture during training is correct, whether the main muscle groups used match the training action, and whether the order of muscle group exertion in the training action is correct, etc.

[0056] The body posture sensor 10 includes an inertial sensor 110, an electromyography sensor 120a, and an electromyography sensor 120b. The inertial sensor 110 is used to sense multiple body posture data related to the user's training actions. The inertial sensor 110 can be set on the user's body or limbs according to the user's training actions. For example, when the user is running, the inertial sensor 110 can be set at positions such as the user's waist, outside of the leg, on the shoes, etc. The body posture data related to the running action includes stride frequency, stride length, vertical amplitude, body tilt angle, double-foot contact time, movement trajectory of the feet, etc. These body posture data are all related to the economy during running and can effectively examine the user's posture during running. In practice, the inertial sensor 110 is, for example, a dynamic sensor such as an acceleration sensor (G-Sensor), an angular velocity sensor, a gyro sensor, a stride frequency sensor, etc., but is not limited thereto.

[0057] The electromyography sensor 120a and the electromyography sensor 120b are used to sense multiple electromyography data related to the user's training actions. The electromyography sensor 120a and the electromyography sensor 120b can be attached or worn above the user's core muscle groups or related muscle groups according to the user's training actions, such as the left and right thighs, left and right calves, left and right arms, bilateral back muscles, or bilateral chest muscles, etc., to collect the electromyography data of the muscles. In practice, the sensor for collecting electromyography data can be a contact or non-contact electromyography sensor, which will not be elaborated here.

[0058] Please refer to again Figure 1. The sensing module 20 is coupled to the body posture sensor 10 to output a plurality of limb torsion data according to the body posture data and output a plurality of muscle group activation time data according to the electromyography data. Specifically, after performing spatial coordinate transformation on a plurality of body posture data sensed by the inertial sensor 110, the sensing module 20 outputs a plurality of limb torsion data. In addition, after performing dynamic electromyography (EMG) processing on a plurality of electromyography data sensed by the electromyography sensor 120a and the electromyography sensor 120b, the sensing module 20 outputs a plurality of muscle group activation time data.

[0059] The operation module 30 is coupled to the sensing module 20 to convert the limb torsion data and the muscle group activation time data into a torque-skeleton coordinate and a muscle strength eigenvalue-skeleton coordinate respectively according to the skeleton coordinate system, perform a fusion calculation on the torque-skeleton coordinate and the muscle strength eigenvalue-skeleton coordinate, calculate evaluation data for the training action according to the result of the fusion calculation, and determine that the training action corresponds to one of a plurality of known motion actions according to the evaluation data. The detailed description will be elaborated later. In practice, the operation module 30 may be a central processing unit (CPU), a digital signal processor (DSP), a plurality of microprocessors, one or more microprocessors combined with a digital signal processor core, a controller, a microcontroller, an application specific integrated circuit (ASIC), a field programmable gate array circuit (FPGA), any other type of integrated circuit, a state machine, a processor according to an advanced reduced instruction set machine (ARM), and the like, but is not limited thereto.

[0060] The display module 40 is coupled to the operation module 30 to display the evaluation data and the known motion actions. In one embodiment, the display module 40 can display the evaluation data and the known motion actions in text, icons, and charts. In another embodiment, the display module 40 can further display the evaluation data and the known motion actions with sound, images combined with text, icons, and charts. In practice, the display module 40 can be an electronic device with a display function such as a display, a tablet computer, or a personal computer, or can also be a display device on a treadmill, but is not limited thereto.

[0061] Figure 2 is a schematic diagram of the fusion calculation performed by the operation module 30 in the multi-modal perception collaborative training system 1 shown in an embodiment of the present disclosure. Please refer to Figure 1 、2 The skeletal coordinate system 31 is input into the operation module 30, and a plurality of limb torsion data and a plurality of muscle group activation time data are synchronized and successively input into the operation module 30. The operation module 30 performs a coordinate system conversion 301 on the plurality of limb torsion data according to the skeletal coordinate system 31, converting the plurality of limb torsion data into a force-skeletal coordinate system; then, the operation module 30 performs a conversion 302 to convert the force-skeletal coordinate system into a torque-skeletal coordinate system. At the same time, the operation module 30 performs a coordinate system conversion 301 on the plurality of muscle group activation time data according to the skeletal coordinate system 31, converting the plurality of muscle group activation time data into a muscle strength activation time-skeletal coordinate system; then, the operation module 30 performs a conversion 303 to convert the muscle strength activation time-skeletal coordinate system into a muscle strength eigenvalue-skeletal coordinate system.

[0062] Next, the operation module 30 continuously performs a fusion calculation 304 on the torque-skeletal coordinate system and the muscle strength eigenvalue-skeletal coordinate system, calculates evaluation data for the user's training actions according to the result of the fusion calculation 304, and outputs the evaluation data to the display module 40.

[0063] Specifically, the evaluation data is a kind of data used to quantify the user's exercise effect after the operation module 30 converts and fuses a plurality of limb torsion data and a plurality of muscle group activation time data. In one embodiment, the operation module 30 uses K-Means Clustering (KMC) to perform the fusion calculation 304 on the torque-skeletal coordinate system and the muscle strength eigenvalue-skeletal coordinate system.

[0064] As Figure 1 shown, the operation module 30 determines that the user's training action corresponds to one of a plurality of known exercise actions according to the evaluation data, and outputs the determined known exercise action to the display module 40. The display module 40 displays the evaluation data and the determined known exercise action to provide the user with the ability to observe whether the posture is correct when performing the training action, whether the main muscle groups used match the training action, and whether the muscle group force output sequence of the training action is correct, etc.

[0065] In one embodiment, the user can link with community data through the multi-modal perception collaborative training system 1 described in the present disclosure, and upload the evaluation data and the known exercise action to a community website or a training website. Or the user can interact with other users performing the same known exercise action on the community website or the training website regarding each other's evaluation data. Or, the user can have an online discussion with a remote sports coach based on the evaluation data uploaded by himself / herself to the community website or the training website.

[0066] In one embodiment, if the training action being performed by the user is the same as the known motion action determined by the operation module, it means that the user's body posture data and electromyography data conform to the display that this known motion action should have. Then the user can confirm that the posture of the training action being performed is correct, the main muscle groups used match the training action, and the muscle force output sequence of the training action is correct, etc.

[0067] In another embodiment, if the training action being performed by the user is different from the known motion action determined by the operation module, it may mean that the user's body posture data and electromyography data do not fully conform to the display that this known motion action should have. Then the user can further confirm or adjust the posture of the training action being performed, the main muscle groups used, or further confirm or adjust the muscle force output sequence of the training action being correct, etc. Or, the user can also further confirm whether the inertial sensor 110, the electromyography sensor 120a, and the electromyography sensor 120b are set in the correct positions.

[0068] Figure 3 It is a block diagram of the multi-modal perception collaborative training system 1 shown according to an embodiment of the present disclosure. Please refer to Figure 3 , when the training action performed by the user does not require the use of training equipment, the body posture sensor 10 includes a body inertial sensor 110a, an electromyography sensor 120a, and an electromyography sensor 120b. The body inertial sensor 110a is the same as the inertial sensor 110 disposed on the user's body or limbs as described above, and the electromyography sensor 120a and the electromyography sensor 120b are the same as the electromyography sensors attached or worn above the user's core muscle groups or related muscle groups as described above, which will not be elaborated here.

[0069] In one embodiment, when the training action performed by the user does not require the use of training equipment, the body inertial sensor 110a outputs a plurality of body posture data to the sensing module 20. After the sensing module 20 performs spatial coordinate conversion 21 on the plurality of body posture data sensed by the body inertial sensor 110a, it outputs a plurality of limb torsion data. In addition, after the sensing module 20 performs dynamic EMG processing 22 on the plurality of electromyography data sensed by the electromyography sensor 120a and the electromyography sensor 120b, it outputs a plurality of muscle group activation time data.

[0070] Since only the body inertial sensor 110a is included in the body posture sensor 10, the skeleton coordinate system input to the operation module 30 is the human skeleton coordinate system 31a. A plurality of limb torsion data and a plurality of muscle group activation time data are synchronized and successively input to the operation module 30. The operation module 30 performs conversion and fusion calculations on the plurality of limb torsion data and the plurality of muscle group activation time data according to the human skeleton coordinate system 31a, calculates evaluation data, and outputs the evaluation data to the display module 40.

[0071] In one embodiment, when only the body inertial sensor 110a is disposed on the user's body, the human body skeleton coordinate system 31a corresponds to the user's skeleton, including the bones, muscles, etc. of the human body. The user's skeleton can be obtained by the imaging device 32.

[0072] Figure 4 It is a block diagram of the multi-modal perception collaborative training system 1 shown according to another embodiment of the present disclosure. Please refer to Figure 4 , when the training action performed by the user requires the use of training equipment, the body posture sensor 10 includes a body inertial sensor 110a, a equipment inertial sensor 110b, an electromyography sensor 120a, and an electromyography sensor 120b. The equipment inertial sensor 110b can be disposed on the training equipment used by the user when performing the training action according to the user's training action, such as a baseball bat, a hockey stick, etc., for sensing a plurality of body posture data related to the user's training action. The body inertial sensor 110a is the same as the inertial sensor 110 disposed on the user's body or limbs as described above, and the electromyography sensor 120a and the electromyography sensor 120b are the same as the electromyography sensors attached or worn above the user's core muscle group or related muscle groups as described above, which will not be repeated here.

[0073] In one embodiment, when the training action performed by the user requires the use of training equipment, both the body inertial sensor 110a and the equipment inertial sensor 110b output body posture data to the sensing module 20. After the sensing module 20 performs spatial coordinate conversion 21 on the plurality of body posture data sensed by the body inertial sensor 110a and the equipment inertial sensor 110b, a plurality of limb torque data are output. In addition, after the sensing module 20 performs dynamic EMG processing 22 on the plurality of electromyography data sensed by the electromyography sensor 120a and the electromyography sensor 120b, a plurality of muscle group activation time data are output.

[0074] Since the body posture sensor 10 simultaneously includes the body inertial sensor 110a and the equipment inertial sensor 110b, the skeleton coordinate system input to the operation module 30 is the human body / equipment skeleton coordinate system 31b. The plurality of limb torque data and the plurality of muscle group activation time data are synchronously and successively input to the operation module 30. The operation module 30 performs conversion and fusion calculations on the plurality of limb torque data and the plurality of muscle group activation time data according to the human body / equipment skeleton coordinate system 31b, calculates evaluation data, and outputs the evaluation data to the display module 40.

[0075] In one embodiment, when the body inertial sensor 110a is disposed on the user's body and the equipment inertial sensor 110b is disposed on the training equipment at the same time, the human / equipment skeleton coordinate system 31b corresponds not only to the user's body skeleton, including the bones, muscles, etc. of the human body, but also to the equipment skeleton of the training equipment. The body skeleton of the user and the equipment skeleton of the training equipment can both be acquired by the imaging device 32.

[0076] Figure 5 FIG. is a block diagram of the multi-mode perception collaborative training system 1 for updating training data and error data according to an embodiment of the present disclosure. Please also refer to Figure 1 、 5 FIG.. The body posture sensor 10 includes at least two sensors disposed on the user, and the sensors disposed on the user can be one or a combination of an inertial sensor and an electromyography sensor. As Figure 5 shown in FIG., the body posture sensor 10 includes sensors 151, 152, 153, and 154, where the sensors 151, 152, 153, and 154 can be inertial sensors or electromyography sensors respectively. In other words, the sensors 151, 152, 153, and 154 can all be inertial sensors or all be electromyography sensors, and some of the sensors 151, 152, 153, and 154 can be inertial sensors and the remaining sensors can be electromyography sensors.

[0077] Please continue to refer to Figure 5 FIG.. The body posture sensor 10 outputs a plurality of sensed data to the sensing module 20, where the sensed data corresponds to the data output by the sensors included in the body posture sensor 10 that are inertial sensors or electromyography sensors. The sensing module 20 outputs a plurality of limb torque data and / or a plurality of muscle group activation time data according to the sensed data. The operation module 30 performs conversion and fusion calculations on the plurality of limb torque data and the plurality of muscle group activation time data.

[0078] In one embodiment, the multi-modal perception collaborative training system 1 further includes a motion simulation model module 50, a training data database 60, and a motion model database 70. The motion simulation model module 50 is coupled to the operation module 30. The training data database 60 is coupled to the operation module 30 and the motion simulation model module 50, and includes training data and error data corresponding to a variety of known motion models, where the error data is used to determine whether the user's training action is an incorrect action or a dangerous action. The motion model database 70 is coupled to the motion simulation model module 50 and includes a plurality of known motion models. The plurality of known motion models are motion models pre-established based on four or more inertial sensors or electromyography sensors. In practice, the motion simulation model module 50 can be a micro-processor or an embedded controller, and the training data database 60 and the motion model database 70 can be storage media such as a memory or a hard disk, which is not limited in the present disclosure.

[0079] The operation module 30 determines the user's motion scenario based on the number of body sensors used by the user, such as running, aerobic exercise, core muscle group training without using equipment, etc. After the operation module 30 determines the user's motion scenario, it performs motion model pairing with the motion simulation model module 50 based on the motion scenario. The motion simulation model module 50 selects one of the known motion models from the motion model database 70 based on the motion scenario, aiming to find the motion model corresponding to the training action being performed by the user.

[0080] After the motion simulation model module 50 selects the known motion model corresponding to the training action being performed by the user from the motion model database 70, the motion simulation model module 50 reads the training data corresponding to the selected known motion model from the training data database 60 and transmits it back to the operation module 30. The operation module 30 compares and evaluates the data with the training data corresponding to the selected known motion model to calculate the similarity between the user's training action and the selected known motion model.

[0081] When the similarity is greater than or equal to a similarity threshold (such as 0.5), the operation module 30 determines that the user's training action conforms to the selected known motion model, and stores the evaluation data in the training data database 60 to update the training data corresponding to the selected known motion model and establish an AI model. At the same time, the operation module 30 outputs the evaluation data and the known motion action corresponding to the selected known motion model to the display module 40, and the display module 40 further displays the evaluation data and the known motion action.

[0082] Conversely, when the similarity is less than the similarity threshold (e.g., 0.5), the operation module 30 determines that the user's training action does not conform to all the known motion models included in the motion model database 70. At this time, the operation module 30 stores the evaluation data in the training data database 60 to update the error data. At the same time, the operation module 30 outputs the evaluation data and the error motion action message to the display module 40, and the display module 40 further displays the evaluation data and the error motion action message. Among them, the error motion action message is used to remind the user to further confirm or adjust the posture of the training action being performed, the main muscle groups used, or to further confirm or adjust the correct order of muscle force output of the training action. Or, the user can further confirm whether the body posture sensor 10 is set in the correct position.

[0083] Figure 6 FIG. is a schematic diagram of using an inertial sensor 110 (such as an acceleration sensor) to determine the offset degree in a multi-modal perception collaborative training system according to an embodiment of the present disclosure; Figure 7 FIG. is a block diagram of using an inertial sensor 110 to determine the offset degree in a multi-modal perception collaborative training system according to an embodiment of the present disclosure. Please first refer to Figure 6 , when the inertial sensor 110 is fixed on the sensing vehicle 16, and the sensing vehicle 16 is set on the user's body, limbs or clothing in a way of being strapped or worn, during the movement process, the inertial sensor 110 may be offset due to the user's body shaking or limb swinging. Once a relative offset amount d is generated between the inertial sensor 110 and the user's body or limbs, it will affect the accuracy of the body posture data.

[0084] Please also refer to Figure 6 , 7 . The inertial sensor 110 has an offset sensing unit 111. When the sensing vehicle 16 fixing the inertial sensor 110 fits on the user's body or limbs, the sensing data measured by the inertial sensor 110 is the acceleration A1, and the offset sensing unit 111 sets the acceleration A1 as the standard reference value. When a relative offset amount d is generated between one side of the sensing vehicle 16 fixing the inertial sensor 110 and the user's body or limbs, the sensing data measured by the inertial sensor 110 is the acceleration A2. Once there is an error e between the acceleration A2 and the acceleration A1, the offset sensing unit 111 senses the offset data corresponding to the acceleration A2 and outputs it to the sensing module 20.

[0085] The sensing module 20 outputs limb torsion data to the operation module 30 according to the body posture data and the offset data. The operation module 30 compares the evaluation data with the training data of the corresponding selected known motion model to determine whether the relative offset d between the sensing vehicle 16 of the fixed inertial sensor 110 and the user's body exceeds the offset threshold. When the relative offset d is not greater than the offset threshold, it means that the offset degree of the sensing vehicle 16 is not sufficient to affect the accuracy of the body posture data, and the multi-modal perception collaborative training system 1 will continue to operate. On the contrary, when the relative offset d is greater than the offset threshold, it means that the offset degree of the sensing vehicle 16 has affected the accuracy of the body posture data. The operation module 30 will output an abnormal signal to the display module 40, and the display module 40 will display a sensor setting abnormal message.

[0086] When the user is using the exercise equipment, a mechanical sensor can also be set on the exercise equipment to sense the force output state of the user when performing the training action and detect whether the force output on the left and right sides of the user's body is balanced. Once the force output on the left and right sides of the user's body is unbalanced, the multi-modal perception collaborative training system described in the present disclosure can further issue a warning to remind the user to pay attention to the unbalanced force output on the left and right sides of the body.

[0087] Figure 8A It is a schematic diagram of calculating the left-right balance degree by using the body posture sensor and the mechanical sensor 80 in the multi-modal perception collaborative training system shown according to an embodiment of the present disclosure. As Figure 8A shown, when the user is using the Figure 8A training equipment, the user needs to push the flat plate on the fitness equipment with both feet at the same time. The multi-modal perception collaborative training system further includes a mechanical sensor 80. The mechanical sensor 80 is set on the training equipment and coupled to the sensing module to sense a plurality of mechanical data (such as pressure sensing signals) corresponding to the user's training action. When the user pushes the flat plate on the fitness equipment with both feet at the same time, the mechanical sensor 80 senses a plurality of mechanical data (such as pressure sensing signals) corresponding to the user's training action. These mechanical data can be the total pressure sensing signal corresponding to the user's two feet simultaneously applied on the flat plate, or the pressure sensing signals corresponding to the user's left foot and the user's right foot respectively, depending on the setting method or the number of the mechanical sensors 80.

[0088] Please continue to refer to Figure 8AWhen a user is using sports equipment, the electromyography sensor 120a and the electromyography sensor 120b are respectively disposed on the left and right halves of the user's body, and the equipment inertial sensor 110b is disposed on the sports equipment. The electromyography sensor 120a and the electromyography sensor 120b are used to sense a plurality of left-half electromyography data (such as left thigh muscle signals) and a plurality of right-half electromyography data (such as right thigh muscle signals) corresponding to the user's training actions, while the equipment inertial sensor 110b is used to sense a plurality of postural data (such as acceleration signals) related to the user's training actions.

[0089] Figure 8B is another schematic diagram of calculating the left-right balance using the postural sensor and the mechanical sensor 80 in the multi-modal perception collaborative training system shown according to an embodiment of the present disclosure. As Figure 8B shown, when the user is running, since the running action is that the two feet step on the ground in an alternating manner, that is, only one foot touches the ground at a time, the balance of the force exerted by the two legs will affect the running effect and even the safety during running. Therefore, it is necessary to detect the left-right balance for the user's two legs respectively.

[0090] The electromyography sensor 120a and the electromyography sensor 120c are respectively disposed on the left thigh and the left calf of the user's body, the electromyography sensor 120b and the electromyography sensor 120d are respectively disposed on the right thigh and the right calf of the user's body, and the body inertial sensor 110a is disposed on the rear side of the user's waist. The electromyography sensor 120a and the electromyography sensor 120c are used to sense a plurality of left-half electromyography data (such as left thigh and calf muscle signals) corresponding to the user's training actions, the electromyography sensor 120b and the electromyography sensor 120d are used to sense a plurality of right-half electromyography data (such as right thigh and calf muscle signals) corresponding to the user's training actions, and the body inertial sensor 110a is used to sense a plurality of postural data (such as stride length, stride frequency, vertical amplitude, body tilt angle, double-foot contact time, movement trajectory of the two feet, etc., attitude amplitude changes) related to the user's training actions.

[0091] As Figure 8B shown, when the user is using Figure 8BWhen using the training equipment (i.e., treadmill) herein, the user's feet step on the treadmill belt in an alternating manner. The mechanical sensor 80 is disposed on the treadmill belt and coupled to the sensing module to sense a plurality of mechanical data (such as pressure sensing signals) corresponding to the user's training actions. Since the user does not step on the treadmill belt with both feet simultaneously when running, the mechanical sensor 80 can sense a plurality of mechanical data (such as step frequency pressure sensing signals) corresponding to the user's training actions, especially these mechanical data respectively correspond to the step frequency pressure sensing signal of the user's left foot and the step frequency pressure sensing signal of the user's right foot.

[0092] Figure 9 FIG. is a block diagram of calculating the left-right balance degree by using a body posture sensor and a mechanical sensor 80 in a multi-modal perception collaborative training system according to an embodiment of the present disclosure. The sensing module 20 outputs a plurality of pressure data according to a plurality of mechanical data, and outputs a plurality of limb torsion data according to a plurality of body posture data. At the same time, the sensing module 20 respectively outputs a plurality of left half muscle group activation time data and a plurality of right half muscle group activation time data according to a plurality of left half electromyogram data and a plurality of right half electromyogram data. The operation module 30 calculates the left half output value according to a plurality of pressure data, a plurality of limb torsion data and a plurality of left half muscle group activation time data, and calculates the right half output value according to a plurality of pressure data, a plurality of limb torsion data and a plurality of right half muscle group activation time data. Then, the operation module 30 performs another fusion calculation 361 on the left half output data and the right half output data, and calculates the left-right balance degree corresponding to the user's training action according to the result of the another fusion calculation 361.

[0093] When the left-right balance degree is less than or equal to the balance degree threshold, the operation module 30 determines that the left and right half outputs of the user's body are balanced, and continuously calculates the left-right balance degree corresponding to the user's training action according to the result of the another fusion calculation. On the contrary, when the left-right balance degree is greater than the balance degree threshold, the operation module 30 determines that the left and right half outputs of the user's body are unbalanced, and the display module 40 displays the evaluation data and the unbalance message to remind the user to pay attention to the unbalanced condition of the left and right side outputs of the body.

[0094] As Figure 1As shown, in one embodiment, the multi-modal perception collaborative training system 1 further includes a physiological information sensor 90, which is coupled to the sensing module 20 and used to sense multiple physiological data of the user, such as information data of body temperature, heart rate, respiration, skin moisture content, sweat, etc. of the user when performing training actions, and transmit these physiological data to the computing module 30. The computing module 30 can monitor the physiological condition of the user when performing training actions based on these physiological data and determine whether to issue a warning signal to remind the user to stop the training action. In practice, the physiological information sensor 90 can be a sensor that extracts the above physiological values in a contact or non-contact manner, and the present disclosure does not limit it.

[0095] Figure 10 FIG. is a schematic diagram of using the body inertial sensor 110a and the electromyogram sensor 120a for action recognition in a multi-modal perception collaborative training system according to an embodiment of the present disclosure. Please refer to Figure 10 , in one embodiment, the user can use the multi-modal perception collaborative training system for action recognition. The body inertial sensor 110a is arranged on the glove, and the electromyogram sensor 120a is arranged on the wristband.

[0096] When the user wants to perform the dart-throwing action, the glove equipped with the body inertial sensor 110a can be put on the hand of the user for dart-throwing, and the wristband equipped with the electromyogram sensor 120a can be fixed on the wrist of the user for dart-throwing. The body inertial sensor 110a is used to sense multiple postural data related to the hand movement of the user when throwing darts. The electromyogram sensor 120a is used to sense multiple electromyogram data related to the user when throwing darts.

[0097] In addition, the imaging device 32 is used to obtain the posture and body image of the user when throwing darts. When the user throws darts, the hand of the user for dart-throwing will first be lifted, stretched backward, and then thrown forward the dart. Therefore, the posture and body image of the user when throwing darts includes the movement trajectory of the user's hand. In addition, when the user throws darts, the user's body may also use the power of body rotation to throw the dart, so the posture and body image of the user when throwing darts also includes the rotation trajectory of the user's body.

[0098] Next, please refer to Figure 3 、 10 at the same time. The body inertial sensor 110a will output postural data to the sensing module 20. After the sensing module 20 performs spatial coordinate conversion 21 based on multiple postural data sensed by the body inertial sensor 110a, it outputs multiple limb torque data. In addition, after the sensing module 20 performs dynamic EMG processing 22 based on multiple electromyogram data sensed by the electromyogram sensor 120a, it outputs multiple muscle group activation time data.

[0099] The body posture sensor 10 includes both a body inertial sensor 110a and an electromyography sensor 120a. Therefore, the skeleton coordinate system input to the operation module 30 is the human body skeleton coordinate system 31a. A plurality of limb torsion data and a plurality of muscle group activation time data are synchronized and successively input to the operation module 30. The operation module 30 converts and fuses and calculates the plurality of limb torsion data and the plurality of muscle group activation time data according to the human body skeleton coordinate system 31a, calculates evaluation data, and outputs the evaluation data to the display module 40.

[0100] In addition, in one embodiment, the operation module 30 can output the posture image of the user when throwing a dart to the display module 40 (such as a mobile device). The user can view the posture image through the display module 40, and can even view the movement trajectory of the hand and the rotation trajectory of the body by slowing down the posture image. The operation module 30 can also combine with an artificial intelligence AI motion analysis module to perform motion analysis on the posture image of the user when throwing a dart, combine the evaluation data to quantify the exercise effect, and provide suggestions for the user to adjust the hand movement and body rotation.

[0101] Figure 11 It is a flowchart of a multi-modal perception collaborative training method 2 illustrated according to an embodiment of the present disclosure. As Figure 11 shown, the multi-modal perception collaborative training method 2 includes steps S310 to S380.

[0102] In step S310, a plurality of body posture data related to the training action of the user are sensed by at least one inertial sensor of a plurality of body posture sensors, and a plurality of electromyography data related to the training action of the user are sensed by at least one electromyography sensor of these body posture sensors. In step S320, a plurality of limb torsion data are output according to these body posture data, and a plurality of muscle group activation time data are output according to these electromyography data.

[0103] In step S330, these limb torsion data are converted into a moment-skeleton coordinate system according to the skeleton coordinate system. In step S340, these muscle group activation time data are converted into a muscle strength eigenvalue-skeleton coordinate system according to the skeleton coordinate system. The present disclosure does not limit the execution order of step S330 and step S340, and step S330 and step S340 can also be performed simultaneously. In step S350, a fusion calculation is performed on the moment-skeleton coordinate system and the muscle strength eigenvalue-skeleton coordinate system. Among them, the fusion calculation is performed on the moment-skeleton coordinate system and the muscle strength eigenvalue-skeleton coordinate system by using K-Means Clustering (KMC). In step S360, evaluation data is calculated for the training action according to the result of the fusion calculation.

[0104] In step S370, it is determined according to the evaluation data that the training action corresponds to one of multiple known motion actions. In step S380, the evaluation data and the known motion actions are displayed.

[0105] Figure 12 It is a flowchart showing the calculation of evaluation data in the multi-modal perception collaborative training method illustrated according to an embodiment of the present disclosure. As Figure 12 shown, in step S321, coordinate system conversion is performed on multiple limb torsion data and multiple muscle group activation time data according to the skeleton coordinate system. In step S330, these limb torsion data are converted into a force-skeleton coordinate system according to the skeleton coordinate system, and then in step S331, the force-skeleton coordinate system is converted into a moment-skeleton coordinate system according to the skeleton coordinate system. Additionally, in step S340, these muscle group activation time data are converted into a muscle strength activation time-skeleton coordinate system according to the skeleton coordinate system, and then in step S341, the muscle strength activation time-skeleton coordinate system is converted into a muscle strength eigenvalue-skeleton coordinate system according to the skeleton coordinate system. It should be particularly noted that steps S330 to S331 and steps S340 to S341 are two separate processes and can be performed simultaneously or separately.

[0106] Once the muscle strength activation time-skeleton coordinate system and the muscle strength eigenvalue-skeleton coordinate system are converted according to the skeleton coordinate system, in step S350, fusion calculation is performed on the moment-skeleton coordinate system and the muscle strength eigenvalue-skeleton coordinate system. Then, in step S360, evaluation data for the training action is calculated according to the result of the fusion calculation.

[0107] In one embodiment, when the inertial sensor is disposed on the user's body, the skeleton coordinate system corresponds to the user's skeleton, and the user's skeleton can be obtained by a camera device. In one embodiment, when the inertial sensor is disposed on the training equipment, the skeleton coordinate system further corresponds to the skeleton of the training equipment, and the skeleton of the training equipment can be obtained by a camera device.

[0108] In one embodiment, the multi-modal perception collaborative training method further includes determining the user's motion scenario based on the number of body sensors used by the user, and selecting one of multiple known motion models based on the motion scenario.

[0109] In one embodiment, the multi-modal perception collaborative training method further includes comparing the evaluation data with the training data of the corresponding selected known motion model to calculate the similarity between the user's training action and the selected known motion model. When the similarity is greater than or equal to the similarity threshold, it is determined that the user's training action conforms to the selected known motion model, and the training data of the corresponding selected known motion model is updated with the evaluation data, and the evaluation data and the known motion action are displayed. Conversely, when the similarity is less than the similarity threshold, it is determined that the user's training action does not conform to each of all known motion models, the evaluation data is updated as error data, and the evaluation data and the error motion action message are displayed.

[0110] In one embodiment, the inertial sensor has an offset sensing unit, and the inertial sensor is disposed on the user's body. The multi-modal perception collaborative training method further includes sensing a plurality of offset data when a relative offset occurs between the inertial sensor and the user's body, and outputting a plurality of limb torsion data according to the plurality of body posture data and the plurality of offset data. Comparing the evaluation data with the training data of the corresponding selected known motion model to determine whether the relative offset between the inertial sensor and the user's body exceeds the offset threshold. Wherein when the relative offset is greater than the offset threshold, a sensor setting abnormal message is displayed.

[0111] In one embodiment, the multi-modal perception collaborative training method further includes sensing a plurality of mechanical data corresponding to the user's training action through mechanical sensors respectively disposed on the training equipment, sensing a plurality of body posture data corresponding to the user's training action through at least one inertial sensor disposed on the training equipment, and sensing a plurality of left half myoelectric data and a plurality of right half myoelectric data corresponding to the user's training action through at least two myoelectric sensors respectively disposed on the left half and the right half of the user's body. Outputting a plurality of pressure data according to the plurality of mechanical data, outputting a plurality of limb torsion data according to the plurality of body posture data, and respectively outputting a plurality of left half muscle group activation time data and a plurality of right half muscle group activation time data according to the plurality of left half myoelectric data and the plurality of right half myoelectric data. Calculating a left half output value according to the plurality of pressure data, the plurality of limb torsion data, and the plurality of left half muscle group activation time data, and calculating a right half output value according to the plurality of pressure data, the plurality of limb torsion data, and the plurality of right half muscle group activation time data. Performing another fusion calculation on the left half output data and the right half output data, and calculating the left-right balance degree corresponding to the user's training action according to the result of the another fusion calculation.

[0112] In one embodiment, when the left - right balance degree is less than or equal to the balance degree threshold, it is determined that the force outputs of the left and right halves of the user's body are balanced, and the left - right balance degree corresponding to the user's training action is continuously calculated according to the result of another fusion calculation. Conversely, when the left - right balance degree is greater than the balance degree threshold, it is determined that the force outputs of the left and right halves of the user's body are unbalanced, and the evaluation data and imbalance message are displayed.

[0113] In one embodiment, the multi - mode perception collaborative training method further includes sensing a plurality of physiological data of the user, such as information data of body temperature, heart rate, respiration, skin moisture content, sweat, etc. when the user is performing a training action. Based on these physiological data, the physiological condition of the user during the training action is monitored to determine whether to send a warning signal to remind the user to stop the training action.

[0114] In summary, the multi - mode perception collaborative training system and multi - mode perception collaborative training method provided by the present disclosure can enable the users in the gym to know whether the posture of using fitness equipment is correct, whether the muscle groups used and the training actions match, or whether the muscle group force output sequence of the training action is correct without the guidance of a coach, avoid sports injuries, and can also quantify the exercise effect into an index to provide a way for coaches and athletes to discuss improvement methods.

[0115] Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present invention, not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that: they can still modify the technical solutions recorded in the foregoing embodiments, or perform equivalent replacements for some or all of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A multi-modal perception collaborative training system, comprising: A plurality of body sensors, including: At least one inertial sensor for sensing a plurality of body data related to the training actions of a user; and At least one electromyography sensor for sensing a plurality of electromyography data related to the training actions of the user; A sensing module coupled to these body sensors for outputting a plurality of limb torsion data according to these body data and outputting a plurality of muscle group activation time data according to these electromyography data; An operation module coupled to the sensing module for performing the following steps: Converting these limb torsion data into a moment-skeleton coordinate system according to a skeleton coordinate system; Converting these muscle group activation time data into a muscle force eigenvalue-skeleton coordinate system according to the skeleton coordinate system; Performing a fusion calculation on the moment-skeleton coordinate system and the muscle force eigenvalue-skeleton coordinate system; Calculating evaluation data for the training action according to the result of the fusion calculation; and Determining that the training action corresponds to one of a plurality of known motion actions according to the evaluation data; and A display module coupled to the operation module for displaying the evaluation data and the known motion actions.

2. The multi-modal perception collaborative training system according to claim 1, wherein the operation module is further configured to perform the following steps: Converting these limb torsion data into a force-skeleton coordinate system according to the skeleton coordinate system, and then converting the force-skeleton coordinate system into the moment-skeleton coordinate system; and Converting these muscle group activation time data into a muscle force activation time-skeleton coordinate system according to the skeleton coordinate system, and then converting the muscle force activation time-skeleton coordinate system into the muscle force eigenvalue-skeleton coordinate system.

3. The multi-modal perception collaborative training system according to claim 1, wherein when the at least one inertial sensor is disposed on the user's body, the skeleton coordinate system corresponds to the user's body skeleton, and the user's body skeleton can be obtained by a camera device.

4. The multi-modal perception collaborative training system according to claim 3, wherein when the at least one inertial sensor is disposed on training equipment, the skeleton coordinate system further corresponds to the equipment skeleton of the training equipment, and the equipment skeleton of the training equipment can be obtained by the camera device.

5. The multi-modal perception collaborative training system according to claim 1, further comprising: A motion model database including a plurality of known motion models; And A motion simulation model module coupled to the motion model database and the operation module; Wherein the operation module determines the motion scenario of the user based on the number of these body sensors used by the user; Wherein the operation module is paired with the motion simulation model module based on the motion scenario, and the motion simulation model module selects one of these known motion models from the motion model database based on the motion scenario.

6. The multi-modal perception collaborative training system according to claim 5, further comprising: A training data database coupled to the operation module, the training data database including training data and error data corresponding to each of these known motion models; The operation module compares the evaluation data with the training data of the corresponding selected known motion model to calculate the similarity between the training action of the user and the selected known motion model.

7. The multi-modal perception collaborative training system according to claim 6, wherein when the similarity is greater than or equal to a similarity threshold, the operation module determines that the training action of the user conforms to the selected known motion model, stores the evaluation data in the training data database to update the training data of the corresponding selected known motion model, and the display module displays the evaluation data and the known motion action; when the similarity is less than a similarity threshold, the operation module determines that the training action of the user does not conform to these known motion models, stores the evaluation data in the training data database to update the error data, and the display module displays the evaluation data and error motion action messages.

8. The multi-modal perception collaborative training system according to claim 6, wherein the at least one inertial sensor has an offset sensing unit, and the at least one inertial sensor is disposed on the user's body to sense a plurality of offset data when a relative offset occurs between the at least one inertial sensor and the user's body, and the sensing module outputs the limb torsion data according to the body posture data and the offset data; wherein the operation module compares the evaluation data with the training data of the corresponding selected known motion model to determine whether the relative offset between the at least one inertial sensor and the user's body exceeds an offset threshold; wherein when the relative offset is greater than the offset threshold, the display module displays a sensor setting abnormal message.

9. The multi-modal perception collaborative training system according to claim 1, further comprising: a mechanical sensor, coupled to the sensing module, disposed on the training equipment to sense a plurality of mechanical data corresponding to the training action of the user; wherein the at least one inertial sensor is disposed on the training equipment to sense the body posture data corresponding to the training action of the user; wherein the at least one electromyography sensor is respectively disposed on the left half and the right half of the user's body to sense a plurality of left half electromyography data and a plurality of right half electromyography data corresponding to the training action of the user; wherein the sensing module outputs a plurality of pressure data according to the mechanical data and outputs the limb torsion data according to the body posture data; wherein the sensing module respectively outputs a plurality of left half muscle group activation time data and a plurality of right half muscle group activation time data according to the left half electromyography data and the right half electromyography data; wherein the operation module is further configured to perform the following steps: calculate a left half output value according to the pressure data, the limb torsion data, and the left half muscle group activation time data; calculate a right half output value according to the pressure data, the limb torsion data, and the right half muscle group activation time data; perform another fusion calculation on the left half output data and the right half output data; and calculate the left-right balance corresponding to the training action of the user according to the result of the another fusion calculation.

10. The multi-modal perception collaborative training system according to claim 9, wherein when the left-right balance degree is less than or equal to a balance degree threshold, the operation module determines that the output of the left half and the right half of the user's body is balanced, and continuously calculates the left-right balance degree of the training action corresponding to the user according to the result of the other fusion calculation; and when the left-right balance degree is greater than the balance degree threshold, the operation module determines that the output of the left half and the right half of the user's body is unbalanced, and the display module displays the evaluation data and the imbalance message.

11. A multi-modal perception collaborative training method, comprising: sensing a plurality of body posture data related to a training action of a user through at least one inertial sensor of a plurality of body posture sensors, and sensing a plurality of electromyogram data related to the training action of the user through at least one electromyogram sensor of these body posture sensors; outputting a plurality of limb torsion data according to these body posture data, and outputting a plurality of muscle group activation time data according to these electromyogram data; converting these limb torsion data into a torque-skeleton coordinate system according to a skeleton coordinate system; converting these muscle group activation time data into a muscle strength eigenvalue-skeleton coordinate system according to the skeleton coordinate system; performing a fusion calculation on the torque-skeleton coordinate system and the muscle strength eigenvalue-skeleton coordinate system; calculating evaluation data for the training action according to the result of the fusion calculation; determining that the training action corresponds to one of a plurality of known motion actions according to the evaluation data; and displaying the evaluation data and the known motion action.

12. The multi-modal perception collaborative training method according to claim 11, further comprising: converting these limb torsion data into a force-skeleton coordinate system according to the skeleton coordinate system, and then converting the force-skeleton coordinate system into the torque-skeleton coordinate system; and converting these muscle group activation time data into a muscle strength activation time-skeleton coordinate system according to the skeleton coordinate system, and then converting the muscle strength activation time-skeleton coordinate system into the muscle strength eigenvalue-skeleton coordinate system.

13. The multi-modal perception collaborative training method according to claim 11, wherein when the at least one inertial sensor is disposed on the user's body, the skeleton coordinate system corresponds to the user's body skeleton, and the user's body skeleton can be obtained by a camera device.

14. The multi-modal perception collaborative training method according to claim 13, wherein when the at least one inertial sensor is disposed on training equipment, the skeleton coordinate system further corresponds to the equipment skeleton of the training equipment, and the equipment skeleton of the training equipment can be obtained by the camera device.

15. The multi-modal perception collaborative training method according to claim 11, further comprising: determining the motion scenario of the user based on the number of these body posture sensors used by the user; selecting one of a plurality of known motion models based on the motion scenario.

16. The multi-modal perception collaborative training method according to claim 15, further comprising: comparing the evaluation data with the training data corresponding to the selected known motion model to calculate the similarity between the training action of the user and the selected known motion model.

17. The multimodal perception collaborative training method according to claim 16, wherein when the similarity is greater than or equal to a similarity threshold, it is determined that the training action of the user conforms to the selected known motion model, and the training data corresponding to the selected known motion model is updated with the evaluation data, and the evaluation data and the known motion action are displayed; when the similarity is less than a similarity threshold, it is determined that the training action of the user does not conform to these known motion models, and the evaluation data is updated as error data, and the evaluation data and error motion action message are displayed.

18. The multimodal perception collaborative training method according to claim 16, wherein the at least one inertial sensor has an offset sensing unit, and the at least one inertial sensor is disposed on the body of the user, and the multimodal perception collaborative training method further includes: when a relative offset amount is generated between the at least one inertial sensor and the body of the user, sensing a plurality of offset data, and outputting the limb torsion data according to the body posture data and the offset data; and comparing the evaluation data with the training data corresponding to the selected known motion model to determine whether the relative offset amount between the at least one inertial sensor and the body of the user exceeds an offset threshold; wherein when the relative offset amount is greater than the offset threshold, a sensor setting abnormal message is displayed.

19. The multimodal perception collaborative training method according to claim 11, further includes: sensing a plurality of mechanical data corresponding to the training action of the user through mechanical sensors respectively disposed on the training equipment; sensing the body posture data corresponding to the training action of the user through the at least one inertial sensor disposed on the training equipment; sensing a plurality of left half electromyogram data and a plurality of right half electromyogram data corresponding to the training action of the user through the at least one electromyogram sensor respectively disposed on the left half and the right half of the body of the user; outputting a plurality of pressure data according to the mechanical data, and outputting the limb torsion data according to the body posture data; respectively outputting a plurality of left half muscle group activation time data and a plurality of right half muscle group activation time data according to the left half electromyogram data and the right half electromyogram data; calculating a left half output value according to the pressure data, the limb torsion data and the left half muscle group activation time data; calculating a right half output value according to the pressure data, the limb torsion data and the right half muscle group activation time data; performing another fusion calculation on the left half output data and the right half output data; and calculating the left-right balance corresponding to the training action of the user according to the result of the another fusion calculation.

20. The multimodal perception collaborative training method according to claim 19, wherein when the left-right balance is less than or equal to a balance threshold, it is determined that the left half and the right half of the user's body exert force in balance, and the left-right balance corresponding to the training action of the user is continuously calculated according to the result of the another fusion calculation; and When the left-right balance degree is greater than the balance degree threshold value, it is determined that the force output of the left half and the right half of the user's body is unbalanced, and the evaluation data and the unbalanced message are displayed.

Images