Multi-imu fusion action recognition method and system

Abstract

The application discloses a multi-IMU fusion action recognition method, comprising the following steps: acquiring IMU data of multiple key points of a user in motion through multiple IMU sensors; when scoring a user action, acquiring key points corresponding to the current user action as selected key points; acquiring IMU data corresponding to the selected key points as selected IMU data, and scoring the user action according to the selected IMU data. The application further discloses a multi-IMU fusion action recognition system. The multi-IMU fusion action recognition method and system effectively reduce the sample amount of the training model by collecting targeted IMU data for different user actions, also reduce the amount of IMU data acquired each time, alleviate the pressure on computing resources, and improve user experience.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of action recognition, and particularly relates to a multi-IMU fusion action recognition method and system. BACKGROUND

[0002] Due to the complexity of human action postures, in a sports course, a model needs to be synthesized by multiple monitoring point data of the same posture to determine whether the action is completed, and the values of the posture and acceleration provided by the IMU are used to determine whether the action posture is completed accurately or how much it meets the expectation to score the quality of the action posture.

[0003] In the prior art, all IMU data is fused and processed to recognize or score the action posture through a model, and when the amount of IMU data is large, it is easy to cause high calculation delay and consume a large amount of computing resources, thereby reducing the user experience. SUMMARY

[0004] The technical problem to be solved by the present application is that the action posture detection in the prior art uses all IMU data for fusion processing, which has large calculation amount and high calculation delay, and the purpose is to provide a multi-IMU fusion action recognition method and system to solve the above problems.

[0005] The present application is implemented by the following technical solutions:

[0006] In one aspect, the present embodiment provides a multi-IMU fusion action recognition method, comprising:

[0007] obtaining IMU data of multiple key points of a user in a movement through multiple IMU sensors;

[0008] When scoring the user action, the key points corresponding to the current user action are obtained as selected key points;

[0009] obtaining the IMU data corresponding to the selected key points as selected IMU data, and scoring the user action according to the selected IMU data.

[0010] In the prior art, when a user performs an action posture detection, IMU data collected by all IMU sensors needs to be synchronously collected, and then a pre-trained model is used for action recognition or scoring. However, in order to accurately detect an action, IMU data of many key parts needs to be detected, for example, in a step-in-place action, if the detection accuracy needs to be ensured, IMU data of positions such as left hand, left shoulder, right hand, right shoulder, left ankle, right ankle, left knee, and right knee needs to be detected. When the user completes the current action and enters the next action, the detected positions may change. For example, in the detection of a boxing action, the lower body action may not be detected, and in the prior art, all data is collected and then fused and recognized, which is very large in calculation amount each time, and model training is also difficult.

[0011] In the implementation of the embodiment, IMU data of motion key points is acquired, and only key points of the current user action are selected when scoring the user action, and the data is fused and recognized for scoring. For example, ten IMU sensors are worn on the current user, which are left hand, left shoulder, right hand, right shoulder, left ankle, right ankle, left knee, right knee, hip, and neck. After the IMU sensors are turned on, the ten IMU sensors continuously provide detected IMU data. When it is recognized that the current user action is a step-in-place running action, only data of eight IMU sensors of left hand, left shoulder, right hand, right shoulder, left ankle, right ankle, left knee, and right knee is acquired and fused and recognized. When the user action changes to boxing, only data of four IMU sensors of left hand, left shoulder, right hand, and right shoulder is acquired and fused and recognized. It should be noted that in the example, the user action can be recognized by a model, or can be recognized by a corresponding fitness video, and can also be recognized by image recognition. The embodiment is not limited more herein. In the embodiment, the selected IMU data can be input into a corresponding scoring model for recognition, or can be recognized by other prior art methods. The embodiment is not limited more herein. The embodiment acquires IMU data of different user actions, effectively reduces the sample amount of the trained model, reduces the amount of acquired IMU data each time, reduces the computing resource pressure, and improves the user experience.

[0012] Further, the key points corresponding to the current user action are acquired as the selected key points.

[0013] The current user action is recognized according to the IMU data of the multiple key points, and a standard action corresponding to the current user action is generated.

[0014] The key points corresponding to the standard action are queried, and the key points are used as the selected key points.

[0015] Further, the scoring of the user action according to the selected IMU data comprises:

[0016] Upon listening to the recognition start instruction, all the selected IMU data are stored simultaneously, and all the selected IMU data are spliced into spliced data;

[0017] The spliced data are input into a recognition model corresponding to the user action to score the user action.

[0018] Further, the scoring of the user action according to the selected IMU data comprises:

[0019] According to the standard action corresponding to the user action, the selected IMU data are classified to form classified IMU data; and the classified IMU data of multiple categories are distributed along a time axis;

[0020] According to the start time and the duration of each classified IMU data, the classified IMU data are processed to form multiple recognition data distributed along a time axis;

[0021] The multiple recognition data are sequentially input into a recognition model corresponding to the recognition data along a time axis direction, and the user action is scored according to data output by the multiple recognition models.

[0022] Further, the standard action corresponds to a detection sequence of key points; and the detection sequence is obtained according to a time of movement at different positions in the standard action;

[0023] According to the standard action corresponding to the user action, the selected IMU data are classified to form classified IMU data, which comprises:

[0024] According to the detection sequence corresponding to the standard action, the selected IMU data are classified.

[0025] In another aspect, the embodiment provides a multi-IMU fusion action recognition system, comprising:

[0026] An acquisition unit configured to acquire IMU data of multiple key points of a user in movement through multiple IMU sensors;

[0027] A processing unit configured to, when scoring a user action, acquire key points corresponding to a current user action as selected key points;

[0028] The scoring unit is configured to acquire IMU data corresponding to the selected key points as selected IMU data, and score the user action according to the selected IMU data.

[0029] Further, the processing unit is further configured to:

[0030] identify the current user action according to the IMU data of the plurality of key points, and generate a standard action corresponding to the current user action;

[0031] query the key points corresponding to the standard action, and take the key points as the selected key points.

[0032] Further, the scoring unit is further configured to:

[0033] when the identification start instruction is listened to, store all the selected IMU data at the same time, and splice all the selected IMU data into spliced data;

[0034] input the spliced data into an identification model corresponding to the user action to score the user action.

[0035] Further, the scoring unit is further configured to:

[0036] classify a plurality of the selected IMU data according to the standard action corresponding to the user action to form classified IMU data; and the classified IMU data of a plurality of categories is distributed along a time axis;

[0037] determine the start time and the duration of each of the classified IMU data according to the standard action corresponding to the user action, and process the classified IMU data according to the start time and the duration to form a plurality of identification data distributed along the time axis;

[0038] input the plurality of identification data into an identification model corresponding to the identification data along the time axis direction in sequence, and score the user action according to the data output by the plurality of identification models.

[0039] Further, the standard action corresponds to a detection sequence of key points; the detection sequence is acquired according to the time of movement at different positions in the standard action;

[0040] The scoring unit is further configured to:

[0041] classify a plurality of the selected IMU data according to the detection sequence corresponding to the standard action.

[0042] Compared with the prior art, the present application has the following advantages and beneficial effects:

[0043] The multi-IMU fusion action recognition method and system effectively reduces the sample size of the training model, reduces the amount of IMU data obtained each time, reduces the computing resource pressure, and improves the user experience. BRIEF DESCRIPTION OF DRAWINGS

[0044] The accompanying drawings, which are included to provide a further understanding of the embodiments of the application and are incorporated in and constitute a part of this application, illustrate embodiments of the application and together with the description serve to explain the principles of the application. In the drawings:

[0045] Figure 1 The method steps of the embodiments of the application are shown in the figure.

[0046] Figure 2 The system architecture of the embodiments of the application is shown in the figure. DETAILED DESCRIPTION

[0047] To make the objects, technical solutions and advantages of the application clearer, further detailed description of the application will be given below in combination with the embodiments and the drawings, the illustrative embodiments of the application and the description thereof are only used to explain the application and do not limit the application.

[0048] EMBODIMENTS

[0049] Please refer to Figure 1 The flowchart of the multi-IMU fusion action recognition method provided by the embodiments of the application can be applied to the multi-IMU fusion action recognition system in Figure 2 , and further, the multi-IMU fusion action recognition method can specifically include the contents described in steps S1-S3.

[0050] S1: obtaining IMU data of multiple key points of a user in motion through multiple IMU sensors;

[0051] S2: when scoring the user action, obtaining the key points corresponding to the current user action as selected key points;

[0052] S3: obtaining the IMU data corresponding to the selected key points as selected IMU data, and scoring the user action according to the selected IMU data.

[0053] In the prior art, when a user performs an action posture detection, IMU data collected by all IMU sensors needs to be synchronously collected, and then a pre-trained model is used for action recognition or scoring. However, in order to accurately detect an action, IMU data of many key parts need to be detected, for example, in a step-in-place action, if the accuracy of detection needs to be ensured, IMU data of positions such as left hand, left shoulder, right hand, right shoulder, left ankle, right ankle, left knee, and right knee need to be detected. When the user completes the current action and enters the next action, the detected positions may change. For example, in the detection of a boxing action, the lower body action may not be detected, and in the prior art, all data is often collected and then fused and recognized, which is very large in calculation amount each time, and model training is also difficult.

[0054] In the implementation of the embodiment, IMU data of motion key points are acquired, and only key points of the current user action are selected when scoring the user action, and the data are fused and recognized. For example, ten IMU sensors are worn on the current user, which are left hand, left shoulder, right hand, right shoulder, left ankle, right ankle, left knee, right knee, hip, and neck. After the IMU sensors are turned on, the ten IMU sensors continuously provide detected IMU data. When it is recognized that the current user action is a step-in-place running action, only data of eight IMU sensors of left hand, left shoulder, right hand, right shoulder, left ankle, right ankle, left knee, and right knee are acquired and fused and recognized. When the user action changes to a boxing action, only data of four IMU sensors of left hand, left shoulder, right hand, and right shoulder are acquired and fused and recognized. It should be noted that in the example, the user action can be recognized by a model, or can be recognized by a corresponding fitness video, and can also be recognized by image recognition. The embodiment is not limited more herein. In the embodiment, the selected IMU data can be input into a corresponding scoring model for recognition, or can be recognized by other prior art methods. The embodiment is not limited more herein. The embodiment acquires IMU data of different user actions, effectively reduces the sample amount of the training model, reduces the amount of IMU data acquired each time, reduces the pressure on computing resources, and improves user experience.

[0055] In one embodiment, acquiring the key points corresponding to the current user action as the selected key points includes:

[0056] recognizing the current user action according to the IMU data of the multiple key points, and generating a standard action corresponding to the current user action;

[0057] querying the key points corresponding to the standard action, and taking the key points as the selected key points.

[0058] In the implementation of the embodiment, the current user action is recognized through the IMU data, and the current user action is obtained. This method can ensure that the user action is scored each time the user performs an action, even if there is no fitness video to be used as a reference during the user's fitness process. In the embodiment, the standard action is pre-configured, and each standard action has a corresponding key point. The standard action and the corresponding key point can be mapped through a standard action table, so as to facilitate query.

[0059] In one embodiment, scoring the user action according to the selected IMU data comprises:

[0060] When the recognition start instruction is listened to, all the selected IMU data are stored at the same time, and all the selected IMU data are spliced into spliced data;

[0061] The spliced data is input into a recognition model corresponding to the user action to score the user action.

[0062] In the implementation of the embodiment, the splicing of the selected IMU data is performed through the recognition start instruction. The splicing process can use any splicing method in the prior art, or the IMU data can be aligned according to the time axis and then spliced. The recognition model in the embodiment corresponds to different standard actions, and the standard action corresponds to the user action, which makes the user action and the recognition model have a corresponding relationship. The recognition model is trained using samples and belongs to a prior art, which will not be described in detail in this application.

[0063] For example, for an action that needs to be recognized by multiple IMU data at the same time, an instruction is sent to each thread that receives the IMU data at the same time when the action is to be recognized, wherein each IMU device corresponds to a thread, and each thread starts to receive and preprocess the respective data. After a preset time, each preprocessed IMU data is transmitted to a thread for data fusion, which splices multiple IMU data, and then transmits the spliced data to a trained model to recognize the specific action. The preset time is a time period set according to the target action to be recognized. For example, when detecting the action of "left hand swinging back + right hand swinging forward + left leg lifting", it is an action, and "right hand swinging back + left hand swinging forward + right leg lifting" is another action. When the password is "one time", the data of the left hand, the right hand and the left leg are spliced into one data for recognition.

[0064] In one embodiment, scoring the user action according to the selected IMU data comprises:

[0065] According to the standard action corresponding to the user action, the selected IMU data is classified to form classified IMU data; and the classified IMU data of multiple categories is distributed along a time axis;

[0066] According to the start time and the duration of each classified IMU data, the classified IMU data is processed to form multiple recognition data distributed along a time axis;

[0067] The recognition data is sequentially input into a recognition model corresponding to the recognition data along a time axis direction, and the user action is scored according to data output by the recognition model.

[0068] In one embodiment, the standard action corresponds to a detection sequence of key points; and the detection sequence is obtained according to a time of movement at different positions in the standard action;

[0069] According to the standard action corresponding to the user action, the selected IMU data is classified to form classified IMU data; and the classified IMU data of multiple categories is distributed along a time axis;

[0070] According to the detection sequence corresponding to the standard action, the selected IMU data is classified.

[0071] When the embodiment is implemented, the inventor finds that, due to the complexity of human movement, when multiple IMU sensors are arranged on the human body, the data between the multiple IMU sensors will affect each other, resulting in inaccurate scoring results. For example, in a boxing action, the left fist is needed to be thrown first and then the right fist is needed to be thrown. The body rotation caused by throwing the left fist will be synchronously detected in the detection of the right fist. Therefore, it is difficult to accurately determine the accuracy when collecting the punching angular velocity, acceleration and the like of the right fist. Therefore, the embodiment provides another implementation scheme for scoring the user action. In the embodiment, the standard action is decomposed to generate multiple sub-actions, and each sub-action corresponds to a classified key point. The classification of the IMU data is performed by the key point corresponding to the sub-action. Each group of classified IMU data corresponds to a sub-action. The sub-actions are arranged along a time axis, and therefore the classified IMU data is also arranged along the time axis. It should be understood that the classified IMU data arranged along the time axis as described herein refers to that the starting point of the classified IMU data is arranged along the time axis, and different classified IMU data may overlap on the time axis.

[0072] The recognition data is determined according to the time length of the classified IMU data corresponding to the sub-action. Each recognition data corresponds to a recognition model, which only scores the corresponding sub-action, and finally the scores of all sub-actions are integrated to obtain the score of the user action, which can be integrated in a linear fitting manner or other existing technologies.

[0073] For example, during left-right straight punch combination training, left punch, left punch, right punch, and right punch are required, and a set of actions is completed. At this time, the sub-actions of the standard action are divided into four: left punch, left punch, right punch, and right punch. The IMU data corresponding to the left punch and the left punch is the left elbow, the left shoulder, and the left wrist, while the IMU data corresponding to the right punch and the right punch is the right elbow, the right shoulder, and the right wrist. Then, the IMU data of the user during left-right straight punch combination training is classified into four categories corresponding to left punch, left punch, right punch, and right punch. For left punch data, the classified IMU data is processed into recognition data according to the preset start time and duration of left punch, and the other three sub-actions are processed accordingly. In this way, each action in a set of actions of the user can be accurately evaluated, and the evaluation result is more accurate.

[0074] Please refer to Figure 2 Based on the same inventive concept, a multi-IMU fusion action recognition system is also provided, which comprises:

[0075] The acquisition unit is configured to acquire IMU data of multiple key points of the user in the movement through multiple IMU sensors;

[0076] The processing unit is configured to select key points corresponding to the current user action as selected key points when scoring the user action;

[0077] The scoring unit is configured to acquire IMU data corresponding to the selected key points as selected IMU data, and score the user action according to the selected IMU data.

[0078] In one embodiment, the processing unit is further configured to:

[0079] identify the current user action according to the IMU data of the multiple key points, and generate a standard action corresponding to the current user action;

[0080] query the key points corresponding to the standard action, and take the key points as the selected key points.

[0081] In one embodiment, the scoring unit is further configured to:

[0082] When the recognition start instruction is detected, all the selected IMU data is stored at the same time, and all the selected IMU data is spliced into spliced data.

[0083] The spliced ​​data is input into the recognition model corresponding to the user action to score the user action.

[0084] In one embodiment, the scoring unit is further configured to:

[0085] The selected IMU data are classified according to the standard action corresponding to the user action to form classified IMU data; the classified IMU data of multiple categories are distributed along the time axis;

[0086] The start time and duration of each category IMU data are determined based on the standard action corresponding to the user action, and the category IMU data are processed according to the start time and duration to form multiple identification data distributed along the time axis;

[0087] Multiple recognition data are sequentially input into the recognition model corresponding to the recognition data along the time axis, and the user action is scored based on the data output by the multiple recognition models.

[0088] In one embodiment, the standard action corresponds to a detection sequence of key points; the detection sequence is obtained based on the time taken for movement at different positions in the standard action.

[0089] The scoring unit is also configured to:

[0090] The selected IMU data are classified according to the detection order corresponding to the standard action.

[0091] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0092] In several embodiments provided in the present application, it should be understood that the disclosed apparatus and method can be implemented by other manners. For example, the apparatus embodiments described above are merely illustrative, for example, the division of the units is merely a logical function division, and actual implementation can have another division manner, for example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the units shown or discussed can be indirect coupling or communication connection through some interfaces, devices or units, and can also be electrical, mechanical or other form of connection.

[0093] The units described as separate components can or can not be physically separated, and it is obvious to those skilled in the art that the units and algorithm steps of each example described in combination with the embodiments disclosed herein can be realized by electronic hardware, computer software or a combination of both. In order to clearly illustrate the interchangeability of hardware and software, the components and steps of each example have been described in the above description in general terms. Whether the functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. The skilled person can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present application.

[0094] In addition, each functional unit in each embodiment of the present application can be integrated into one processing unit, or each unit can exist physically, or two or more units can be integrated into one unit. The integrated unit can be realized in the form of hardware or software functional unit.

[0095] When the integrated unit is realized in the form of software functional unit and sold or used as an independent product, it can be stored in a computer readable storage medium. Based on such understanding, the technical solutions of the present application essentially or the part that contributes to the prior art, or the whole or part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes a plurality of instructions for causing a computer device (which can be a personal computer, a server, or a grid device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The foregoing storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a magnetic disk or an optical disk, and various program code storage media.

[0096] The above detailed description of the specific embodiments of the present application has been given to understand the purpose, technical solutions and beneficial effects of the present application. It should be understood that the above description is only a specific embodiment of the present application and is not used to limit the protection scope of the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A multi-IMU fusion action recognition method, characterized in that, include: IMU data from multiple key points during the user's movement is acquired using multiple IMU sensors; When scoring user actions, the key points corresponding to the current user action are obtained as the key points to be selected. The IMU data corresponding to the selected key point is obtained as the selection IMU data, and the user action is scored based on the selection IMU data; Obtaining key points corresponding to the current user action as key points for selection includes: Identify the current user's action based on IMU data from multiple key points, and generate the standard action corresponding to the current user's action; Query the key points corresponding to the standard action, and use the key points as the selected key points; Scoring the user's actions based on the selected IMU data includes: The selected IMU data are classified according to the standard action corresponding to the user action to form classified IMU data; the classified IMU data of multiple categories are distributed along the time axis; The start time and duration of each category IMU data are determined based on the standard action corresponding to the user action, and the category IMU data are processed according to the start time and duration to form multiple identification data distributed along the time axis; Multiple recognition data are sequentially input into the recognition model corresponding to the recognition data along the time axis, and the user action is scored based on the data output by the multiple recognition models; The standard action corresponds to a detection sequence of key points; the detection sequence is obtained based on the time taken for different positions to move within the standard action. The selected IMU data are classified according to the standard action corresponding to the user action to form classified IMU data, including: The selected IMU data are classified according to the detection order corresponding to the standard action.

2. The multi-IMU fusion action recognition method according to claim 1, characterized in that, Scoring the user's actions based on the selected IMU data includes: Upon receiving the recognition start command, all selected IMU data are stored simultaneously, and all selected IMU data are concatenated into concatenated data. The spliced ​​data is input into the recognition model corresponding to the user action to score the user action.

3. A multi-IMU fusion motion recognition system, characterized in that, include: The acquisition unit is configured to acquire IMU data of the user at multiple key points during movement via multiple IMU sensors; The processing unit is configured to obtain the key points corresponding to the current user action as the key points to be selected when scoring the user action; The scoring unit is configured to acquire the IMU data corresponding to the selected key point as the selection IMU data, and score the user action based on the selection IMU data; The processing unit is further configured to: Identify the current user's action based on IMU data from multiple key points, and generate the standard action corresponding to the current user's action; Query the key points corresponding to the standard action, and use the key points as the selected key points; The scoring unit is also configured to: The selected IMU data are classified according to the standard action corresponding to the user action to form classified IMU data; the classified IMU data of multiple categories are distributed along the time axis; The start time and duration of each category IMU data are determined based on the standard action corresponding to the user action, and the category IMU data are processed according to the start time and duration to form multiple identification data distributed along the time axis; Multiple recognition data are sequentially input into the recognition model corresponding to the recognition data along the time axis, and the user action is scored based on the data output by the multiple recognition models; the standard action corresponds to the detection order of key points; the detection order is obtained based on the movement time of different positions in the standard action; The scoring unit is also configured to: The selected IMU data are classified according to the detection order corresponding to the standard action.

4. The multi-IMU fusion action recognition system according to claim 3, characterized in that, The scoring unit is also configured to: Upon receiving the recognition start command, all selected IMU data are stored simultaneously, and all selected IMU data are concatenated into concatenated data. The spliced ​​data is input into the recognition model corresponding to the user action to score the user action.

Images