Method, device and electronic equipment for evaluating a movement posture

Abstract

The application provides a motion posture evaluation method and device and electronic equipment, and the method comprises the following steps: obtaining the to-be-evaluated motion posture data of a current user, wherein the to-be-evaluated motion posture data is posture data obtained based on a motion video of the current user shot by an optical camera; performing space-time alignment and evaluation on the to-be-evaluated motion posture data based on reference motion posture data, and determining the motion posture evaluation result of the current user; wherein the reference motion posture data is posture data determined based on a motion video of a target user wearing an IMU shot by the optical camera. In the application, the reference motion posture data is obtained in the early stage by combining the optical camera and the IMU, and the to-be-evaluated motion posture data is obtained by only using the optical camera in the later stage of evaluating the motion posture of the user, so that the IMU does not need to be worn every time and the optical camera does not need to be used every time, the accuracy of the human motion posture evaluation is improved, and the application range of the human motion posture evaluation is also improved.

Description

Technical Field

[0001] This invention relates to the field of human posture assessment technology, and in particular to a method, device and electronic device for evaluating movement posture. Background Technology

[0002] With the continuous development of technology, the study of human movement posture has become increasingly convenient and efficient. For example, wearable sensor systems containing control circuits, inertial measurement units (IMUs), and plantar pressure insoles can be used to evaluate user movement posture. In other words, when a user wears an IMU and plantar pressure insoles, the movement posture is evaluated by collecting IMU acceleration signals and gait information during walking.

[0003] However, existing methods for evaluating motion posture rely on IMU acceleration signals, and each evaluation requires the user to wear an IMU, resulting in low accuracy and significantly limited applicability of motion posture evaluation. Summary of the Invention

[0004] This invention provides a motion posture evaluation method, device, and electronic device to address the shortcomings of existing technologies that rely on IMU-based IMU acceleration signals for evaluating human posture, which requires all users to wear an IMU, resulting in low accuracy and significantly limited applicability. By combining an optical camera and an IMU when acquiring reference motion posture data in the early stages, and only requiring an optical camera to acquire the motion posture data to be evaluated when evaluating the user's motion posture in the later stages, the invention achieves the purpose of evaluating the current user's motion posture. This eliminates the need for wearing an IMU and using an optical camera for each evaluation, thereby significantly improving the accuracy and applicability of human motion posture evaluation.

[0005] This invention provides a method for evaluating motion posture, comprising:

[0006] Obtain the motion posture data of the current user to be evaluated, wherein the motion posture data to be evaluated is posture data obtained based on the motion video of the current user captured by an optical camera;

[0007] Based on reference motion posture data, the motion posture data to be evaluated is spatiotemporally aligned and evaluated to determine the current user's motion posture evaluation result.

[0008] The reference motion posture data is posture data determined based on motion videos captured by the optical camera for a target user wearing an IMU.

[0009] According to a motion posture evaluation method provided by the present invention, the step of performing spatiotemporal alignment and evaluation on the motion posture data to be evaluated based on reference motion posture data to determine the motion posture evaluation result of the current user includes:

[0010] Spatiotemporal alignment is performed based on the motion posture data to be evaluated and the reference motion posture data;

[0011] Based on the successful spatiotemporal alignment result, the reference motion posture data is decomposed into multiple time micro-elements, each of which corresponds to a reference action in the reference motion posture data.

[0012] Determine the different time segments corresponding to the plurality of time micro-elements in the motion posture data to be evaluated; each time segment corresponds to an action in the motion posture data to be evaluated.

[0013] Based on the variance statistics of the different time segments, the time score result of the motion posture data to be evaluated is determined;

[0014] Based on the Euclidean distance between the key point location information of each time segment, the spatial evaluation result between the motion posture data to be evaluated and the reference motion posture data is determined.

[0015] The time score and the spatial evaluation result are determined as the current user's motion posture evaluation results.

[0016] According to a motion posture evaluation method provided by the present invention, the step of performing spatiotemporal alignment based on the motion posture data to be evaluated and the reference motion posture data includes:

[0017] Based on a preset action segmentation and recognition annotation model, the set of actions to be identified and annotated in the motion posture data to be evaluated is determined;

[0018] Based on the motion posture data to be evaluated, determine the length information of the current user's body parts;

[0019] Based on the body part length information, the reference motion posture data is corrected to determine the individualized motion posture data of the current user;

[0020] The action set and the individualized motion posture data are spatiotemporally aligned.

[0021] According to the motion posture evaluation method provided by the present invention, the training process of the preset action segmentation and recognition annotation model includes:

[0022] Construct an initial action segmentation and recognition annotation model containing a batch normalization layer, a one-dimensional convolutional layer, a one-dimensional max pooling layer, and a fully connected layer;

[0023] Obtain a sample dataset, which includes posture data obtained from motion videos of different first sample users captured by the optical camera and IMU acceleration signals collected from the first sample users wearing IMUs.

[0024] The sample dataset is divided into a sample training set and a sample test set;

[0025] The initial action segmentation and recognition annotation model is trained using the sample training set to identify and annotate different actions, thereby determining the intermediate action segmentation and recognition annotation model after training.

[0026] The trained intermediate action segmentation and recognition annotation model is tested using the sample test set, and the intermediate action segmentation and recognition annotation model corresponding to the test result reaching the preset accuracy rate is determined as the preset action segmentation and recognition annotation model.

[0027] According to a motion posture evaluation method provided by the present invention, the process of acquiring the reference motion posture data includes:

[0028] The system acquires motion videos of the target user captured by the optical camera and obtains target IMU acceleration signals from the IMU worn by the target user. The target user is a user who moves according to standard motion requirements and wears an IMU at different joint points.

[0029] The reference motion posture data is obtained based on the motion video of the target user, the acceleration signal of the target IMU, and the preset motion posture estimation network model.

[0030] According to a motion posture evaluation method provided by the present invention, the step of obtaining the motion posture data to be evaluated of the current user includes:

[0031] The system acquires motion video of the current user captured by the optical camera, wherein the current user is a user who moves freely and does not wear an IMU at any joint point;

[0032] Based on the current user's motion video and the preset motion posture estimation network model, obtain the current user's motion posture data to be evaluated.

[0033] According to the motion posture evaluation method provided by the present invention, the training process of the preset motion posture estimation network model includes:

[0034] The motion video and IMU acceleration signal of the second sample user captured by the optical camera are acquired; the second sample user is a user who wears an IMU at different joint points and moves according to standard motion requirements.

[0035] Based on the sample IMU acceleration signal, determine the position information of different joints of the second sample user;

[0036] Based on the motion video of the second sample user and the different joint position information, the initial motion pose estimation network model containing the DeepPose network and the temporal convolutional network is trained to perform two-dimensional pose estimation, process the two-dimensional pose into three-dimensional pose, and compensate for the missing and misjudged motion information in the three-dimensional pose, thereby determining the preset motion pose estimation network model.

[0037] According to a motion posture evaluation method provided by the present invention, the method further includes:

[0038] Information is displayed based on the motion posture evaluation results.

[0039] The present invention also provides a motion posture evaluation device, comprising:

[0040] The acquisition module is used to acquire the motion posture data of the current user to be evaluated, which is the posture data obtained based on the motion video of the current user captured by the optical camera;

[0041] The evaluation module is used to perform spatiotemporal alignment and evaluation on the motion posture data to be evaluated based on reference motion posture data, and to determine the motion posture evaluation result of the current user.

[0042] The reference motion posture data is posture data determined based on motion videos captured by the optical camera for a target user wearing an IMU.

[0043] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the motion posture evaluation method as described above.

[0044] The present invention provides a motion posture evaluation method, apparatus, and electronic device. The motion posture evaluation method involves a terminal device first acquiring the motion posture data to be evaluated from the current user, and then performing spatiotemporal alignment and evaluation based on reference motion posture data to determine the current user's motion posture evaluation result. Since the motion posture data to be evaluated is posture data obtained from motion videos of the current user captured by an optical camera, and the reference motion posture data is posture data determined from motion videos of a target user wearing an IMU captured by an optical camera, the method achieves the purpose of evaluating the current user's motion posture by combining the optical camera and IMU when acquiring the reference motion posture data initially, and only requiring the optical camera to acquire the motion posture data to be evaluated when evaluating the user's motion posture later. This eliminates the need for an IMU and an optical camera for each evaluation, improving the accuracy of motion video capture and thus significantly increasing the accuracy and applicability of human motion posture evaluation. Attached Figure Description

[0045] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0046] Figure 1 This is a flowchart illustrating the motion posture evaluation method provided by the present invention;

[0047] Figure 2 This is a schematic diagram of the pose processing of the temporal convolutional network provided by the present invention;

[0048] Figure 3 This is a schematic diagram of the three-dimensional pose estimation process provided by the present invention;

[0049] Figure 4 This is a logic block diagram of the motion posture evaluation method provided by the present invention;

[0050] Figure 5 This is an execution block diagram of the motion posture evaluation method provided by the present invention;

[0051] Figure 6 This is a schematic diagram of the motion posture evaluation device provided by the present invention;

[0052] Figure 7 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0053] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0054] Current methods for evaluating human posture involve wearing IMUs on the user's right thigh, calf, and instep, along with a pressure-sensitive insole on the right foot. These methods utilize an STM32F767 chip and RS485 serial communication to evaluate posture based on acquired IMU acceleration signals and gait information during walking. Since the movements of various parts of the user's body are rigid body movements, and the IMU measures linear and angular acceleration in three directions, existing methods rely on IMU acceleration signals. Furthermore, each posture evaluation requires the user to wear an IMU, resulting in low accuracy and significantly limited applicability.

[0055] To address the aforementioned technical problems, this invention provides a motion posture evaluation method, device, and electronic device, which are described below in conjunction with... Figures 1-7 This invention describes a motion posture evaluation method, apparatus, and electronic device. The execution subject of the motion posture evaluation method can be a terminal device or a server. The terminal device can be a personal computer (PC), portable device, laptop, smartphone, tablet computer, portable wearable device, or other electronic device. The server can refer to a single server or a server cluster or cloud computing center composed of multiple servers. The specific form of the terminal device or server is not limited here. The following method embodiments use a terminal device as an example for illustration.

[0056] Reference Figure 1 This is a flowchart illustrating the motion posture evaluation method provided by the present invention, as shown below. Figure 1 As shown, this motion posture evaluation method includes the following steps:

[0057] Step 110: Obtain the motion posture data of the current user to be evaluated. The motion posture data to be evaluated is the posture data obtained from the motion video of the current user captured by the optical camera.

[0058] In this context, the current user is the user whose posture is to be evaluated; the optical camera can be a visible light camera, such as a monocular or binocular camera. Furthermore, the optical camera can be mounted on the terminal device or set up independently and communicate with the terminal device. The specific location of the optical camera is not limited here. The number of current users can be one or more. Again, the number of current users is not limited here. Further, the motion video can be a series of video frames captured by the optical camera targeting a moving user. Each video frame in the motion video is an RGB image, where RGB represents the three color channels: red (R), green (G), and blue (B).

[0059] Specifically, the terminal device acquires the current user's motion posture data to be evaluated, and can control the activation of the optical camera to capture motion video of the current user during movement. For example, the optical camera can be activated to capture motion video when the current user performs several consecutive movements. At this time, the terminal device performs analysis and processing based on the motion video captured by the optical camera, such as performing motion posture recognition on each image frame in the motion video, thereby acquiring the current user's motion posture data to be evaluated.

[0060] Step 120: Based on the reference motion posture data, perform spatiotemporal alignment and evaluation on the motion posture data to be evaluated, and determine the current user's motion posture evaluation result; the reference motion posture data is the posture data determined based on the motion video taken by the optical camera for the target user wearing an IMU.

[0061] The target users are those wearing IMUs and exercising according to standard movement requirements. These requirements can include the movement standards for each movement within a set of actions, such as the head tilt angle and the angle between the arms during an extension movement. Furthermore, the spatiotemporal evaluation can include both temporal and spatial evaluations.

[0062] Specifically, the terminal device performs spatiotemporal alignment and evaluation on the motion posture data to be evaluated based on reference motion posture data. For example, it templates the reference motion posture data to determine a standard motion posture template, and then performs time-dimensional and spatial-dimensional posture evaluations on the motion posture data to be evaluated based on the standard motion posture template, thereby determining the spatiotemporal evaluation result of the motion posture data to be evaluated, and then uses the spatiotemporal evaluation result of the motion posture data to be evaluated as the current user's motion posture evaluation result.

[0063] The motion posture evaluation method provided by this invention involves a terminal device first acquiring the motion posture data to be evaluated of the current user, and then performing spatiotemporal alignment and evaluation based on reference motion posture data to determine the current user's motion posture evaluation result. Since the motion posture data to be evaluated is posture data obtained from motion videos of the current user captured by an optical camera, and the reference motion posture data is posture data determined from motion videos of a target user wearing an IMU captured by an optical camera, the method achieves the purpose of evaluating the current user's motion posture by combining the optical camera and IMU when acquiring the reference motion posture data initially, and only requiring the optical camera to acquire the motion posture data to be evaluated when evaluating the user's motion posture later. This eliminates the need for an IMU and an optical camera for each evaluation, improving the accuracy of motion video capture and thus significantly increasing the accuracy and applicability of human motion posture evaluation.

[0064] Optionally, the specific implementation process of step 110 may include:

[0065] First, acquire motion video of the current user captured by an optical camera. The current user is a user who moves freely and does not wear an IMU at any joint point. Then, based on the current user's motion video and a preset motion posture estimation network model, acquire the motion posture data of the current user to be evaluated.

[0066] Specifically, for the current user who needs motion posture evaluation, the terminal device first acquires motion video of the current user performing random and continuous actions, and then inputs the acquired motion video into a preset motion posture estimation network model to obtain the motion posture data to be evaluated output by the preset motion posture estimation network model.

[0067] The motion posture evaluation method provided by this invention involves a terminal device acquiring the user's motion posture data to be evaluated by inputting the user's current motion video into a preset motion posture estimation network model. This, combined with the preset motion posture estimation network model, improves the accuracy of acquiring the motion posture data to be evaluated, and also effectively enhances the accuracy and reliability of cloud-based posture evaluation.

[0068] Optionally, the specific process of acquiring the reference motion posture data in step 120 may include:

[0069] First, the motion video of the target user captured by the optical camera and the acceleration signal of the target IMU worn by the target user are obtained. The target user is a user who moves according to the standard action requirements and wears an IMU at different joint points. Then, based on the motion video of the target user, the acceleration signal of the target IMU and the preset motion posture estimation network model, reference motion posture data are obtained.

[0070] The target user wears IMUs at various joint points on their body. These joint points can be any movable joint on the target user's body, such as the neck joint, both knee joints, both ankle joints, both shoulder joints, both elbow joints, both hip joints, and both wrist joints. Furthermore, IMUs may or may not be worn at the lumbar joint. No specific restrictions are set here.

[0071] Specifically, in order to facilitate the quick and accurate evaluation of motion posture in the later stage, the terminal device can pre-acquire and store reference motion posture data. That is, the target user is first instructed to move according to the standard action requirements, and the optical camera is simultaneously activated to shoot motion video during the target user's movement. At the same time, the target IMU acceleration signal is obtained from each IMU worn by the target user. Then, the target user's motion video and the obtained target IMU acceleration signal are input into the preset motion posture estimation network model to obtain the reference motion posture data output by the preset motion posture estimation network model.

[0072] It should be noted that since the target user wears IMUs at different joints, the position of each joint when the target user performs different actions can be determined based on the different target IMU acceleration signals acquired during the movement. This allows for automatic labeling of each action of the target user during the movement phase and compensation for the loss and misjudgment of movement information caused by factors such as occlusion and clothing smudges, thereby determining accurate and reliable reference movement posture data.

[0073] The motion posture evaluation method provided by this invention involves a terminal device acquiring reference motion posture data by inputting the motion video of the target user into a preset motion posture estimation network model. This method, combined with the preset motion posture estimation network model, IMU (Inductively Coupled Unit), and optical technology, significantly improves the accuracy and reliability of the reference motion posture data, while also enhancing its precision, thus laying the foundation for accurate evaluation of the current user's motion posture.

[0074] Optionally, the training process of the preset motion pose estimation network model may include:

[0075] First, the motion video of the second sample user captured by the optical camera and the acceleration signal of the sample IMU are acquired. The second sample user is a user who wears an IMU at different joints and moves according to standard motion requirements. Then, the IMU acceleration signal is further sampled to determine the position information of different joints of the second sample user. Then, based on the motion video of the second sample user and the position information of different joints, the initial motion pose estimation network model containing DeepPose network and temporal convolutional network is trained to perform two-dimensional pose estimation, two-dimensional pose processing into three-dimensional pose, and compensation for missing and misjudged motion information in the three-dimensional pose, and to determine the preset motion pose estimation network model.

[0076] The motion videos of the second sample users can be multiple different motion videos captured for a single second user, or one motion video captured for each of multiple second users. Furthermore, the number of sample IMU acceleration signals corresponds one-to-one with the number of motion videos of the second sample users. Additionally, each motion video of the second sample user can be a monocular RGB motion video.

[0077] Specifically, since an IMU is an acceleration measurement device that measures linear acceleration and angular acceleration in three directions, meaning that an IMU generates acceleration signals, when an IMU is worn at various joint points on the human body, the real-time sample IMU acceleration signal generated by each IMU can reflect the real-time motion state of the human body, including real-time position and rotation signals. At this time, by combining physical laws and mathematical analysis, it can be determined that the acceleration signal is the second derivative of the position and rotation variables. That is, by performing the second derivative of the sample IMU acceleration signal, the position information of different joint points of the second sample user can be determined, that is, the position information of each joint point on the human body of the second sample user with an IMU.

[0078] Based on this, the terminal device can use the motion video of the second sample user and the different key point position information of the second sample user to train the initial motion pose estimation network model. First, the motion video of the second sample user is input into the initial motion pose estimation network model, and the DeepPose network estimates the two-dimensional pose in each video frame, thereby estimating the two-dimensional pose sequence; then, it is combined with... Figure 2 The diagram shown illustrates the pose processing of a temporal convolutional network, which transforms a two-dimensional pose sequence into a three-dimensional pose sequence. The process of estimating the three-dimensional pose for each frame of the motion video of the second sample user can be described as follows: Figure 3As shown, the input image is first passed through a 2D human pose estimation network to estimate the 2D pose, and then through a 3D pose estimation network to estimate the 3D pose. The 2D human pose estimation network is the DeepPose network, and the 3D pose estimation network is a temporal convolutional network. Further, specific key points in the 3D pose sequence are selected to obtain the spatial motion trajectory for visual positioning. This spatial motion trajectory is then matched with the sample IMU acceleration signals corresponding to the motion video of the second sample user. This matching includes using the different joint position information determined based on the matched sample IMU acceleration signals to correct the selected specific key points and to compensate for the loss of key point motion information and misjudgments caused by factors such as occlusion and clothing artifacts. This yields the output motion pose and the trained motion pose estimation network model. After multiple training iterations in this manner, it is determined whether the loss of the trained motion pose estimation network model meets the preset loss requirement. If the loss meets the preset loss requirement, training is stopped, and the motion pose estimation network model at the point of training cessation is identified as the preset motion pose estimation network model. Conversely, if the loss does not meet the preset loss requirement, the motion pose estimation network model is trained again using motion videos from a new second sample user and new sample IMU acceleration signals. This process continues until the preset motion pose estimation network model at the point of training cessation is obtained. Specific key points may include the head, hands, wrists, elbows, shoulders, feet, knees, and hips.

[0079] It should be noted that since the advantage of motion pose estimation based on monocular RGB motion video is that the contextual information provided by adjacent video frames helps to predict the pose of the current video frame, in the case of occlusion, the pose of the current video frame can be reasonably inferred based on the pose of the frames before and after the current video frame. In addition, since the bone length of the same second user remains unchanged in a motion video, the constraint of bone length consistency can be introduced to limit it, which helps to output a more stable 3D pose.

[0080] It should be noted that, considering that temporal convolutional networks are essentially convolutions in the time domain and their greatest advantage over recurrent neural networks lies in their ability to process multiple 2D pose sequences in parallel, temporal convolutional networks have low computational complexity and fewer parameters. Therefore, dilated convolutions can be further utilized to expand the receptive field of temporal convolutional networks. The specific network structure can employ a fully convolutional network with residual connections. In addition, temporal convolutional networks can incorporate semi-supervised training methods. The main idea is to add a trajectory prediction model to predict the absolute coordinates of the root joints, projecting the absolute 3D pose in the camera coordinate system back to the 2D plane, thus introducing a reprojection loss. Semi-supervised methods can better improve model training accuracy when 3D labels are limited.

[0081] It should be noted that the spatial motion trajectory of visual positioning is matched with the sample IMU acceleration signal corresponding to the motion video of the second sample user. On the one hand, the sample IMU acceleration signal is used to correct the key points of visual positioning and add real-time motion direction and velocity information. The main factor at play here is the acceleration sensitivity of the sample IMU acceleration signal. On the other hand, the sample IMU acceleration signal is used to fill in the missing key points caused by factors such as occlusion. The main factor at play here is that the sample IMU acceleration signal is independent of the visual image. In the motion key point data of the combined sample IMU acceleration signal and visual information, motion can be recorded using 3D data modeling software. These motion records will be stored as standard motion posture templates for evaluation in subsequent test motions.

[0082] The motion posture evaluation method provided by this invention involves a terminal device training an initial motion posture estimation network model containing a DeepPose network and a temporal convolutional network using motion videos of a second sample user and sample IMU acceleration signals. This training process includes two-dimensional posture estimation, two-dimensional posture conversion to three-dimensional posture, and compensation for missing and misjudged motion information in the three-dimensional posture. This determines a preset motion posture estimation network model. By combining this method with training the DeepPose network, temporal convolutional network, and sample IMU acceleration signal compensation techniques, the accuracy and precision of the preset motion posture estimation network model are effectively improved.

[0083] Optionally, the specific implementation process of step 120 may include:

[0084] First, spatiotemporal alignment is performed between the motion posture data to be evaluated and the reference motion posture data. Then, based on the successful spatiotemporal alignment, the reference motion posture data is decomposed into multiple time micro-elements, each corresponding to a reference action within the reference motion posture data. Further, different time segments corresponding to these multiple time micro-elements in the motion posture data to be evaluated are determined; each time segment corresponds to an action within the motion posture data to be evaluated. Next, based on the variance statistics of different time segments, the temporal score of the motion posture data to be evaluated is determined. Based on the Euclidean distance between the keypoint locations in each time segment, the spatial evaluation result between the motion posture data to be evaluated and the reference motion posture data is determined. Finally, the temporal score and spatial evaluation result are determined as the current user's motion posture evaluation result.

[0085] Specifically, the terminal device performs temporal and spatial alignment based on the motion posture data to be evaluated and the reference motion posture data. Based on the spatiotemporal alignment result, which represents both temporal and spatial alignment, a new timeline is established. This new timeline serves to match the time scaling caused by uneven motion speed. First, the reference motion posture data is decomposed into N time micro-elements, where N is a positive integer greater than 0. To meet computational requirements, each time micro-element is set to 0.01 seconds. Then, the N time micro-elements are mapped to the motion posture data to be evaluated. Ideally, each micro-element in the motion posture data to be evaluated should be consistent with the standard motion posture template. However, due to the complexity of actual motion, there may be arbitrary scaling on the time axis, specifically, some decomposed movements may be faster, while others may be slower. To quantify this motion difference, the time length of each time micro-element corresponding to the segment in the motion posture data to be evaluated can be estimated, determining the different time segments corresponding to multiple time micro-elements in the motion posture data to be evaluated. Finally, the sample variance of these different time segments is used as the time score result (diff) of the motion posture data to be evaluated. t The calculation formula is as follows:

[0086]

[0087] Among them, t i For the i-th time segment, μ t This represents the sample mean of N time segments; the sample variance of different time segments is the variance statistic of different time segments.

[0088] For spatial analysis, keypoint location information for each time segment can be averaged over time to distribute the time dimension across multiple identical time points. Since there is a one-to-one correspondence between the standard motion posture template and the motion posture data to be evaluated at these time points, the Euclidean distance between the calculated keypoint location information for each time segment can be determined as the spatial evaluation result between the motion posture data to be evaluated and the reference motion posture data. The calculation formula is as follows:

[0089]

[0090]

[0091]

[0092] Among them, diff x The diff represents the x-axis score between the motion posture data to be evaluated and the reference motion posture data. y The diff represents the y-axis score between the motion posture data to be evaluated and the reference motion posture data. z f represents the z-axis score between the motion posture data to be evaluated and the reference motion posture data. xi Let g be the x-coordinate of the keypoint f in the i-th time segment. xi Let f be the x-coordinate of the key point g in the i-th time segment. yi Let g be the y-coordinate of the keypoint f in the i-th time segment. yi Let f be the y-coordinate of the key point g in the i-th time segment. zi Let g be the z-coordinate of the keypoint f in the i-th time segment. zi Let g be the coordinate value of the key point g in the i-th time segment on the z-axis.

[0093] The motion posture evaluation method provided by this invention involves a terminal device first performing spatiotemporal alignment based on the motion posture data to be evaluated and reference motion posture data, then decomposing the reference motion posture data into multiple time micro-elements, and finally determining the different time segments corresponding to these multiple time micro-elements in the motion posture data to be evaluated. The Euclidean distance between the variance statistics of different time segments and the key point location information of each time segment is then used to determine the current user's motion posture evaluation result. This combination of micro-element decomposition technology and spatiotemporal evaluation technology improves the flexibility, versatility, reliability, and accuracy of motion posture evaluation.

[0094] Optionally, spatiotemporal alignment can be performed based on the motion posture data to be evaluated and the reference motion posture data, including:

[0095] First, based on the preset action segmentation and recognition annotation model, the action set identified and annotated in the motion posture data to be evaluated is determined; then, based on the motion posture data to be evaluated, the length information of the current user's body parts is determined; further, based on the length information of the body parts, the reference motion posture data is corrected to determine the current user's individualized motion posture data; then, the action set and individualized motion posture data are spatiotemporally aligned.

[0096] Specifically, the terminal device obtains the set of identified and labeled actions from the motion posture data to be evaluated by inputting it into a preset action segmentation and recognition annotation model. It also extracts the length information of the current user's body parts from the motion posture data. This length information can include, but is not limited to, the lengths of the upper arm, forearm, torso, thigh, and lower leg. The current user's body part length information is used to correct the parameters of a standard motion posture template, specifically the parameters of the upper arm, forearm, torso, thigh, and lower leg, thereby determining the current user's individualized motion posture data, or individualized motion posture template. Based on this, the terminal device performs spatiotemporal alignment of the action set and the individualized motion posture data. This can be achieved by first parsing the individualized motion posture data into keypoint motion data, and then spatiotemporally aligning the identified and labeled action set from the motion posture data to be evaluated with the parsed keypoint motion data.

[0097] It should be noted that, in order to improve the efficiency of spatiotemporal alignment, the current user's motion posture data to be evaluated can be obtained by instructing the current user to repeatedly perform specified actions and continuously execute different specified actions. Then, a preset action segmentation and recognition labeling model is used to segment and label the continuous actions of the motion posture data to be evaluated, and these segments are aligned with the current user's individualized motion posture data. The alignment includes both spatial alignment and temporal alignment.

[0098] Spatial alignment refers to the alignment of the spatial positions of keypoints, while temporal alignment refers to the alignment of the temporal sequences of each keypoint. These are two interdependent factors. Therefore, the two alignments can be achieved simultaneously. To avoid overly complex calculations, this invention uses a least-squares loss function to match the motion pose data to be evaluated with a standard motion pose template. This is achieved by estimating a set of scaling and bias coefficients on the time and spatial axes. The formulas for calculating the scaling coefficient β1 and the bias coefficient b1 are as follows:

[0099] (β1,b1)=argmin(f.-(g.β1+b1))

[0100] Where f. represents the coordinates of keypoint f on the x, y, and z axes, and g. represents the coordinates of keypoint g on the x, y, and z axes. The matched motion pose data g′ to be evaluated is:

[0101] g′=g.β1+b1

[0102] Another principle of matching is that the total time and maximum amplitude of the motion posture data to be evaluated and the standard motion posture template are consistent. This is achieved by adding another set of scaling factors β2 on the time axis and space axis, and the calculation formula is as follows:

[0103]

[0104] The matched motion posture data to be evaluated, g″, is: g″ = g′β2. Furthermore, the matched motion parameters to be evaluated are used for motion scoring.

[0105] The motion posture evaluation method provided by this invention involves a terminal device first determining the identified and labeled action set in the motion posture data to be evaluated, then correcting the reference motion posture data based on the current user's body part length information, and finally aligning the action set and the corrected individualized motion posture data spatiotemporally. This achieves the purpose of spatiotemporally aligning the motion posture data to be evaluated and the reference motion posture data, ensuring that the motion posture evaluation is more accurate and reliable. Furthermore, since the method of determining the identified and labeled action set in the motion posture data to be evaluated based on a preset action segmentation and identification labeling model can also improve the accuracy and intelligence of the action set identification.

[0106] Optionally, the training process of the preset action segmentation and recognition annotation model may include:

[0107] First, an initial action segmentation and recognition annotation model is constructed, containing a batch normalization layer, a one-dimensional convolutional layer, a one-dimensional max pooling layer, and a fully connected layer. Then, a sample dataset is acquired, including motion posture data obtained from motion videos of different first-sample users captured by an optical camera, and IMU acceleration signals collected from first-sample users wearing IMUs. The sample dataset is further divided into a training set and a test set. Next, the training set is used to train the initial action segmentation and recognition annotation model to identify and annotate different actions, determining the intermediate action segmentation and recognition annotation model after training. Finally, the test set is used to test the intermediate action segmentation and recognition annotation model, and the intermediate action segmentation and recognition annotation model corresponding to the test result reaching a preset accuracy rate is determined as the preset action segmentation and recognition annotation model.

[0108] Specifically, for the constructed initial action segmentation and recognition annotation model, one-dimensional convolutional layers can effectively extract motion information, while max pooling layers reduce the model size and significantly improve its robustness. Furthermore, skip connections are employed in the initial action segmentation and recognition annotation model to obtain better feature selection from the IMU acceleration signal. Experimental results show that the proposed method achieves good classification results, possesses practical value, and exhibits strong robustness and generalization ability. Moreover, the sample dataset can be used to determine the training and test sets by selecting 200ms time windows in an 8:2 ratio.

[0109] Based on this, the initial action segmentation and recognition annotation model is trained using a sample training set to identify and annotate different actions. That is, the motion phases in the motion posture data are automatically labeled using IMU acceleration signals to identify and annotate the start and end positions of different actions. Specifically, the initial action segmentation and recognition annotation model first divides the sample training set into 20 consecutive one-dimensional motion data within 10 milliseconds, with a total size of 120 vectors. Then, the 120 vectors are converted into classification vectors of size 9. Finally, cross-entropy is used as the loss function and stochastic gradient descent is used as the optimizer to judge the training effect of the action segmentation and recognition annotation model after each training. Furthermore, based on the initial weights and thresholds of the initial action segmentation and recognition annotation model, the learning rate is set to 0.002 and decays to 0.9 every 20 rounds. In this way, the initial action segmentation and recognition annotation model is trained to identify and annotate different actions. After 200 rounds of training, the intermediate action segmentation and recognition annotation model can achieve an accuracy of 97.28% after being tested on the sample test set. At this point, training can be stopped and the test result is confirmed to have reached the preset accuracy. The intermediate action segmentation and recognition annotation model corresponding to the test result reaching the preset accuracy is determined to be the preset action segmentation and recognition annotation model.

[0110] The motion posture evaluation method provided by this invention involves a terminal device training and testing an initial action segmentation and recognition annotation model containing batch normalization layers, one-dimensional convolutional layers, one-dimensional max pooling layers, and fully connected layers using a sample training set and a sample test set. This process determines a preset action segmentation and recognition annotation model. By combining batch normalization, convolution, pooling, and fully connected techniques with IMU's automatic labeling and compensation techniques, the accuracy of action segmentation and recognition annotation can be effectively improved, thereby also effectively improving the accuracy and reliability of the preset action segmentation and recognition annotation model.

[0111] Optionally, after step 130, the motion posture evaluation method provided by the present invention may further include:

[0112] Information is displayed based on the results of motion posture evaluation.

[0113] Specifically, when the current motion posture evaluation results include both temporal and spatial evaluation results, the terminal device can display the temporal evaluation results on the timeline of the display interface, and simultaneously display the spatial evaluation results. Furthermore, when both the temporal and spatial evaluation results are scores, the device can feed back the temporal and spatial scores of the current user's motion posture at different times in the form of an error chart; for example, the temporal score for action A performed by the current user on the 12th is 3 points, while the temporal score for action A performed by the current user on the 13th is 1 point. This allows the current user to intuitively view the motion posture evaluation results at key points during the usage phase.

[0114] The motion posture evaluation method provided by this invention enables the terminal device to display the current user's motion posture evaluation results intuitively to the current user, thereby ensuring that the current user can obtain motion errors in a timely and accurate manner, and improving feedback and improvement effects.

[0115] Reference Figure 4 Here is a logic block diagram of the motion posture evaluation method provided by the present invention, as shown below. Figure 4 As shown, the terminal device integrates motion data from a preset motion posture estimation network model based on motion videos of the target user captured by an optical camera and acceleration signals from various IMUs worn by the target user. This integration determines reference motion posture data. Based on the reference motion posture data and the preset motion posture estimation network model, a reference sequence is determined, i.e., the target user's reference motion posture data. Simultaneously, motion analysis is performed on the current user's motion videos captured by the optical camera and the preset motion posture estimation network model to determine the sequence to be tested, i.e., the current user's motion posture data to be evaluated. Finally, based on the reference motion posture data, the motion posture data to be evaluated is spatiotemporally aligned and evaluated, i.e., time scoring and spatial scoring are performed. This yields the time and spatial scores of the current user's motion posture at different times, which are then fed back to the current user in the form of an error chart. The specific processes involved can be referred to in the aforementioned embodiments. They will not be repeated here.

[0116] Reference Figure 5 Here is an execution block diagram of the motion posture evaluation method provided by the present invention, as shown below. Figure 5As shown, the motion posture evaluation method provided by this invention can include a standard motion posture template generation stage and a motion posture scoring stage. For the standard motion posture template generation stage, automatic labeling of the motion process and compensation for missing or misjudged motion information can be performed based on a preset motion posture estimation network model and the acquired target IMU acceleration signal to determine reference motion posture data for the target user. A standard motion posture template is then determined based on the reference motion posture data. Based on the body part length information of the current user in the motion posture data to be evaluated, the standard motion posture template is parameter-corrected to determine the individualized motion posture template for the current user. For the motion posture scoring stage, the current user's motion posture data to be evaluated is first obtained based on the preset motion posture estimation network model. Then, based on the reference motion posture data and the current user's individualized motion posture template, the current user's motion posture is spatiotemporally aligned and evaluated, i.e., the temporal score and spatial score are determined. The specific processes involved can also be referred to in the aforementioned embodiments. They will not be repeated here.

[0117] The motion posture evaluation device provided by the present invention is described below. The motion posture evaluation device described below can be referred to in correspondence with the motion posture evaluation method described above.

[0118] Reference Figure 6 This is a schematic diagram of the motion posture evaluation device provided by the present invention, as shown below. Figure 6 As shown, the motion posture evaluation device 600 includes:

[0119] The acquisition module 610 is used to acquire the motion posture data to be evaluated of the current user. The motion posture data to be evaluated is the posture data obtained based on the motion video of the current user captured by the optical camera.

[0120] The evaluation module 620 is used to perform spatiotemporal alignment and evaluation of the motion posture data to be evaluated based on the reference motion posture data, and to determine the current user's motion posture evaluation result; wherein, the reference motion posture data is posture data determined based on motion video taken by an optical camera for a target user wearing an IMU.

[0121] Optionally, the evaluation module 620 can be used to perform spatiotemporal alignment based on the motion posture data to be evaluated and the reference motion posture data; based on the successful spatiotemporal alignment result, decompose the reference motion posture data into multiple time micro-elements, each time micro-element corresponding to a reference action in the reference motion posture data; determine the different time segments corresponding to the multiple time micro-elements in the motion posture data to be evaluated; each time segment corresponds to an action in the motion posture data to be evaluated; determine the time score result of the motion posture data to be evaluated based on the variance statistics of different time segments; determine the spatial evaluation result between the motion posture data to be evaluated and the reference motion posture data based on the Euclidean distance between the key point location information of each time segment; and determine the time score result and the spatial evaluation result as the motion posture evaluation result of the current user.

[0122] Optionally, the evaluation module 520 can also be used to determine the set of identified and labeled actions in the motion posture data to be evaluated based on a preset action segmentation and recognition annotation model; determine the length information of the current user's body parts based on the motion posture data to be evaluated; correct the reference motion posture data based on the body part length information to determine the current user's individualized motion posture data; and perform spatiotemporal alignment of the action set and the individualized motion posture data.

[0123] Optionally, the motion posture evaluation device provided by the present invention may further include a training module for constructing an initial action segmentation and recognition annotation model containing a batch normalization layer, a one-dimensional convolutional layer, a one-dimensional max pooling layer, and a fully connected layer; acquiring a sample dataset, which includes posture data obtained from motion videos of different first sample users captured by an optical camera and IMU acceleration signals collected from the first sample users wearing IMUs; dividing the sample dataset into a sample training set and a sample test set; using the sample training set to train the initial action segmentation and recognition annotation model for the recognition and annotation of different actions, and determining the intermediate action segmentation and recognition annotation model after training; using the sample test set to test the intermediate action segmentation and recognition annotation model after training, and determining the intermediate action segmentation and recognition annotation model corresponding to the test result reaching a preset accuracy as the preset action segmentation and recognition annotation model.

[0124] Optionally, the acquisition module 610 can be used to acquire motion videos of the target user captured by an optical camera and to acquire target IMU acceleration signals from the IMU worn by the target user. The target user is a user who moves according to standard motion requirements and wears an IMU at different joint points. Based on the motion videos of the target user, the target IMU acceleration signals, and a preset motion posture estimation network model, reference motion posture data is acquired.

[0125] Optionally, the acquisition module 610 can also be used to acquire the motion video of the current user captured by the optical camera, where the current user is a user who moves freely and does not wear an IMU at any joint point; based on the motion video of the current user and a preset motion posture estimation network model, the motion posture data of the current user to be evaluated is acquired.

[0126] Optionally, the training module can also be used to acquire motion videos of a second sample user captured by an optical camera and sample IMU acceleration signals; the second sample user is a user who wears an IMU at different joints and moves according to standard motion requirements; based on the sample IMU acceleration signals, the position information of different joints of the second sample user is determined; based on the motion videos of the second sample user and the position information of different joints, the initial motion pose estimation network model containing the DeepPose network and the temporal convolutional network is trained to perform two-dimensional pose estimation, two-dimensional pose processing into three-dimensional pose, and compensation for missing and misjudged motion information in the three-dimensional pose, thereby determining the preset motion pose estimation network model.

[0127] Optionally, the motion posture evaluation device provided by the present invention may further include a display module for displaying information based on the motion posture evaluation results.

[0128] Figure 7 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 7 As shown, the electronic device 700 may include: a processor 710, a communication interface 720, a memory 730, and a communication bus 740, wherein the processor 710, the communication interface 720, and the memory 730 communicate with each other via the communication bus 740. The processor 710 can call logical instructions in the memory 730 to execute a motion attitude evaluation method, which includes:

[0129] Obtain the current user's motion posture data to be evaluated. The motion posture data to be evaluated is the posture data obtained from the motion video of the current user captured by the optical camera.

[0130] Based on reference motion posture data, spatiotemporal alignment and evaluation are performed on the motion posture data to be evaluated to determine the current user's motion posture evaluation result.

[0131] The reference motion posture data is posture data determined based on motion videos captured by an optical camera targeting a user wearing an IMU.

[0132] Furthermore, the logical instructions in the aforementioned memory 730 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, essentially, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0133] On the other hand, the present invention also provides a computer program product, the computer program product comprising a computer program that can be stored on a non-transitory computer-readable storage medium, and when the computer program is executed by a processor, the computer is able to execute the motion posture evaluation method provided by the above methods, the method comprising:

[0134] Obtain the current user's motion posture data to be evaluated. The motion posture data to be evaluated is the posture data obtained from the motion video of the current user captured by the optical camera.

[0135] Based on reference motion posture data, spatiotemporal alignment and evaluation are performed on the motion posture data to be evaluated to determine the current user's motion posture evaluation result.

[0136] The reference motion posture data is posture data determined based on motion videos captured by an optical camera targeting a user wearing an IMU.

[0137] In another aspect, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the motion attitude evaluation method provided by the methods described above, the method comprising:

[0138] Obtain the current user's motion posture data to be evaluated. The motion posture data to be evaluated is the posture data obtained from the motion video of the current user captured by the optical camera.

[0139] Based on reference motion posture data, spatiotemporal alignment and evaluation are performed on the motion posture data to be evaluated to determine the current user's motion posture evaluation result.

[0140] The reference motion posture data is posture data determined based on motion videos captured by an optical camera targeting a user wearing an IMU.

[0141] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0142] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0143] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method of evaluating a motion posture, characterized by, The method comprises the following steps: obtaining the to-be-evaluated motion posture data of the current user, wherein the to-be-evaluated motion posture data is posture data obtained based on a motion video of the current user shot by an optical camera; determining a set of recognized and labeled actions in the to-be-evaluated motion posture data based on a preset action segmentation and recognition labeling model; determining the length information of the body parts of the current user based on the to-be-evaluated motion posture data; correcting the reference motion posture data based on the length information of the body parts to determine the individualized motion posture data of the current user; spatiotemporally aligning the set of actions and the individualized motion posture data; based on the successful result of spatiotemporal alignment, decomposing the reference motion posture data into a plurality of time microelements, each of which corresponds to a reference action in the reference motion posture data; determining different time segments in the to-be-evaluated motion posture data corresponding to the plurality of time microelements; each of the time segments corresponds to an action in the to-be-evaluated motion posture data; determining the time score result of the to-be-evaluated motion posture data based on the variance statistics of the different time segments; determining the spatial evaluation result between the to-be-evaluated motion posture data and the reference motion posture data based on the Euclidean distance between the key point position information of each of the time segments; determining the time score result and the spatial evaluation result as the motion posture evaluation result of the current user; wherein the reference motion posture data is posture data determined based on a motion video of a target user wearing an IMU shot by the optical camera.

2. The motion posture evaluation method according to claim 1, characterized by, The training process of the preset action segmentation and recognition labeling model comprises: constructing an initial action segmentation and recognition labeling model containing a batch normalization layer, a convolution layer, a max pooling layer and a fully connected layer; obtaining a sample data set, wherein the sample data set comprises posture data obtained based on motion videos of different first sample users shot by the optical camera and IMU acceleration signals collected from the first sample users wearing IMUs; dividing the sample data set into a sample training set and a sample test set; using the sample training set to perform recognition and labeling training of different actions on the initial action segmentation and recognition labeling model to determine a trained intermediate action segmentation and recognition labeling model; using the sample test set to test the trained intermediate action segmentation and recognition labeling model to determine that the intermediate action segmentation and recognition labeling model corresponding to the time when the test result reaches a preset accuracy is the preset action segmentation and recognition labeling model.

3. The motion posture evaluation method according to claim 1, characterized by, The acquisition process of the reference motion posture data comprises: obtaining a motion video of a target user shot by the optical camera and a target IMU acceleration signal obtained from an IMU worn by the target user, wherein the target user is a user who moves according to standard action requirements and wears an IMU at each joint; obtaining the reference motion posture data based on the motion video of the target user, the target IMU acceleration signal and a preset motion posture estimation network model.

4. The motion posture evaluation method according to claim 3, characterized by, The method for obtaining the to-be-evaluated motion posture data of the current user comprises: acquire a motion video of a current user captured by the optical camera, the current user being a user with random motion and no IMU worn on any joint; acquire the to-be-evaluated motion posture data of the current user based on the motion video of the current user and the preset motion posture estimation network model.

5. The motion posture evaluation method according to claim 3 or 4, characterized by, The training process of the preset motion posture estimation network model comprises: acquire a motion video of a second sample user captured by the optical camera and a sample IMU acceleration signal, the second sample user being a user with IMUs worn on different joints and moving according to a standard action requirement; determine the position information of different joints of the second sample user based on the sample IMU acceleration signal; determine the preset motion posture estimation network model through two-dimensional posture estimation, three-dimensional posture processing, and training of compensating for missing and misjudged motion information in the three-dimensional posture of an initial motion posture estimation network model containing a DeepPose network and a time convolution network based on the motion video of the second sample user and the position information of different joints of the second sample user.

6. The motion posture evaluation method according to claim 1, characterized by, The method further comprises: display information based on the motion posture evaluation result.

7. A motion posture evaluation device, characterized in that, comprise: an acquisition module configured to acquire to-be-evaluated motion posture data of a current user, the to-be-evaluated motion posture data being posture data obtained based on a motion video of the current user captured by an optical camera; an evaluation module configured to determine a set of recognized actions in the to-be-evaluated motion posture data based on a preset action segmentation and recognition labeling model; determine body part length information of the current user based on the to-be-evaluated motion posture data; correct reference motion posture data based on the body part length information to determine individualized motion posture data of the current user, and perform spatiotemporal alignment on the set of actions and the individualized motion posture data; based on a successful spatiotemporal alignment result, decompose the reference motion posture data into a plurality of time microelements, each of the time microelements corresponding to a reference action in the reference motion posture data; determine different time segments corresponding to the plurality of time microelements in the to-be-evaluated motion posture data, each of the time segments corresponding to an action in the to-be-evaluated motion posture data, determine a time score result of the to-be-evaluated motion posture data based on variance statistics of the different time segments, and determine a spatial evaluation result between the to-be-evaluated motion posture data and the reference motion posture data based on the Euclidean distance between key point position information of each of the time segments; determine the time score result and the spatial evaluation result as a motion posture evaluation result of the current user. The reference motion posture data is posture data determined based on a motion video of a target user wearing an IMU captured by the optical camera.

8. An electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, The processor implements the motion posture evaluation method according to any one of claims 1 to 6 when executing the program.

Images