Method and system for pose detection and real-time feedback

Abstract

Disclosed in the present invention are a method and system for pose detection and real-time feedback. The method comprises: when a user performs exercise movements, an application program capturing the current video frame image in real time, and sending same to a back end; using a pose estimation model to estimate the coordinates of a user pose on the basis of the video frame image; using a pose similarity algorithm to calculate similarity scores for the current movement pose on the basis of the coordinates of a standard pose and the coordinates of the user pose; performing feedback on the basis of the similarity scores, and displaying correct and incorrect parts of the current movement pose; and the application program continuing capturing of the next image frame until the user finishes the exercise movements, and recording and displaying a pose count. By using a pose estimation model and a pose similarity algorithm, the pose of a user is more accurately estimated and evaluated; and correct and incorrect parts of a pose are dynamically fed back in real time, so as to guide the user to adjust movements in a timely manner, thereby overcoming the limitations of delayed feedback in traditional methods.

Description

A method and system for posture detection and real-time feedback TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence image recognition, in particular to a method and system for posture detection and real-time feedback. BACKGROUND

[0002] With the improvement of health awareness and the progress of technology, posture detection technology has gradually become an important means to promote personal health, optimize sports performance and prevent sports injuries. Traditional posture detection technology is mostly based on two-dimensional image analysis, which captures the user's image through a camera and compares it with the preset standard posture to evaluate the user's posture accuracy. However, this two-dimensional analysis method has many limitations in practical application and cannot meet the growing needs of health management and sports training.

[0003] Currently, for posture detection, there are the following technical problems:

[0004] (1) Since only relying on two-dimensional image information, the existing technology cannot comprehensively and accurately capture the three-dimensional posture information of the user, especially in posture evaluation at different angles, which limits the accuracy of posture detection and easily leads to misjudgment due to angle deviation, thereby increasing the risk of injury;

[0005] (2) In sports training that requires repeated counting, traditional methods often rely on manual counting or simple image processing technology, which is not only inefficient but also prone to errors, affecting training effectiveness and data reliability;

[0006] (3) Existing posture detection technology can only provide static or post-analysis results, lacking the ability to dynamically adjust user movements from multiple angles in real time during exercise, which hinders the application effect of improving exercise efficiency and preventing injuries. SUMMARY

[0007] The purpose of the present application is to provide a method and system for posture detection and real-time feedback to solve the problems of existing posture detection technology mentioned in the background art, such as the inability to comprehensively capture three-dimensional posture information and correct user's incorrect posture.

[0008] To achieve the above purpose, the present application adopts the following technical solutions:

[0009] According to one aspect of the present application, a method for posture detection and real-time feedback is provided, the method comprising:

[0010] The user performs an exercise action, the application program captures the current video frame image in real time and sends it to the backend;

[0011] using a pose estimation model, estimating user pose coordinates from the video frame image;

[0012] using a pose similarity algorithm, calculating a similarity score for each limb and the overall of the current action pose from the standard pose coordinates and the user pose coordinates;

[0013] providing feedback according to the similarity score, displaying correct and incorrect parts of the current action pose;

[0014] the application continues to capture the next frame image until the user ends the exercise action, records and displays the pose count.

[0015] Based on the foregoing scheme, the user pose coordinates are three-dimensional key point coordinates; the pose estimation model receives the current video frame image as input, extracts a two-dimensional feature map of the video frame image using a neural network, trains the model to predict the three-dimensional key point coordinates from the two-dimensional feature map, and fits on a three-dimensional pose dataset.

[0016] Based on the foregoing scheme, the standard pose for each exercise action is found from a database or other data source according to the timestamp, and the standard pose is converted to the three-dimensional standard pose coordinates.

[0017] Based on the foregoing scheme, the standard pose also includes being collected from real-time streaming media for users to obtain immediate feedback.

[0018] Based on the foregoing scheme, the user pose coordinates are aligned with the standard pose coordinates, so that the user pose coordinates and the standard pose coordinates are in the same reference coordinate system.

[0019] Based on the foregoing scheme, the pose similarity algorithm uses limb-to-limb similarity scores to evaluate the similarity between the user pose coordinates and the standard pose coordinates, including calculating the position vector of each limb key point of the user pose and the standard pose respectively, and calculating the scalar product of the position vectors to obtain the similarity score of each limb, and weighting the similarity scores of each limb to obtain the overall similarity score.

[0020] Based on the foregoing scheme, the feedback according to the overall similarity score includes comparing the overall similarity score with a set threshold to determine whether the current pose is correct or close to correct; displaying the user's pose in real time on the user interface and marking the correct and incorrect parts with different colors or markers.

[0021] Based on the foregoing scheme, if the overall similarity score reaches the set threshold, the current pose is considered correct or close to correct, and positive feedback is provided to the user; if the set threshold is not reached, the current pose is considered to be corrected, and corrective feedback is provided to the user.

[0022] Based on the foregoing scheme, the application continuously captures and processes video frames before the user ends the exercise action, judges whether the exercise action is ended through user input or automatic detection, and displays the results of posture counting on the user interface.

[0023] According to another aspect of the present application, a posture detection and real-time feedback system is provided, which comprises a video capture module, a posture estimation module, a similarity evaluation module, a feedback display module, a counting and recording module, and an interactive control module.

[0024] The video capture module is used to capture video frame images of the user performing the exercise action in real time.

[0025] The posture estimation module receives the video frame images and estimates user posture coordinates using a posture estimation model.

[0026] The similarity evaluation module calculates the similarity scores of each limb and the overall current action posture based on the user posture coordinates and standard posture coordinates using a posture similarity algorithm.

[0027] The feedback display module shows the correct and incorrect parts of the current action posture to the user based on the similarity scores.

[0028] The counting and recording module records the number of times the user correctly completes the exercise action each time and displays the total count when the user ends the exercise.

[0029] The interactive control module processes user interaction instructions, including but not limited to starting, pausing, and ending the exercise.

[0030] As can be seen from the above technical solutions, the present application has at least the following advantages and positive effects compared to the prior art:

[0031] (1) By using a posture estimation model and a posture similarity algorithm, the present application can more accurately estimate and evaluate the user's posture, especially in different angles and complex motion scenarios, significantly improving accuracy and reducing false positives due to perspective bias.

[0032] (2) The automated and intelligent posture counting function allows users to avoid the tediousness and inefficiency of manual counting, reduces the likelihood of human error, improves data accuracy and reliability, and thus improves exercise and training efficiency.

[0033] (3) Real-time and dynamic feedback of correct and incorrect parts of the posture helps users adjust their actions in a timely manner, ensures correct posture, avoids the limitations of lagging feedback in traditional methods, and improves exercise effectiveness and safety.

[0034] The present application should be understood to encompass all such variations, modifications and alterations of the application. BRIEF DESCRIPTION OF DRAWINGS

[0035] The accompanying drawings incorporated herein and forming a part of the specification illustrate embodiments consistent with the present application and serve to explain the principles of the present application. It is apparent that the accompanying drawings are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative effort based on the accompanying drawings. In the drawings:

[0036] Fig. 1 is a schematic diagram of a method for posture detection and real-time feedback according to the present application;

[0037] Fig. 2 is a schematic diagram of a process for outputting user posture coordinates by a posture estimation model according to the present application;

[0038] Fig. 3 is a schematic diagram of a process for posture similarity algorithm and providing feedback according to the present application;

[0039] Fig. 4 is a schematic diagram of a system for posture detection and real-time feedback according to the present application. DETAILED DESCRIPTION

[0040] In order to more clearly illustrate the objects, technical solutions and advantages of the present application, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings of the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, rather than all the embodiments of the present application. The example embodiments can be implemented in various forms, and should not be understood as being limited to the examples described herein; on the contrary, the example embodiments are provided to make the ideas of the example embodiments fully conveyed to those skilled in the art.

[0041] In addition, the described features, structures or characteristics can be combined in any suitable manner in one or more embodiments. In the following description, many specific details are provided to give a full understanding of the embodiments of the present application. However, those skilled in the art will realize that the technical solutions of the present application can be practiced without one or more of the specific details, or other methods, components, devices, steps, etc. can be used. In other cases, well-known methods, devices, implementations or operations are not shown or described in detail to avoid obscuring the aspects of the present application.

[0042] The block diagrams shown in the drawings are only functional entities, and do not necessarily correspond to physically independent entities. That is, these functional entities can be implemented in the form of software, or in one or more hardware modules or integrated circuits, or in different networks and / or processor devices and / or microcontroller devices.

[0043] The flowcharts shown in the accompanying drawings are merely illustrative and do not necessarily include all content and operations / steps, nor do they necessarily have to be performed in the described order. For example, some operations / steps can be broken down, while others can be combined or partially combined; therefore, the actual execution order may change depending on the specific circumstances.

[0044] The present invention will now be described in detail with reference to specific embodiments:

[0045] Example 1

[0046] As shown in Figure 1, this embodiment provides a method for attitude detection and real-time feedback. The specific steps of this method are as follows:

[0047] S1: When a user performs an exercise, the application captures the current video frame image in real time and sends it to the backend.

[0048] In this embodiment, the user uses an application on a mobile device to perform exercise training. When the user starts to perform exercise movements, the application captures a real-time video stream and extracts video frame images.

[0049] Specifically, when the application starts, it requests user authorization to access the device's camera to ensure that the application has obtained camera access permissions; it uses the mobile device's camera API to capture video streams in real time and starts real-time video preview; for video frame capture, a timer can be set or the video stream's callback mechanism can be used to capture the image of the current frame from the video stream at regular intervals (e.g., 30 frames per second); when extracting video frames, video frames can be extracted from the video stream at a fixed frame rate (e.g., 30 FPS); and the captured images are processed as necessary, such as resizing and grayscale conversion, to adapt to the input requirements of the pose estimation model.

[0050] Furthermore, the resolution and frame rate of the video frames should be set according to actual usage requirements to ensure that the resolution and frame rate are sufficient to support accurate pose estimation; considering the impact of lighting conditions on image quality, automatic exposure adjustment or image enhancement algorithms may need to be added; and the efficiency of image capture and processing should be ensured to avoid affecting the smoothness of the video.

[0051] S2: Using a pose estimation model, estimate the user's pose coordinates based on the video frame images.

[0052] In this embodiment, the user pose coordinates are the coordinates of three-dimensional key points of the body; the pose estimation model receives the current video frame image as input, uses a neural network to extract two-dimensional feature maps of the video frame image, trains the model to predict the coordinates of the three-dimensional key points based on the two-dimensional feature maps, and fits the model on the three-dimensional pose dataset.

[0053] Specifically, a pre-trained pose estimation model is downloaded from a server or loaded from local storage when the application is started or first used; the extracted video frame image is pre-processed and then input into the pose estimation model, and the model outputs the two-dimensional coordinates of the body key points (such as the head, shoulders, knees, etc.).

[0054] Further, as shown in FIG. 2, the pose estimation model receives a video frame image as input, extracts a two-dimensional feature map of each frame image using a CNN, which contains rich spatial information for subsequent three-dimensional pose estimation; a neural network model is trained on a dataset containing a large number of pictures, two-dimensional key point annotations (as features), and corresponding three-dimensional key point coordinates, to reconstruct and predict the X, Y, Z three-dimensional coordinates of each key point from the two-dimensional key point coordinates. Optionally, a statistical model is fitted on the three-dimensional pose dataset, which can map two-dimensional coordinates to three-dimensional coordinates to further enhance the prediction ability of three-dimensional key point coordinates. Optionally, after obtaining the user pose coordinates, non-maximum suppression (NMS) and other techniques can be applied to optimize the key point detection results, and time smoothing and geometric constraints can be used to further improve the stability and accuracy of pose estimation.

[0055] Further, the pose estimation model can be trained end-to-end to directly output three-dimensional poses from input images, avoiding the multi-stage processing in traditional methods.

[0056] S3: Using a pose similarity algorithm, calculate the similarity score of each limb and the overall current action pose based on the standard pose coordinates and the user pose coordinates.

[0057] In this embodiment, the standard poses of each exercise action are collected from a database or other data sources, and the standard poses are converted into three-dimensional standard pose coordinates. The standard poses also include those collected from real-time streaming media for users to obtain immediate feedback. Alternatively, a set of standard pose key point coordinates can be predefined for each exercise action, and a set of standard three-dimensional pose coordinates can be defined according to the requirements of the exercise action, which can be a collection of positions of multiple key points.

[0058] Further, the user pose coordinates and the standard pose coordinates are aligned so that the user pose coordinates and the standard pose coordinates are in the same reference coordinate system. Optionally, affine transformation or similarity transformation and other methods are used to align the user pose coordinates and the standard pose coordinates to the same reference system, minimizing the error between them.

[0059] Optionally, the user's body part positions in the virtual space are adjusted to maximize alignment with the standard pose using a maximum overlap approach; the user's pose is translated and rotated to minimize the spatial difference between the user's pose and the coach's pose; image processing or deep learning techniques can be used to identify and track the user's key body parts, i.e. key points, and then guide the user to adjust the pose by calculating the offset between the user's pose coordinates and the corresponding points in the standard pose coordinates.

[0060] Optionally, the user's pose is aligned with the standard pose using a transformation basis vector approach, in three-dimensional space, basis vectors usually refer to the direction vectors of coordinate axes (such as x, y, z axes); by transforming these basis vectors, the rotation and translation of the entire pose framework can be achieved, so that the user's pose matches the coach's pose; by calculating the appropriate transformation matrix (such as rotation matrix and translation matrix), it can ensure that every detail of the user's pose is accurately aligned with the coach's pose.

[0061] Optionally, the user's pose is aligned with the standard pose using a transformation basis vector approach, in three-dimensional space, basis vectors usually refer to the direction vectors of coordinate axes (such as x, y, z axes); by transforming these basis vectors, the rotation and translation of the entire pose framework can be achieved, so that the user's pose matches the coach's pose; by calculating the appropriate transformation matrix (such as rotation matrix and translation matrix), it can ensure that every detail of the user's pose is accurately aligned with the coach's pose.

[0062] Further, as shown in FIG. 3, the pose similarity algorithm uses limb-by-limb similarity scores to evaluate the similarity between the user's pose coordinates and the standard pose coordinates, including calculating the position vector of each limb key point of the user's pose and the standard pose respectively, and calculating the scalar product of the position vectors to obtain the similarity score of each limb, and then weighting and averaging the similarity scores of each limb to obtain the overall similarity score.

[0063] Specifically, limb-by-limb similarity scores are usually used to evaluate the degree of similarity between two poses (such as a standard pose and a user's pose), and this algorithm usually involves comparing the positions and orientations of corresponding limbs in the poses.

[0064] In this embodiment, first define the key point coordinates of each limb in the user's pose and the standard pose, each limb is represented by two key points (similar to a line segment with two endpoints), forming a position vector. The position vector of each limb is calculated as follows:

[0065] Let the three-dimensional coordinates of the two key points of limb i be K i,1 = (x1, y1, z1) and K i,2= (x2, y2, z2), then the position vector of the limb is: P i = K i,2 = K i,1 = (x2-x1, y2-y1, z2-z1).

[0066] Further, according to the user's limb and the standard limb the scalar product of the two vectors to measure the similarity between the user's pose and the standard pose, where the scalar product ranges from [-1, 1], where 1 represents complete similarity (overlap), and -1 represents complete opposite limb movements. Further, considering the angle and the degree of overlap, the scalar product can be appropriately transformed or combined with other metrics (such as vector length ratio, cosine of the angle, etc.) to more accurately reflect the similarity, and normalized to obtain the similarity score of each limb; the similarity scores of all limbs are weighted and averaged to obtain the overall similarity score between the user's pose and the standard pose, to evaluate the similarity between the two.

[0067] S4: According to the overall similarity score, feedback is given to display the correct and incorrect parts of the current action pose;

[0068] In this embodiment, the overall similarity score is compared with a set threshold to determine whether the current pose is correct or close to correct; the user's pose is displayed in real time on the user interface, and the correct and incorrect parts are marked with different colors or markers.

[0069] Further, if the overall similarity score reaches the set threshold, the current pose is considered correct or close to correct, and positive feedback is provided to the user; if the set threshold is not reached, the current pose needs to be corrected, and corrective feedback is provided to the user. Further, feedback can also be provided to the user according to the similarity score of the limb. Optionally, when it is determined to be correct or close to correct, the interface displays positive feedback such as "Well done!" and "Keep it up!" etc.; when it is determined to be correct, the incorrect parts of the user's pose are highlighted or marked with different colors, and corrective feedback such as "Please adjust the position of your arms" is displayed. Optionally, according to the overall similarity score or the similarity score of each limb, the user is given positive or corrective feedback through a voice assistant or text prompts on the screen.

[0070] Further, the system provides render feedback on frame, which means that feedback information is rendered on every video frame, regardless of whether the similarity score is sufficient or not, so that the user can intuitively see whether his / her posture is correct and make continuous adjustments according to the continuous feedback. Optionally, the real-time image of the user is superimposed with the standard posture model, and the difference is highlighted by color, line or shadow, etc. A virtual "mirror" is created to display the comparison between the user's action and the standard posture, so that the user feels like interacting with his / her mirror image. Animation or arrows can be added to the feedback to indicate how the user should adjust his / her posture or action.

[0071] S5: The application continues to capture the next frame of image until the user ends the exercise action, records and displays the posture count;

[0072] In this embodiment, the operations of steps S1 to S4 are repeated before the user ends the exercise action, and the video frames are continuously captured and processed. The posture counter is increased every time a user successfully completes a standard posture (i.e. the similarity score reaches the set threshold). The user inputs (such as clicking a button) or automatic detection (such as detecting a long period of inaction) are used to determine whether the exercise action is ended, and the results of the posture count are displayed on the user interface for the user's reference.

[0073] Embodiment 2

[0074] As shown in FIG. 4, the embodiment exemplarily presents a posture detection and real-time feedback system, which includes a video capture module, a posture estimation module, a similarity evaluation module, a feedback display module, a counting and recording module, and an interactive control module.

[0075] The video capture module is used to capture video frame images of the user performing the exercise action in real time; the posture estimation module receives the video frame images and estimates the user posture coordinates using a posture estimation model; the similarity evaluation module aligns the user posture coordinates and the standard posture coordinates, and then calculates the similarity score of each limb and the overall similarity score of the current action posture using a posture similarity algorithm; the feedback display module shows the correct and incorrect parts of the current action posture to the user according to the similarity score; the counting and recording module records the number of times the user correctly completes the exercise action each time, and displays the total count when the user ends the exercise; the interactive control module processes the user's interactive instructions, including but not limited to starting, pausing, and ending the exercise.

[0076] Specifically, the video capture module captures video frame images of the user performing the exercise action in real time, uses the camera API of the device to capture the video stream, and extracts the image data of each frame; an interface is implemented to provide the image frame data for subsequent module processing.

[0077] The pose estimation module receives video frame images, estimates the pose coordinates of the user using a pose estimation model, loads a pre-trained pose estimation model, processes the input image frames, and outputs the key point coordinates of the user; an interface is implemented to receive image frames and output user pose coordinate data.

[0078] The similarity evaluation module calculates the similarity score of the current pose using a pose similarity algorithm based on the aligned user pose coordinates and standard pose coordinates; an interface is implemented to receive user pose coordinates and standard pose coordinates and output the similarity score of each limb and the overall similarity score.

[0079] The feedback display module shows the user the correct and incorrect parts of the pose based on the overall similarity score, and uses a graphical interface to superimpose and highlight on the video frame; an interface is implemented to receive the similarity score and user pose coordinates and output visual feedback to the user interface. Further, if the similarity score reaches a set threshold, the current pose is considered correct or close to correct, and positive feedback is provided to the user; if the set threshold is not reached, the current pose is considered to need correction, and corrective feedback is provided to the user.

[0080] The counting and recording module processes user interaction instructions such as starting, pausing, and ending exercise, listens to user input, and calls the functions of the corresponding modules; an interface is implemented to receive user instructions, call the functions of other modules, and display status updates to the user.

[0081] Other embodiments of the application will be apparent to those skilled in the art from consideration of the specification and practice of the application disclosed herein. This application is intended to cover any variations, uses or adaptations of the application following, in general, the principles of the application and including such departures from the present disclosure as come within known or customary practice in the art to which the application pertains. It is intended that the specification and examples be considered as exemplary only, with the true scope and spirit of the application being indicated by the following claims. It is understood that the application is not to be limited to the precise structures and procedures described herein and as such various modifications and changes can be made by those skilled in the art without departing from the scope of the application. The scope of the application is indicated by the appended claims rather than the foregoing description.

Claims

1. A method of posture detection and real-time feedback, characterized in that, The method comprises: The user performs an exercise action, the application program captures the current video frame image in real time, and sends it to the backend; Using a pose estimation model, estimate the user's pose coordinates according to the video frame image; Using a pose similarity algorithm, calculate the similarity score of each limb and the whole of the current action pose according to the standard pose coordinates and the user's pose coordinates; According to the similarity score, feedback is given, and the correct and incorrect parts of the current action pose are displayed; The application program continues to capture the next frame image until the user ends the exercise action, and records and displays the pose count.

2. The method of claim 1, wherein, The user's pose coordinates are three-dimensional key point coordinates; the pose estimation model receives the current video frame image as input, uses a neural network to extract the two-dimensional feature map of the video frame image, trains the model to predict the three-dimensional key point coordinates according to the two-dimensional feature map, and fits on a three-dimensional pose dataset.

3. The method of claim 1, wherein, Collect standard poses for each exercise action from a database or other data sources, and convert the standard poses into three-dimensional standard pose coordinates.

4. The method of claim 3, wherein, The standard poses also include those collected from real-time streaming media for users to obtain immediate feedback.

5. The method of gesture detection and real-time feedback according to claim 1, wherein, Align the user's pose coordinates with the standard pose coordinates so that the user's pose coordinates and the standard pose coordinates are in the same reference coordinate system.

6. The method of gesture detection and real-time feedback according to claim 1, wherein, The pose similarity algorithm uses limb-to-limb similarity scores to evaluate the similarity between the user's pose coordinates and the standard pose coordinates, including calculating the position vector of each limb key point of the user's pose and the standard pose respectively, and calculating the scalar product of the position vectors to obtain the similarity score of each limb. Weighted average of the similarity score of each limb to get the overall similarity score.

7. The method of gesture detection and real-time feedback according to claim 1, wherein, The feedback according to the similarity score includes comparing the similarity score with a set threshold to determine whether the current pose is correct or close to correct; displaying the user's pose in real time on the user interface and marking the correct and incorrect parts with different colors or markers.

8. The method of gesture detection and real-time feedback according to claim 7, wherein, If the similarity score reaches the set threshold, the current pose is considered correct or close to correct, and positive feedback is provided to the user; if the set threshold is not reached, the current pose needs to be corrected, and corrective feedback is provided to the user.

9. The method of gesture detection and real-time feedback according to claim 1, wherein, Before the user ends the exercise action, the application program continuously captures and processes video frames, determines whether the exercise action is ended through user input or automatic detection, and displays the pose count result on the user interface.

10. A system for posture detection and real-time feedback, characterized in that, It includes a video capture module, a pose estimation module, a similarity evaluation module, a feedback display module, a counting and recording module, and an interactive control module. The video capture module is used to capture video frame images of the user performing an exercise action in real time; The pose estimation module receives the video frame image and estimates the user's pose coordinates using a pose estimation model; The similarity evaluation module calculates the similarity score of each limb and the whole of the current action pose using a pose similarity algorithm according to the user's pose coordinates and standard pose coordinates; The feedback display module shows the correct and incorrect parts of the current action pose to the user according to the similarity score; The counting and recording module records the number of times that the user correctly completes the exercise action each time, and displays the total count when the user ends the exercise; The interactive control module processes the interactive instructions of the user, and the interactive instructions include but are not limited to starting, pausing and ending the exercise.

Images