Method and electronic device for measuring an user's posture based on skeleton data
The method corrects skeleton data by replacing outlier joint coordinates with previous frame values, addressing inaccuracies in camera-based posture estimation and enhancing accuracy and reliability.
Patent Information
- Application Number
- KR1020250167539
- Authority / Receiving Office
- KR · KR
- Patent Type
- Patents
- Current Assignee / Owner
- Filing Date
- 2025-11-07
- Publication Date
- 2026-11-10
- Estimated Expiration
- 2045-11-07
AI Technical Summary
Conventional camera-based human posture estimation technologies suffer from inaccuracies due to sensor noise, lighting changes, and joint occlusion, leading to reduced sensitivity and reliability in posture recognition, and lack customization for different types of movements.
A method that corrects skeleton data by identifying and replacing outlier joint coordinate values with those from the previous frame, using a threshold-based comparison of current and average joint positions, and adjusting thresholds based on anatomical characteristics and movement type.
Improves posture measurement accuracy and reliability by reducing the impact of sensor errors and providing customized feedback through time-series-based correction techniques.
Smart Images

Figure 112025124707619-PAT00001_ABST
Abstract
Description
Technology Field
[0001] The present disclosure relates to a method and electronic device for measuring a user's posture based on skeleton data. Specifically, the present disclosure relates to a method and electronic device for measuring a user's posture based on corrected skeleton data after detecting anomalies and correcting the skeleton data. Background Technology
[0002] Camera-based human posture estimation technology estimates the positions of a user's joints to generate skeleton data, which is then utilized for motion analysis, posture evaluation, and healthcare. However, there is a problem in which outliers occur, where the coordinates of some joints deviate from their actual positions due to sensor noise, lighting changes, and joint occlusion, thereby degrading the accuracy of posture recognition results.
[0003] Furthermore, conventional technologies are limited to processing average values based on static thresholds or a fixed number of frames, resulting in reduced sensitivity and reliability during posture changes. Additionally, despite the fact that the joints requiring analysis differ depending on the type of movement, the application of consistent standards makes it difficult to provide customized error analysis or feedback. Therefore, there is a need for technology capable of providing accurate posture measurement and real-time feedback through time-series-based correction techniques and the setting of reference joints for each movement. means of solving the problem
[0004] A method according to one embodiment may include the step of acquiring the skeleton data, which includes the coordinate values of each of the user's plurality of joints, on a frame-by-frame basis through at least one sensor including a camera.
[0005] A method according to one embodiment may include the step of calculating the average coordinate value of each of the coordinate values of a plurality of joints of the user based on the skeleton data obtained from N frames prior to the current frame.
[0006] A method according to one embodiment may include a step of determining whether the difference between the coordinate value of each of the user's multiple joints obtained in the current frame and the average coordinate value exceeds a threshold value.
[0007] A method according to one embodiment may include the step of determining at least one joint coordinate value that exceeds a threshold value among a plurality of joints of the user obtained in the current frame as an outlier.
[0008] A method according to one embodiment may include the step of correcting the skeleton data by replacing at least one joint coordinate value of the user determined to be an outlier with a coordinate value of a joint corresponding to at least one joint determined to be an outlier, which is obtained in the frame immediately preceding the current frame.
[0009] A method according to one embodiment may include the step of measuring the user's posture based on the corrected skeleton data. Brief explanation of the drawing
[0010] FIG. 1 is a flowchart illustrating a method according to one embodiment. FIG. 2 is a flowchart illustrating a method according to one embodiment. FIG. 3 is a flowchart illustrating a method according to one embodiment. FIG. 4 is a flowchart illustrating a method according to one embodiment. FIG. 5 is a flowchart illustrating a method according to one embodiment. FIG. 6 is a flowchart illustrating a method according to one embodiment. FIG. 7 is a flowchart illustrating a method according to one embodiment. FIG. 8 is a flowchart illustrating a method according to one embodiment. FIG. 9 is a flowchart illustrating a method according to one embodiment. FIG. 10 is a flowchart illustrating a method according to one embodiment. FIG. 11 is a block diagram illustrating the components of an electronic device according to one embodiment. Specific details for implementing the invention
[0011] The terms used in this specification will be briefly explained, and the invention will be described in detail.
[0012] The terms used in this invention have been selected based on currently widely used general terms, taking into account their functions within the invention; however, these terms may vary depending on the intent of those skilled in the art, case law, the emergence of new technologies, etc. Additionally, in specific cases, terms have been arbitrarily selected by the applicant, and in such cases, their meanings will be described in detail in the relevant description of the invention. Therefore, the terms used in this invention should be defined not merely by their names, but based on their meanings and the overall content of the invention.
[0013] When a part of a specification is described as "including" a certain component, this means that, unless specifically stated otherwise, it does not exclude other components but may include additional components. Furthermore, terms such as "...part" or "module" as used in the specification refer to a unit that processes at least one function or operation, and this may be implemented in hardware or software, or as a combination of hardware and software.
[0014] Additionally, the description 'at least one of A, B, and C' means that it may be any one of 'A', 'B', 'C', 'A and B', 'A and C', 'B and C', and 'A, B, and C'.
[0015] It should be understood that the combinations of blocks and flowcharts in each flowchart can be executed by one or more computer programs containing computer-executable instructions. One or more computer programs may be stored entirely in a single memory or may be partitioned and stored in multiple different memories.
[0016] All functions or operations described in this document may be processed by a single processor or a combination of processors. A single processor or a combination of processors is a circuitry that performs processing and may include circuitry such as an AP (Application Processor), CP (Communication Processor), GPU (Graphical Processing Unit), NPU (Neural Processing Unit), MPU (Microprocessor Unit), SoC (System on Chip), IC (Integrated Chip), etc.
[0017] A processor may include various processing circuits and / or multiple processors. For example, the term “processor” as used herein, including in the claims, may include at least one processor and various processing circuits. In at least one processor, one or more processors may be configured to perform the various functions described herein in a distributed manner, individually and / or collectively. As used herein, “processor,” “at least one processor,” and “one or more processors” may be configured to perform various functions. However, these terms cover, without limitation, situations where one processor performs some of the functions and other processor(s) perform other parts of the functions, and situations where a single processor can perform all functions. Additionally, at least one processor may include a combination of processors performing various functions of the disclosed functions in a distributed manner. At least one processor may execute program instructions to achieve or perform various functions.
[0018] The following describes embodiments with reference to the attached drawings so that those skilled in the art can easily implement the present invention. However, the present invention may be embodied in various different forms and is not limited to the embodiments described herein. Furthermore, in order to clearly explain the present invention in the drawings, parts unrelated to the explanation have been omitted, and similar parts throughout the specification are denoted by similar reference numerals.
[0019] In the embodiments of this specification, the term "user" may mean a person controlling a system, function, or operation.
[0020] FIG. 1 is a flowchart illustrating a method according to one embodiment.
[0021] In step 110, the method according to one embodiment can acquire skeleton data including coordinate values of each of a plurality of joints of a user on a frame-by-frame basis through at least one sensor including a camera.
[0022] A method according to one embodiment can acquire skeleton data, including coordinate values of each user joint, on a frame-by-frame basis through at least one sensor including a camera, in order to analyze the posture of a user.
[0023] The sensor is an image-based sensing device that enables the estimation of human body joints in a two-dimensional or three-dimensional form, and may be a device including, for example, an RGB camera, an infrared camera, a depth camera, or a combination thereof.
[0024] The sensor can capture the user's movements from various angles, such as the front, side, or rear, and the captured video data can be used to generate time-series position data at the joint level.
[0025] The image acquired by the sensor can be used as input to a skeleton estimation algorithm or a joint estimation network, and can be configured to identify the point in time when the user's posture changes from a static state to a dynamic state, or vice versa.
[0026] The sensor can be a fixed sensor that can be installed in an indoor environment or home training space, can be implemented as a module attached to a user-wearable device, and can be extended to a multi-sensor environment to improve the accuracy of posture analysis per exercise.
[0027] A method according to one embodiment can generate skeleton data by estimating the position of a user's human joints based on image data acquired from a sensor and connecting the position information of each estimated joint.
[0028] A method according to one embodiment may configure the coordinate values of each joint included in the skeleton data into three-dimensional coordinate values for each joint corresponding to an anatomical reference point of the human body. For example, the joints may include at least one of the neck, both shoulders, elbows, wrists, spine, hips, knees, or ankles. However, they are not limited thereto.
[0029] A method according to one embodiment can express the coordinate values of a joint based on a world coordinate system or an image coordinate system composed of an X-axis, a Y-axis, and a Z-axis.
[0030] A method according to one embodiment can identify time-series changes in user posture by acquiring and storing skeleton data for each frame in chronological order.
[0031] A method according to one embodiment can derive the relative positional relationship between joints, joint movement speed, or joint alignment status by analyzing frame-by-frame skeleton data.
[0032] In step 120, the method according to one embodiment can calculate the average coordinate value of each of the coordinate values of a plurality of joints of the user based on skeleton data obtained from N frames prior to the current frame.
[0033] The current frame may refer to a video frame that includes the reference point at which user pose analysis is performed, among the time-series frames in which skeleton data is continuously acquired.
[0034] The current frame may be a single frame corresponding to a specific point in time in video captured in real time by a sensor, for example, an RGB camera or a depth camera.
[0035] The current frame may refer to the most recently acquired frame among video frames that are accumulated and stored chronologically along with past frames. For example, while a user is performing an action of raising or lowering their arms, the frame captured at the last point of that action can be identified as the current frame. For instance, if a user is performing a squat, the frame containing the moment when the knees reach their lowest point can be the current frame.
[0036] The current frame can be set as a reference point for determining the amount of change, direction of movement, or whether there is a motion error by comparing it with multiple previously acquired frames.
[0037] A method according to one embodiment can extract coordinate values corresponding to joints of the same type from N frames, respectively, and calculate an average coordinate value by considering the temporal position change of the joint. N may mean an integer greater than or equal to 2.
[0038] A method according to one embodiment defines the coordinate values of each joint as X-axis, Y-axis, and Z-axis position values in three-dimensional space, and can calculate the average coordinate values as the average position values for each of the X-axis, Y-axis, and Z-axis directions.
[0039] A method according to one embodiment may define the average coordinate value as the result of dividing the cumulative position value of coordinate values obtained for the same joint in a plurality of frames arranged in chronological order by the number of joints.
[0040] For example, in the case of the right shoulder joint, the average coordinate value may consist of the average of the X-axis position values, the average of the Y-axis position values, and the average of the Z-axis position values extracted from each of the N frames containing the joint coordinate values corresponding to the right shoulder.
[0041] A method according to one embodiment can provide reference position information in which the average coordinate value is less affected by abrupt changes in coordinate values caused by temporary sensor errors or joint estimation failures.
[0042] A method according to one embodiment analyzes the difference between the joint coordinate value at the current time point and the average coordinate value, and can be used as a comparison standard to determine the stability or abnormality of user operation.
[0043] A method according to one embodiment may set the number of frames N used when calculating average coordinate values to a fixed value, or dynamically set a new reference frame based on the point where a sudden change in user action is detected.
[0044] In step 130, the method according to one embodiment can determine whether the difference between the coordinate value of each of the user's multiple joints obtained in the current frame and the average coordinate value exceeds a threshold value.
[0045] A method according to one embodiment can determine whether the difference between the coordinate values of each joint exceeds a preset threshold by comparing the coordinate values of each user joint included in a frame acquired at the current time with the average coordinate values calculated based on previous frames.
[0046] A method according to one embodiment can calculate the Euclidean distance between the coordinate value obtained in the current frame and the average coordinate value for joints of the same type, and compare the calculated distance value with a threshold value corresponding to the joint.
[0047] A method according to one embodiment may define the coordinate values of a joint based on a three-dimensional coordinate system composed of the X-axis, Y-axis, and Z-axis to calculate the Euclidean distance.
[0048] A method according to one embodiment may determine that the current position of a joint has moved rapidly from its average position if the difference in distance between joint coordinate values is greater than or equal to a threshold value corresponding to the joint.
[0049] A method according to one embodiment can establish criteria for determining position change that reflect the anatomical characteristics and range of motion of each joint by setting different threshold values for each joint.
[0050] For example, a method according to one embodiment may set a relatively large threshold value for joints with frequent movement, such as the wrist or ankle, and set a relatively small threshold value for joints with a limited range of motion, such as the spine or hip joint. However, it is not limited thereto.
[0051] In step 140, the method according to one embodiment can determine at least one joint coordinate value that exceeds a threshold value among a plurality of user joints obtained in the current frame as an outlier.
[0052] A method according to one embodiment can determine whether there is an abnormality in a user joint based on the difference between the average coordinate value calculated based on previous frames and the coordinate value of each user joint included in a frame acquired at the current time.
[0053] A method according to one embodiment calculates the three-dimensional distance difference between the user joint coordinate value obtained in the current frame and the average coordinate value for the same type of user joint, and if the three-dimensional distance difference is greater than or equal to a threshold value set for each joint, the user joint coordinate value can be determined as an outlier.
[0054] A method according to one embodiment may apply the Euclidean distance formula based on the coordinate values of the user joint to calculate the distance difference.
[0055] For example, if the coordinate value of the elbow joint deviates from the average position by more than a threshold value, the method according to one embodiment can classify the elbow joint coordinate value as an outlier.
[0056] For example, if the coordinate value of an elbow joint deviates by more than 30 millimeters from the average coordinate value, the method according to one embodiment may classify the elbow joint coordinate value as an outlier. For example, if the coordinate value of a knee joint deviates by more than 20 millimeters from the average coordinate value, the method according to one embodiment may determine the knee joint coordinate value as an outlier. However, it is not limited thereto.
[0057] A method according to one embodiment may set a distance threshold from the average position for each joint differently depending on the anatomical characteristics and range of motion of the joint.
[0058] For example, the method according to one embodiment may set a distance threshold of 40 millimeters for the wrist joint and 10 millimeters for the spine joint. However, it is not limited thereto.
[0059] A method according to one embodiment can determine the coordinate value of a knee joint or hip joint as an outlier if the joint coordinate value in the current frame deviates sharply from the average coordinate value due to instability in skeleton estimation or failure of joint detection at the knee joint or hip joint.
[0060] A method according to one embodiment stores the result of determining user joint coordinate values that are judged to be outliers and can be utilized to evaluate the stability of the user posture or to improve the reliability of skeleton data.
[0061] In step 150, the method according to one embodiment can correct skeleton data by replacing at least one joint coordinate value of a user determined to be an outlier with a coordinate value of a joint corresponding to at least one joint determined to be an outlier, which is obtained in the frame immediately preceding the current frame.
[0062] A method according to one embodiment can correct skeleton data by replacing a joint coordinate value determined to be an outlier among the coordinate values of each user joint included in a frame acquired at the current time with a user joint coordinate value of the same type included in a frame acquired one frame earlier than the current time.
[0063] A method according to one embodiment can search for a coordinate value corresponding to the same joint in the previous frame based on the joint name or index of a joint determined to be an outlier, and overwrite the location of the outlier coordinate value in the current frame with the corresponding coordinate value.
[0064] A method according to one embodiment may preferentially refer to user joint coordinate values obtained in the previous frame if it is determined that such values have higher sensor reliability than the previous average coordinate values and more accurately reflect the flow of changes in the actual user's posture.
[0065] For example, in a method according to one embodiment, if the right wrist joint coordinate value in the current frame deviates by more than 40 millimeters from the average position and is determined to be an outlier, the right wrist joint coordinate value obtained in the previous frame can be replaced with the wrist joint coordinate value of the current frame.
[0066] For example, the method according to one embodiment can correct skeleton data by applying the left hip joint coordinate value of the previous frame instead of the coordinate value of the current frame when the coordinate value of the left hip joint changes abruptly in the current frame. However, it is not limited thereto.
[0067] In step 160, the method according to one embodiment can measure the user's posture based on corrected skeleton data.
[0068] A method according to one embodiment can measure the posture state of a user based on skeleton data corrected by removing outliers.
[0069] A method according to one embodiment can determine the positional relationship or alignment state between specific parts of a user's body by using each joint coordinate value included in the corrected skeleton data.
[0070] A method according to one embodiment can measure whether there is left-right asymmetry in shoulder alignment by analyzing the height and depth differences of the left and right shoulder joints.
[0071] A method according to one embodiment can determine whether the pelvic alignment is horizontal or the anterior-posterior tilt state by analyzing the height and depth differences of the left and right hip joints.
[0072] A method according to one embodiment can determine whether the user's neck is aligned in a vertical direction or tilted in a specific direction based on the relative positions of the head, neck, and shoulder joints.
[0073] A method according to one embodiment can determine that a lateral tilt of the neck exists when the neck joint coordinate value deviates by more than a certain distance from the centerline of the shoulder joint coordinate value.
[0074] A method according to one embodiment can determine the forward or backward tilt of the neck when the depth difference between the head joint and the shoulder joint deviates from a reference range.
[0075] A method according to one embodiment can analyze the user's static posture or postural stability during exercise performance by calculating the spatial relationship between the user's body parts based on the joint coordinate values corrected in this way and converting the calculation result into a posture evaluation index.
[0076] FIG. 2 is a flowchart illustrating a method according to one embodiment.
[0077] Steps 110, 120, and 130 of Fig. 2 correspond to steps 110, 120, and 130 of Fig. 1, respectively.
[0078] In step 210, the method according to one embodiment can identify a transformation frame in which the user's posture is transformed through a camera.
[0079] A method according to one embodiment can identify a frame containing a point in time when a user posture is converted from a static state to a dynamic state or from a dynamic state to a static state as a conversion frame, based on a time series change of skeleton data continuously acquired from a camera.
[0080] A transformation frame may refer to a frame that includes the point in time when the user's posture changes from a static state to a dynamic state or from a dynamic state to a static state among the time-series changes of skeleton data continuously acquired from a camera.
[0081] A transformation frame may refer to a frame that includes a point in time when the amount of change in the distance traveled or the velocity of a coordinate value at at least one of a plurality of joints increases or decreases by more than a set reference value.
[0082] A method according to one embodiment can determine a transformed frame by analyzing time-series position data of each joint coordinate value and determining a frame in which the amount of change in distance or speed at one or more of a plurality of joints increases or decreases by more than a preset change criterion.
[0083] For example, a method according to one embodiment can identify a frame containing the point in time when the movement occurred as a transformed frame when the coordinate values of a hip joint or a spine joint remain within a certain range in a series of frames and then suddenly move beyond a certain distance.
[0084] For example, a method according to one embodiment can identify a frame containing an inflection point of movement as a transformation frame when the amount of time-series positional change of a shoulder or knee joint changes from a fine movement to a large movement. However, it is not limited thereto.
[0085] A method according to one embodiment can set a range for calculating average coordinate values based on a transformation frame, distinguish movement phase intervals, or adjust outlier detection conditions.
[0086] A method according to one embodiment stores the results of identifying the transformed frames and can use them as a standard for setting the posture analysis range based on the start or end time of a specific exercise motion.
[0087] In step 220, the method according to one embodiment can determine the number of frames for calculating the average coordinate value of each of the user's multiple joints based on the transformation frame.
[0088] A method according to one embodiment can determine the number of frames for calculating average coordinate values based on a transformation frame.
[0089] A method according to one embodiment can identify a section where the movement of each user joint is stabilized based on a transformation frame, and set the number of consecutive frames included in the section as the number of frames N for calculating the average coordinate value.
[0090] A method according to one embodiment can continuously calculate the difference between frames of joint coordinate values in chronological order.
[0091] A method according to one embodiment may determine a frame containing such a state as a transformation frame, which is a reference point for setting the number of frames for calculating average coordinate values, when the rate of change of the movement distance, speed, or direction of movement of a joint is maintained at or below a preset reference value for a certain period of time.
[0092] For example, in a method according to one embodiment, if the amount of change in the coordinate value of the right knee joint is 3 millimeters or less over 5 consecutive frames, 5 frames may be set as the number of frames for calculating the average coordinate value. For example, in a method according to one embodiment, if the rate of change in the coordinate value of both shoulder joints is maintained at 10 millimeters or less per second for 100 milliseconds or more, the number of frames may be set based on the number of frames included in that interval. However, it is not limited thereto.
[0093] A method according to one embodiment can independently determine the frame stability of each joint by analyzing the frame-to-frame change amount of joint coordinate values collected time-series for each joint.
[0094] A method according to one embodiment can calculate a frame segment maintained in a stable state for each of a plurality of joints and determine the total number of frames to be used for calculating the average coordinate value based on the number of frames in which stability is secured in common across the plurality of joints.
[0095] A method according to one embodiment may set different frame stability judgment conditions or time intervals depending on whether the user's exercise type is a static posture maintenance movement or a repetitive movement performance movement.
[0096] For example, in static movements that maintain a specific posture, such as a plank, the method according to one embodiment may utilize a frame interval that is stably maintained for 20 frames or more for calculating the average coordinate value. For example, in exercises where the movement is repeated, such as a squat, the method according to one embodiment may set only a short stable interval of 5 to 10 frames as a standard for calculating the average coordinate value. However, it is not limited thereto.
[0097] In step 230, the method according to one embodiment can calculate the average coordinate value of each of the coordinate values of a plurality of joints of the user based on skeleton data obtained from a determined number of frames.
[0098] A method according to one embodiment can calculate an average coordinate value using the coordinate values of each user joint included in the skeleton data obtained from a determined number of consecutive frames.
[0099] A method according to one embodiment can calculate an average coordinate value by calculating the average of coordinate values recorded for each determined frame for the same type of user joint.
[0100] A method according to one embodiment calculates an average coordinate value for each of a plurality of joints in the same manner and can then be used as a reference value for comparison with the joint coordinate value obtained in the current frame.
[0101] A method according to one embodiment can update average coordinate values in real time or calculate them locally, limited to sections where posture analysis is required.
[0102] The method according to one embodiment can utilize the average coordinate value calculated in this way as a reference coordinate for outlier detection and correction operations by comparing the difference with the current joint coordinate value.
[0103] FIG. 3 is a flowchart illustrating a method according to one embodiment.
[0104] Steps 140, 150, and 160 of Fig. 3 correspond to steps 140, 150, and 160 of Fig. 1, respectively.
[0105] In step 310, the method according to one embodiment can calculate a matching deviation by comparing the image of the user obtained in the current frame through the camera with the coordinate values of the joint corresponding to at least one joint of the user determined to be an outlier.
[0106] A method according to one embodiment can calculate a matching deviation by comparing the coordinates on the image where a joint of the same type as a user joint determined to be an outlier is expected to be located with the coordinates on the skeleton data, based on a frame-by-frame user image acquired at the current time through a camera.
[0107] Matching deviation may refer to the degree to which the same type of user joint positions do not match between video images and skeleton data at the same point in time. Matching deviation may be a quantitative value representing the difference in distance or direction between the joint coordinate values included in the skeleton data and the corresponding joint positions derived through image analysis.
[0108] A method according to one embodiment can obtain image-based reference coordinate values by identifying a body part containing a joint determined to be an outlier in an image and analyzing the contour, color distribution, or movement features of the said part.
[0109] For example, if the wrist joint is determined to be an outlier, the method according to one embodiment can calculate an image-based wrist coordinate value at the intersection of straight lines extending the contours of the fingertips and elbows, or at the location where a skin color boundary is formed. For example, if the knee joint is determined to be an outlier, the method according to one embodiment can estimate image-based knee coordinates by searching the flexion region between the thigh and the calf. However, it is not limited thereto.
[0110] A method according to one embodiment can determine that if the alignment deviation is greater than or equal to a preset reference value, the skeleton coordinate value is inconsistent with the image and can be determined as an outlier subject to correction.
[0111] A method according to one embodiment records alignment deviation values for each joint and can be used for correction history or skeleton reliability evaluation.
[0112] In step 320, the method according to one embodiment can obtain corrected coordinate values by correcting the coordinate values of a joint corresponding to at least one joint that is determined to be an outlier and obtained from the frame immediately preceding the current frame based on the alignment deviation.
[0113] A method according to one embodiment can obtain corrected coordinate values by reflecting the alignment deviation and correcting the joint coordinate values of the same type as the user joints determined to be outliers among the skeleton data obtained in the frame immediately preceding the current frame.
[0114] The corrected coordinate value may refer to a new joint coordinate value calculated using the difference between the reference position estimated through image analysis and the coordinate value of the same joint included in the skeleton data acquired in the frame immediately preceding the current frame as the correction standard.
[0115] A method according to one embodiment can consider the magnitude and direction of the alignment deviation and determine the direction of movement to a reference position in order to calculate a correction coordinate value.
[0116] A method according to one embodiment can calculate a corrected coordinate value by applying a correction coefficient and reflecting the vector between the past coordinate value and the reference position by a certain ratio.
[0117] For example, in the case where the wrist joint is determined to be an outlier, the method according to one embodiment may set the coordinates moved by 50% toward the hand position in the image, based on the wrist coordinate values obtained in the previous frame, as the corrected coordinate values. For example, in the case where the knee joint is determined to be an outlier, the method according to one embodiment may apply a correction coefficient proportional to the distance difference between the image-based knee coordinates and the past knee coordinates, and set the coordinate values reflecting only a portion of the distance difference as the corrected coordinate values. However, it is not limited thereto.
[0118] A method according to one embodiment can variably set a correction factor according to the magnitude of the match deviation. For example, a method according to one embodiment can set the correction factor to 0.7 when the match deviation is large, and to 0.3 when the match deviation is small.
[0119] A method according to one embodiment can verify whether the calculated correction coordinate value is not an abnormal position in terms of body structure, and only if verified, replace the outlier coordinate value in the skeleton data with the correction coordinate value.
[0120] In step 330, the method according to one embodiment can correct skeleton data by replacing at least one joint coordinate value of the user, which is determined to be an outlier, with a corrected coordinate value.
[0121] A method according to one embodiment can correct skeleton data by replacing the corrected coordinate value calculated by reflecting the alignment deviation with the coordinate value of a user joint determined to be an outlier.
[0122] A method according to one embodiment can identify a location where a joint coordinate value of the same type is recorded within skeleton data based on the type of user joint determined to be an outlier, and replace the recorded coordinate value with a corrected coordinate value.
[0123] For example, in the case where the skeleton data is in a JSON format consisting of key-value pairs, the method according to one embodiment may change the coordinate value included in the "left_knee" key to a corrected coordinate value if the left knee joint is determined to be an outlier. For example, in the case where the skeleton data is in an array structure having fixed indices, the method according to one embodiment may replace the coordinate value stored at a specific index with a corrected coordinate value if the right wrist joint is mapped to that index. However, it is not limited thereto.
[0124] A method according to one embodiment can preserve the original quality of skeleton data by selectively correcting only the coordinate values of the joints determined to be outliers while maintaining all joint coordinate values other than those determined to be outliers.
[0125] A method according to one embodiment may store the result of determining whether an outlier is present and whether correction is applied together as metadata in the skeleton data.
[0126] A method according to one embodiment can lead to a step of user posture evaluation, exercise accuracy analysis, or real-time feedback provision based on the skeleton data updated in this way.
[0127] FIG. 4 is a flowchart illustrating a method according to one embodiment.
[0128] Steps 150 and 160 of Fig. 4 correspond to steps 150 and 160 of Fig. 3, respectively.
[0129] In step 410, the method according to one embodiment can calculate the height difference and depth difference of the user's left and right shoulder joints from the user's shoulder joint coordinate values included in the skeleton data.
[0130] A method according to one embodiment can calculate the height difference and depth difference of the left and right shoulder joints based on the coordinate values of the user's left shoulder joint and right shoulder joint, respectively, included in the skeleton data.
[0131] The coordinate values of the left shoulder joint and the right shoulder joint, respectively, may include a y-axis coordinate value containing a vertical component and a z-axis coordinate value containing an anterior-posterior component.
[0132] A method according to one embodiment can calculate the height difference by calculating the absolute value of the difference between the y-axis coordinate value of the left shoulder joint and the y-axis coordinate value of the right shoulder joint.
[0133] A method according to one embodiment can calculate the depth difference by calculating the absolute value of the difference between the z-axis coordinate value of the left shoulder joint and the z-axis coordinate value of the right shoulder joint.
[0134] A method according to one embodiment may determine that the alignment of the left and right shoulders is likely to be asymmetrical if the height difference or depth difference between the left and right shoulder joints is greater than or equal to a preset reference value.
[0135] Calculations of height and depth differences can be used as basic data to determine the tilt of the user's posture, whether one shoulder is elevated or protruding forward, etc.
[0136] In step 420, the method according to one embodiment can determine whether the user's shoulders are asymmetrical based on the height difference and depth difference of the user's left and right shoulder joints.
[0137] A method according to one embodiment can determine whether the user's shoulder is asymmetric in the horizontal or front-back direction based on the height difference and depth difference calculated based on the coordinate values of the left shoulder joint and the right shoulder joint, respectively.
[0138] A method according to one embodiment calculates the absolute difference between the y-axis coordinate value of the left shoulder joint and the y-axis coordinate value of the right shoulder joint, and if the height difference is greater than or equal to a preset reference value, it can be determined as vertical asymmetry.
[0139] A method according to one embodiment calculates the absolute difference between the z-axis coordinate value of the left shoulder joint and the z-axis coordinate value of the right shoulder joint, and if the depth difference is greater than or equal to a reference value, it can be determined asymmetry in the anterior-posterior direction.
[0140] For example, the method according to one embodiment can classify the shoulder asymmetry in both the vertical and front-back directions when the y-axis coordinate of the left shoulder is 2 cm or more higher than the right shoulder and the z-axis coordinate is 1.5 cm or more forward. However, it is not limited thereto.
[0141] A method according to one embodiment can determine a shoulder asymmetry state if at least one of the height difference and the depth difference exceeds a reference value.
[0142] FIG. 5 is a flowchart illustrating a method according to one embodiment.
[0143] Steps 150 and 160 of Fig. 5 correspond to steps 150 and 160 of Fig. 4, respectively.
[0144] In step 510, the method according to one embodiment can calculate the height difference and depth difference of the user's left and right hip joints from the user's hip joint coordinate values included in the skeleton data.
[0145] A method according to one embodiment can calculate the height difference and depth difference between the left and right hip joints by using the user's left hip joint coordinate values and right hip joint coordinate values included in the skeleton data.
[0146] A method according to one embodiment can calculate the height difference between the left and right hip joints by referring to the y-axis coordinate value corresponding to the vertical component of each of the left hip joint coordinate value and the right hip joint coordinate value.
[0147] A method according to one embodiment can calculate the difference in depth between the left and right hip joints by referring to the z-axis coordinate values corresponding to the anterior-posterior direction components of the left hip joint coordinate values and the right hip joint coordinate values, respectively.
[0148] A method according to one embodiment can calculate the height difference by calculating the absolute value of the difference between the y-axis coordinate value of the left hip joint and the y-axis coordinate value of the right hip joint.
[0149] A method according to one embodiment can calculate the depth difference by calculating the absolute value of the difference between the z-axis coordinate value of the left hip joint and the z-axis coordinate value of the right hip joint.
[0150] For example, the method according to one embodiment can determine the possibility that the user's pelvis is asymmetric in both the vertical and anterior-posterior directions if the y-axis coordinate value of the left hip joint is 3 cm lower than the right hip joint and the z-axis coordinate value is located 2 cm further back. However, it is not limited thereto.
[0151] In step 520, the method according to one embodiment can determine whether the user's pelvis is asymmetric based on the height difference and depth difference of the user's left and right hip joints.
[0152] A method according to one embodiment can determine whether the user's pelvic alignment is asymmetrical based on the height difference and depth difference calculated from the left hip joint coordinate value and the right hip joint coordinate value.
[0153] A method according to one embodiment can calculate the height difference by calculating the absolute value of the difference between y-axis coordinate values representing vertical components.
[0154] A method according to one embodiment can calculate the depth difference by calculating the absolute value of the difference between z-axis coordinate values representing the front-rear direction components.
[0155] A method according to one embodiment can determine that the pelvis is not aligned in the vertical or front-back direction if at least one of the height difference or the depth difference is greater than or equal to a reference value.
[0156] For example, the method according to one embodiment can classify a pelvic asymmetry condition if the y-axis coordinate value of the left hip joint is 2.5 cm or more lower than the right hip joint, or if the z-axis coordinate value is located 2 cm or more behind. However, it is not limited thereto.
[0157] FIG. 6 is a flowchart illustrating a method according to one embodiment.
[0158] Steps 150 and 160 of Fig. 6 correspond to steps 150 and 160 of Fig. 5, respectively.
[0159] In step 610, the method according to one embodiment can determine the alignment status of the user's head, neck, and shoulders through skeleton data.
[0160] A method according to one embodiment can determine whether the three joints are aligned vertically or horizontally based on the user's head joint coordinates, neck joint coordinates, and left and right shoulder joint coordinates included in the skeleton data.
[0161] A method according to one embodiment can calculate the slope or intersection angle between a virtual line segment connecting the head joint coordinate value and the neck joint coordinate value, and line segments connecting the neck joint coordinate value and the left and right shoulder joint coordinate values.
[0162] A method according to one embodiment can determine that the alignment state is good when the left and right shoulder joint coordinate values have similar values based on the y-axis, the neck joint is located at the center of the shoulder joint, and the head joint is located close to the vertical line above the neck joint.
[0163] A method according to one embodiment can determine that the alignment of the head or neck is unbalanced if the head joint is deviated to the left or right and the line segment between the neck joint and the shoulder joint is tilted by more than a certain angle.
[0164] For example, the method according to one embodiment may classify the alignment of the head and shoulders as poor if the x-axis coordinate value of the head joint deviates by more than 3 cm to the left or right compared to the neck joint, and the y-axis difference between the left and right shoulder joints is more than 2 cm. However, it is not limited thereto.
[0165] A method according to one embodiment can store the alignment status determination result as one of the detail evaluation items.
[0166] In step 620, the method according to one embodiment can determine the tilt of the user's neck based on the alignment state of the user's head, neck, and shoulders.
[0167] A method according to one embodiment can determine how much a user's neck is tilted from a vertical reference line based on the positional relationship of head joint coordinate values, neck joint coordinate values, and left and right shoulder joint coordinate values included in skeleton data.
[0168] A method according to one embodiment can determine whether the neck is tilted to the left or right by calculating the angle formed by a virtual line segment connecting the head joint coordinate value and the neck joint coordinate value with the vertical axis in the direction of gravity.
[0169] A method according to one embodiment can analyze in which direction the neck is biased relative to the horizontal reference line by comparing the neck joint coordinates with the horizontal reference line generated based on the left and right shoulder joint coordinates.
[0170] A method according to one embodiment can also determine whether a turtle neck condition exists by detecting when the head joint protrudes forward (in the positive direction of the z-axis) by a certain distance or more than the neck joint, that is, when the neck is tilted forward.
[0171] For example, the method according to one embodiment can classify a condition as a turtle neck if the z-axis coordinate value of the head joint is located at least 5 cm ahead of the z-axis coordinate value of the neck joint and the line segment connecting the two joints is tilted at least 15 degrees with respect to the vertical axis. However, it is not limited thereto.
[0172] A method according to one embodiment can determine a laterally tilted posture when the height difference between the left and right shoulder joints is 2 cm or more and the neck joint is biased toward the lower side.
[0173] FIG. 7 is a flowchart illustrating a method according to one embodiment.
[0174] Step 160 of Fig. 7 corresponds to Step 160 of Fig. 6.
[0175] In step 710, the method according to one embodiment can output an exercise list consisting of a plurality of exercise items to a display.
[0176] A method according to one embodiment can generate an exercise list including various types of exercise items that a user can perform, and output the exercise list to a display.
[0177] The exercise list may consist of multiple exercise items, including at least one of strength training, aerobic exercise, stretching, or rehabilitation exercise.
[0178] For example, the exercise list may include push-ups, lunges, planks, cycling, treadmill walking, hamstring stretches, calf stretches, shoulder rotation exercises, etc.
[0179] A method according to one embodiment can visualize and output each exercise item on a user interface of a display in the form of text, icons, videos, or animations.
[0180] The exercise list can be provided by listing multiple items simultaneously on the screen, and can be classified and displayed by difficulty level, body part, or purpose.
[0181] A method according to one embodiment can adjust the order of items in an exercise list or highlight specific items based on the user's past exercise history or set goals.
[0182] In step 720, the method according to one embodiment may obtain user input selecting one of a plurality of exercise items included in an exercise list.
[0183] A method according to one embodiment can obtain an input signal to identify one exercise item selected by a user among a plurality of exercise items included in an exercise list.
[0184] User input can be obtained through at least one of a touch gesture on a touch display, a mouse click, a voice command, or hand movement recognition using a gesture recognition sensor.
[0185] For example, when a user touches a 'squat' item on a display with a finger, the method according to one embodiment can recognize an exercise item that matches the touch coordinate values as the selected item. For example, when a user speaks a voice command such as "start lunge exercise," the method according to one embodiment can convert the voice command into text and determine a specific exercise item as the selected item through keyword analysis. However, it is not limited thereto.
[0186] A method according to one embodiment can detect a user's selection input in real time.
[0187] In step 730, the method according to one embodiment can output exercise information corresponding to an exercise item selected according to user input to a display.
[0188] A method according to one embodiment can output exercise information corresponding to a selected exercise item to a display device based on an exercise item selected by a user from a plurality of exercise items.
[0189] Exercise information may include at least one of the starting position, range of motion, number of repetitions, precautions, breathing method, or exercise intensity guidance of the selected exercise item. However, it is not limited thereto.
[0190] A method according to one embodiment can visualize motion information in the form of a static image, text, animation, or video and output it to the screen of a display device.
[0191] For example, if a user selects a 'lunge' exercise item, the method according to one embodiment may output video or graphic data including information regarding the angles of the front knee and back leg, upper body posture, knee position, and breathing cycle during lunge performance to a display device.
[0192] For example, a method according to one embodiment may provide text information and voice output to a display device guiding the stretching holding time, recommendations for maintaining a non-rebounding posture, and breathing rhythms for tension relief when a user selects the 'hamstring stretching' exercise item. However, it is not limited thereto.
[0193] A method according to one embodiment can optimize output quality by automatically adjusting the screen size, brightness, playback speed, or repetition cycle of a display device according to the type of motion information.
[0194] FIG. 8 is a flowchart illustrating a method according to one embodiment.
[0195] Step 730 of Fig. 8 corresponds to Step 730 of Fig. 7.
[0196] In step 810, the method according to one embodiment can determine a reference joint to determine whether the user is performing the exercise correctly according to the exercise information.
[0197] A method according to one embodiment can determine one or more reference joints to determine whether the user is performing the exercise correctly, based on the content of exercise information included in an exercise item selected from an exercise list.
[0198] A reference joint is a joint directly connected to a body part that constitutes the main movement of a selected exercise item, and may refer to a joint that serves as a standard for judging the accuracy and alignment status of exercise performance.
[0199] The reference joint may be, for example, at least one of the elbow joint, knee joint, hip joint, shoulder joint, ankle joint, or wrist joint, and may vary depending on the exercise item.
[0200] A method according to one embodiment can set a joint with high judgment reliability as a reference joint based on major movement parts, movement direction, joint angle limitations, or load conditions included in movement information.
[0201] For example, if a user selects a squat exercise, the method according to one embodiment may determine the knee joint, hip joint, and ankle joint as reference joints. For example, if a user selects a shoulder rotation stretch, the method according to one embodiment may set the shoulder joint and elbow joint as reference joints. However, it is not limited thereto.
[0202] In step 820, the method according to one embodiment can extract user joint coordinate values corresponding to reference joints from skeleton data.
[0203] A method according to one embodiment can extract three-dimensional coordinate values of a joint that matches the reference joint among a plurality of joints included in skeleton data after the reference joint is determined.
[0204] Reference joints correspond to specific parts of the user's body, and the skeleton data may include the frame-unit position of each joint as x-axis, y-axis, and z-axis coordinate values.
[0205] A method according to one embodiment can search for joints having the same identifier within skeleton data based on the name or index of a reference joint, and extract the coordinate values of the user corresponding to the searched joint.
[0206] For example, if the reference joint is set to the 'left knee joint', the method according to one embodiment can extract the x-coordinate, y-coordinate, and z-coordinate values of the joint mapped to the 'left knee joint' in the skeleton data on a frame-by-frame basis.
[0207] In the case where there are multiple reference joints, the method according to one embodiment can extract multiple joint coordinate values corresponding to each reference joint.
[0208] The extracted joint coordinate values can be used as information to analyze the position, direction of movement, or rate of change between frames of a reference joint.
[0209] In step 830, the method according to one embodiment can identify at least one joint of the user where a motion error has occurred based on the user's joint coordinate values corresponding to the reference joint.
[0210] A method according to one embodiment can identify at least one joint of a user in which a motion error has occurred by comparing the joint coordinate values of a user corresponding to one or more joints set as reference joints with reference posture data included in motion information.
[0211] Motion error may refer to a numerical value representing the difference from a reference posture during the user's exercise performance. Motion error can be calculated based on the coordinate values of each of multiple joints or the geometric relationships between multiple joints.
[0212] Motion error can be determined based on at least one of the position error relative to the reference posture, rotation angle error, distance error, alignment error, or error in the rate of change between frames.
[0213] The operation error may include at least one of a position error of a user's multiple joints, a rotation angle error of a multiple joint, a distance error between a multiple joint, an alignment error of a multiple joint, or a velocity error of the time-series positional change of a multiple joint.
[0214] Position error can refer to the difference in distance between the target coordinates defined in a reference position and the actual measured coordinates for a single joint. For example, when the reference coordinate of the wrist joint is 500 millimeters in the x-axis direction but the actual coordinate is 550 millimeters, the position error can be calculated as 50 millimeters.
[0215] The rotation angle error can indicate the degree to which the relative rotation angle between two connected joints differs from the reference rotation angle. For example, if the angle formed by the shoulder joint and the elbow joint is 90 degrees in the reference position but 120 degrees in the actual position, the rotation angle error can be defined as 30 degrees.
[0216] Distance error can represent the difference between the actual distance and the reference distance between two joints. For example, if the reference distance between the knee joint and the ankle joint is 400 millimeters but the actual measured distance is 450 millimeters, the distance error can be defined as 50 millimeters.
[0217] Alignment error may refer to the degree to which multiple joints do not coincide with a reference axis or reference alignment direction. For example, in a reference posture where the head, neck, and spine joints are to be vertically aligned with respect to the y-axis, if the neck joint deviates by more than 30 millimeters from the y-axis, it may be determined that an alignment error has occurred.
[0218] The velocity error of time-series positional variation may refer to the degree to which the speed at which a single joint coordinate value moves over time is inconsistent with the reference speed, or where sudden acceleration or deceleration occurs. For example, if the speed of movement of the hip joint becomes 1.5 times faster or slower than the reference speed during walking exercise, or if the speed changes suddenly while repeating the same posture, it may be determined that a velocity error has occurred.
[0219] A method according to one embodiment can determine whether a user operation error has occurred by combining at least one of a position error, a rotation angle error, a distance error, an alignment error, and a time series velocity error.
[0220] A method according to one embodiment can determine that a joint is an error joint if the joint coordinate value corresponding to a reference joint exceeds a predetermined error range with respect to the position coordinate value of a reference posture, or if the difference between the alignment direction of a reference posture and the real-time joint alignment direction exceeds a set allowable range.
[0221] For example, if the user's knee joint is positioned further forward than the toes or the hip joint is not parallel to the ground during a squat, the method according to one embodiment can identify that a motion error has occurred at the knee joint or the hip joint. For example, if the rate of change in distance between the shoulder joint and the elbow joint is measured to be smaller than a reference value when performing a shoulder rotation stretch, the method according to one embodiment can determine that a motion error has occurred at the shoulder joint. However, it is not limited thereto.
[0222] In step 840, the method according to one embodiment can provide feedback on operation errors.
[0223] A method according to one embodiment can generate posture correction information for correcting a motion error by comparing a reference posture with the joint coordinate values of a user in whom a motion error has occurred, and can visually provide the posture correction information to the user through a display device.
[0224] The posture correction information may include at least one of the direction of movement, direction of rotation, direction of alignment correction, or method of adjusting the intensity of movement of the joint where a motion error occurred.
[0225] For example, if the user's right knee joint moves excessively forward from the reference position, the method according to one embodiment may output text or voice feedback such as "Pull your right knee back" or "Do not let your knee go past your toes."
[0226] For example, if the user's shoulder joint is positioned lower than a reference posture, the method according to one embodiment may output an illustration or animation highlighting the direction of the shoulder to a display device along with a visual guidance phrase saying "Raise your shoulder." However, it is not limited thereto.
[0227] A method according to one embodiment can enhance user-customized feedback by recording the frequency of error occurrence or automatically adjusting exercise intensity when an operation error is continuously repeated.
[0228] Posture correction information is updated in real time and can be provided in a clear and intuitive form so that users can immediately adjust the position of each joint.
[0229] FIG. 9 is a flowchart illustrating a method according to one embodiment.
[0230] Steps 830 and 840 of Fig. 9 correspond to steps 830 and 840 of Fig. 8, respectively.
[0231] In step 910, the method according to one embodiment can generate posture correction information for correcting motion errors by comparing a posture corresponding to motion information with at least one joint where a motion error has occurred.
[0232] A method according to one embodiment can generate posture correction information for correcting motion errors by setting a reference posture from exercise information included in an exercise item selected from an exercise list, and then comparing the coordinate value of a joint where a motion error occurred with the corresponding joint coordinate value in the reference posture.
[0233] Posture correction information can be generated by analyzing the direction and magnitude of the error of the current joint position based on the coordinate values of the reference posture, and may include at least one of the joint movement direction, rotation correction angle, target position, or correction speed.
[0234] For example, if the coordinate value of the right knee joint deviates more than 10 cm forward from the position of the reference posture, the method according to one embodiment can generate correction direction information for moving the right knee backward.
[0235] For example, if the height coordinate value of the left shoulder joint is lower than the reference posture and the rotation angle is measured asymmetrically, the method according to one embodiment can generate complex posture correction information that induces lifting the shoulder upward and simultaneously externally rotating the upper arm. However, it is not limited thereto.
[0236] A method according to one embodiment can precisely generate posture correction information by considering the alignment state of each joint, the distance error from a reference posture, and the trend of change over time, and the generated correction information can be provided to the user in a visual or auditory manner in a subsequent feedback output step.
[0237] In step 920, the method according to one embodiment can output posture correction information to a display.
[0238] A method according to one embodiment can output the generated posture correction information to the screen of a display device.
[0239] Posture correction information may refer to instructional information intended to guide the position or direction of an erroneous joint to the correct state when the user's posture is performed differently from the reference posture.
[0240] Posture correction information may include information on the correction direction of at least one joint where a motion error occurred.
[0241] Posture correction information may include information regarding the direction of movement or rotation to correct the position of each joint coordinate value for one or more joints where a motion error has occurred. For example, if the left shoulder joint is excessively raised above the reference posture, the posture correction information may include information instructing the left shoulder joint to be lowered vertically downward. Additionally, if the right knee joint is excessively moved forward above the reference position, the posture correction information may include information inducing the knee joint to be moved backward. However, it is not limited thereto.
[0242] Posture correction information can be configured to include not only simple planar movement but also rotational and axis alignment movements by considering not only the positional error of each joint but also the rotational angle error or alignment error.
[0243] For example, if the neck joint is tilted to the left relative to the left-right alignment standard, the posture correction information may include a correction direction that rotates the neck joint to the right. However, it is not limited thereto.
[0244] A method according to one embodiment can set posture correction information in a single direction or a multi-axis direction depending on the type and intensity of motion errors for each joint, and can output it to the user in a visual or auditory manner.
[0245] Posture correction information may include guidance information regarding the direction of movement, rotation correction angle, target posture, or adjustment speed of the joint where the motion error occurred.
[0246] A method according to one embodiment can visualize posture correction information as text, arrow icons, color highlights, illustrations, or videos, and then output it by arranging it to correspond to the location of the user's body parts on the screen.
[0247] For example, if posture correction information for a knee joint is generated as "pull your knee back," the method according to one embodiment may output the corresponding guidance text along with an arrow indicating a backward direction around the knee.
[0248] For example, if posture correction information such as "turn the shoulder outward" is generated regarding a rotation error of the shoulder joint, the method according to one embodiment may play an animation visually indicating the direction of rotation of the arm on the screen of a display device. However, it is not limited thereto.
[0249] A method according to one embodiment may provide synchronized voice output in addition to visual output to enhance the delivery of posture correction information, and may also adjust the output speed, language, volume, or emphasis method according to the difficulty level of the exercise item or user settings.
[0250] FIG. 10 is a flowchart illustrating a method according to one embodiment.
[0251] Steps 830, 910, and 920 of Fig. 10 correspond to steps 830, 910, and 920 of Fig. 9, respectively.
[0252] In step 1010, the method according to one embodiment can obtain user motion range information based on skeleton data.
[0253] A method according to one embodiment can obtain information on the user's range of motion by analyzing the positional change of each joint of the user in a plurality of frames included in skeleton data.
[0254] Range of motion information refers to the spatial limits within which each joint can move, and may include the maximum travel distance, maximum rotation angle, or range of change in distance between joints for each joint.
[0255] A method according to one embodiment can calculate the upper and lower limits of the range of motion by aligning the coordinate values of the same joint in a time series in a plurality of frames and then comparing the maximum and minimum values in each coordinate axis direction.
[0256] For example, if the range of motion of the right shoulder joint in the forward and backward directions is analyzed and it is observed that it moves 20 cm in the x-axis direction and 15 cm in the y-axis direction, the method according to one embodiment can determine that the range of motion of the joint is 20 cm in the x-axis direction and 15 cm in the y-axis direction.
[0257] For example, if an analysis of the rotational movement of the left hip joint reveals that movement within a range of ±30 degrees relative to the z-axis is repeatedly observed, the method according to one embodiment may set the said rotational range as the range of motion. However, it is not limited thereto.
[0258] Range of motion information can be utilized as an indicator reflecting deviations in a user's flexibility, joint function, or motor ability, and can be used as a quantitative standard in generating posture correction information.
[0259] In step 1020, the method according to one embodiment can generate posture correction information based on range of motion information.
[0260] A method according to one embodiment can generate posture correction information for correcting joints that have not reached a reference posture by comparing the user's range of motion information and the reference posture included in the exercise information.
[0261] A method according to one embodiment analyzes the difference between the current coordinate value and the reference coordinate value of each joint through posture correction information, and can limit the range of possible correction so that excessive correction is not performed based on range of motion information.
[0262] For example, if the reference rotation angle of the right hip joint is 45 degrees and the maximum possible rotation angle in the user's range of motion information is 30 degrees, the method according to one embodiment can generate posture correction information that limits the correction angle to 30 degrees or less.
[0263] For example, if the reference position of the left shoulder joint is at a height of 20 cm relative to the y-axis but the user's shoulder range of motion information is limited to 15 cm, the method according to one embodiment can generate posture correction information by modifying the target coordinates to a height of 15 cm. However, it is not limited thereto.
[0264] A method according to one embodiment can calculate a correction value within a safe correction range that minimizes errors without inducing excessive movement by comparing the difference between range of motion information and a reference posture.
[0265] The generated posture correction information can subsequently be converted into visual or auditory feedback and output to the user, and can be utilized as a personalized exercise performance guide.
[0266] FIG. 11 is a block diagram illustrating the components of an electronic device according to one embodiment.
[0267] An electronic device (1100) according to one embodiment may include a sensor (1110), a processor (1120), and a memory (1130).
[0268] The sensor (1110) is an image-based sensing device that enables the estimation of human body joints in a two-dimensional or three-dimensional form, and may be a device including, for example, an RGB camera, an infrared camera, a depth camera, or a combination thereof.
[0269] The sensor (1110) can capture the user's movements from various angles, such as the front, side, or rear, and can be used to generate time-series position data at the joint level through the captured video data.
[0270] The image acquired by the sensor (1110) can be used as input to a skeleton estimation algorithm or a joint estimation network, and can be configured to identify the point in time when the user's posture changes from a static state to a dynamic state or vice versa.
[0271] The sensor (1110) may be a fixed sensor that can be installed in an indoor environment or a home training space, may be implemented as a module attached to a user-wearable device, and may be extended to a multi-sensor environment to improve the accuracy of posture analysis per exercise.
[0272] The processor (1120) is configured to control a series of processes to operate the electronic device (1100) according to the embodiments described above, and may be composed of one or more processors.
[0273] The processor (1120) may be composed of hardware components that perform arithmetic, logic, and input / output operations and signal processing. One or more processors included in the processor (1120) may be circuitry such as a System on Chip (SoC) or an Integrated Circuit (IC). The processor (1120) may be composed of at least one of, for example, a Central Processing Unit, a microprocessor, a Graphic Processing Unit, Application Specific Integrated Circuits (ASICs), Digital Signal Processors (DSPs), Digital Signal Processing Devices (DSPDs), Programmable Logic Devices (PLDs), and Field Programmable Gate Arrays (FPGAs), but is not limited thereto.
[0274] The processor (1120) can write data to memory (1130) or read data stored in memory (1130), and in particular, can process data according to predefined operation rules by executing a program or at least one instruction stored in memory (1130).
[0275] The memory (1130) may store various data, programs, or applications for driving and controlling an electronic device (1100) according to one embodiment. The memory (1130) may include non-volatile memory such as a flash memory type, a hard disk type, a multimedia card micro type, a card type memory (e.g., SD or XD memory), ROM (Read-Only Memory), and at least one of an EEPROM (Electrically Erasable Programmable Read-Only Memory) and a PROM (Programmable Read-Only Memory), and volatile memory such as RAM (Random Access Memory) or SRAM (Static Random Access Memory).
[0276] The memory (1130) may store instructions, data structures, and program code that can be read by the processor (1120). In the following embodiments, the processor (1120) may be implemented by executing the instructions or code of the program stored in the memory (1130).
[0277] The method according to the above-described embodiment can be performed by an electronic device comprising a sensor, a memory, and a processor.
[0278] That is, the processor (1120) can control the operation of the electronic device (1100) according to one embodiment so that the electronic device (1100) according to one embodiment performs the operations described above by executing at least one instruction and / or instruction stored in the memory (1130).
[0279] A method according to one embodiment of the present disclosure may be implemented in the form of program instructions that can be executed through various computer means and recorded on a computer-readable medium. The computer-readable medium may include program instructions, data files, data structures, etc., either alone or in combination. The program instructions recorded on the medium may be those specifically designed and configured for the present invention, or may be those known and available to those skilled in the art of computer software. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tapes; optical recording media such as CD-ROMs and DVDs; magneto-optical media such as floptical disks; and hardware devices specifically configured to store and execute program instructions, such as ROM, RAM, and flash memory. Examples of program instructions include machine code, such as that generated by a compiler, as well as high-level language code that can be executed by a computer using an interpreter, etc.
[0280] Additionally, the method according to the disclosed embodiments may be provided by being included in a computer program product. The computer program product may be traded between a seller and a buyer as a product.
[0281] A computer program product may include a software program and a computer-readable storage medium on which the software program is stored. For example, a computer program product may include a product in the form of a software program (e.g., a downloadable app) that is electronically distributed through a manufacturer of an electronic device or an electronic market (e.g., Google Play Store, App Store). For electronic distribution, at least a portion of the software program may be stored on a storage medium or temporarily created. In this case, the storage medium may be a server of the manufacturer, a server of the electronic market, or a storage medium of a relay server that temporarily stores the software program.
[0282] A computer program product may include a storage medium of a server or a storage medium of a client device in a system composed of a server and a client device. Alternatively, if a third device (e.g., a smartphone) is communicationally connected to the server or client device, the computer program product may include a storage medium of the third device. Alternatively, the computer program product may include the S / W program itself, which is transmitted from the server to the client device or the third device, or from the third device to the client device.
[0283] In this case, one of the server, the client device, and the third device may execute the computer program product to perform the method according to the disclosed embodiments. Alternatively, two or more of the server, the client device, and the third device may execute the computer program product to perform the method according to the disclosed embodiments in a distributed manner.
[0284] For example, a server (e.g., a cloud server or an artificial intelligence server, etc.) can execute a computer program product stored on the server to control a client device connected to the server in communication to perform a method according to the disclosed embodiments.
[0285] A method according to one embodiment may include the step of acquiring the skeleton data, which includes the coordinate values of each of the user's plurality of joints, on a frame-by-frame basis through at least one sensor including a camera.
[0286] A method according to one embodiment may include the step of calculating the average coordinate value of each of the coordinate values of a plurality of joints of the user based on the skeleton data obtained from N frames prior to the current frame.
[0287] A method according to one embodiment may include a step of determining whether the difference between the coordinate value of each of the user's multiple joints obtained in the current frame and the average coordinate value exceeds a threshold value.
[0288] A method according to one embodiment may include the step of determining at least one joint coordinate value that exceeds a threshold value among a plurality of joints of the user obtained in the current frame as an outlier.
[0289] A method according to one embodiment may include the step of correcting the skeleton data by replacing at least one joint coordinate value of the user determined to be an outlier with a coordinate value of a joint corresponding to at least one joint determined to be an outlier, which is obtained in the frame immediately preceding the current frame.
[0290] A method according to one embodiment may include the step of measuring the user's posture based on the corrected skeleton data.
[0291] N can be an integer greater than or equal to 2.
[0292] A method according to one embodiment may include the step of identifying a transformation frame in which the user's posture is transformed through the camera.
[0293] A method according to one embodiment may include the step of determining the number of frames for calculating the average coordinate value of each of the user's plurality of joints based on the transformation frame.
[0294] A method according to one embodiment may include the step of calculating the average coordinate value of each of the coordinate values of a plurality of joints of the user based on the skeleton data obtained from the determined number of frames.
[0295] A method according to one embodiment may include the step of calculating a alignment deviation by comparing the image of the user obtained in the current frame through the camera with the coordinate value of a joint corresponding to at least one joint of the user determined to be an outlier.
[0296] A method according to one embodiment may include the step of obtaining a corrected coordinate value by correcting the coordinate value of a joint corresponding to at least one joint determined to be an outlier, which is obtained in the frame immediately preceding the current frame based on the alignment deviation.
[0297] A method according to one embodiment may include the step of correcting the skeleton data by replacing at least one joint coordinate value of the user determined to be an outlier with the corrected coordinate value.
[0298] A method according to one embodiment may include the step of calculating the height difference and depth difference of the user's left and right shoulder joints from the user's shoulder joint coordinate values included in the skeleton data.
[0299] A method according to one embodiment may include a step of determining whether the user's shoulders are asymmetrical based on the height difference and depth difference of the user's left and right shoulder joints.
[0300] A method according to one embodiment may include the step of calculating the height difference and depth difference of the user's left and right hip joints from the user's hip joint coordinate values included in the skeleton data.
[0301] A method according to one embodiment may include a step of determining whether the user's pelvis is asymmetric based on the height difference and depth difference of the user's left and right hip joints.
[0302] A method according to one embodiment may include the step of determining the alignment status of the user's head, neck, and shoulders through the skeleton data.
[0303] A method according to one embodiment may include a step of determining the tilt of the user's neck based on the alignment state of the user's head, neck, and shoulders.
[0304] A method according to one embodiment may include the step of outputting an exercise list composed of a plurality of exercise items to a display.
[0305] A method according to one embodiment may include the step of obtaining input from a user who selects one of a plurality of exercise items included in the exercise list.
[0306] A method according to one embodiment may include the step of outputting exercise information corresponding to an exercise item selected according to the user's input to the display.
[0307] An exercise list according to one embodiment may include at least one of strength training, aerobic exercise, or stretching.
[0308] A method according to one embodiment may include the step of determining a reference joint to determine whether the user is performing the exercise correctly according to the exercise information.
[0309] A method according to one embodiment may include the step of extracting the user's joint coordinate values corresponding to the reference joint from the skeleton data.
[0310] A method according to one embodiment may include the step of identifying at least one joint of the user where an operation error has occurred, based on the joint coordinate value of the user corresponding to the reference joint.
[0311] A method according to one embodiment may include the step of providing feedback on the operation error.
[0312] An operation error according to one embodiment may include at least one of a position error of the plurality of joints of the user, a rotation angle error of the plurality of joints, a distance error between the plurality of joints, an alignment error of the plurality of joints, or a velocity error of the time-series positional change of the plurality of joints.
[0313] A method according to one embodiment may include the step of generating posture correction information for correcting a motion error by comparing a posture corresponding to the motion information with at least one joint where the motion error occurred.
[0314] A method according to one embodiment may include the step of outputting the posture correction information to the display.
[0315] A method according to one embodiment may include posture correction information regarding the correction direction of at least one joint where the operation error occurred.
[0316] A method according to one embodiment may include the step of obtaining user operating range information based on the skeleton data.
[0317] A method according to one embodiment may include the step of generating posture correction information based on the range of motion information.
[0318] Although the embodiments have been described in detail above, the scope of the present invention is not limited thereto, and various modifications and improvements by those skilled in the art using the basic concept of the present invention as defined in the following claims also fall within the scope of the present invention.
Claims
Claim 1 An electronic device for measuring a user's posture based on skeleton data, comprising: a memory for storing a program or at least one instruction; and at least one processor for executing at least one instruction stored in the memory.The electronic device comprises, by the at least one processor executing at least one instruction stored in the memory, acquiring the skeleton data, which includes the coordinate values of each of the user's multiple joints, on a frame-by-frame basis through at least one sensor including a camera; calculating the average coordinate value of each of the user's multiple joints based on the skeleton data acquired in N frames prior to the current frame; determining whether the difference between the coordinate value of each of the user's multiple joints acquired in the current frame and the average coordinate value exceeds a threshold value; determining at least one joint coordinate value among the user's multiple joints acquired in the current frame that exceeds the threshold value as an outlier; correcting the skeleton data by replacing the at least one joint coordinate value of the user determined as an outlier with the coordinate value of a joint corresponding to the at least one joint determined as an outlier, which is acquired in the frame immediately preceding the current frame; and measuring the user's posture based on the corrected skeleton data, wherein N is an integer greater than or equal to 2, and the threshold value is mutually determined for at least some of the multiple joints according to the anatomical characteristics and range of motion of each joint. An electronic device configured differently, wherein when the electronic device calculates the average coordinate value of each of the coordinate values of each of the user's multiple joints, it identifies a transformation frame in which the user's posture is transformed through the camera, determines the number of frames for calculating the average coordinate value of each of the user's multiple joints based on the transformation frame, and calculates the average coordinate value of each of the user's multiple joints based on the skeleton data obtained from the determined number of frames. Claim 2 An electronic device according to claim 1, wherein when correcting the skeleton data, the electronic device calculates a alignment deviation by comparing the image of the user obtained in the current frame through the camera with the coordinate value of a joint corresponding to at least one joint of the user determined to be an outlier, obtains a correction coordinate value by correcting the coordinate value of a joint corresponding to at least one joint of the user determined to be an outlier obtained in the frame immediately preceding the current frame based on the alignment deviation, and corrects the skeleton data by replacing the coordinate value of at least one joint of the user determined to be an outlier with the correction coordinate value. Claim 3 In paragraph 2, the electronic device, when measuring the posture of the user, calculates the height difference and depth difference of the user's left and right shoulder joints from the shoulder joint coordinate values of the user included in the skeleton data, and determines whether the user's shoulders are asymmetrical based on the height difference and depth difference of the user's left and right shoulder joints. Claim 4 In paragraph 3, the electronic device, when measuring the posture of the user, calculates the height difference and depth difference of the user's left and right hip joints from the hip joint coordinate values of the user included in the skeleton data, and determines whether the user's pelvis is asymmetrical based on the height difference and depth difference of the user's left and right hip joints. Claim 5 In paragraph 4, the electronic device determines the alignment state of the user's head, neck, and shoulders through the skeleton data when measuring the user's posture, and determines the tilt of the user's neck based on the alignment state of the user's head, neck, and shoulders. Claim 6 In paragraph 5, the electronic device outputs an exercise list composed of a plurality of exercise items to a display, obtains input from a user selecting one of the plurality of exercise items included in the exercise list, outputs exercise information corresponding to the selected exercise item according to the user's input to the display, and the exercise list includes at least one of strength training, aerobic exercise, or stretching. Claim 7 In claim 6, the electronic device determines a reference joint for determining whether the user correctly performs the exercise according to the exercise information, extracts the user’s joint coordinate values corresponding to the reference joint from the skeleton data, identifies at least one joint of the user where an operation error has occurred based on the user’s joint coordinate values corresponding to the reference joint, provides feedback on the operation error, wherein the operation error includes at least one of a position error of the user’s plurality of joints, a rotation angle error of the plurality of joints, a distance error between the plurality of joints, an alignment error of the plurality of joints, or a velocity error of the time-series position change of the plurality of joints. Claim 8 In claim 7, the electronic device, when providing the feedback, generates posture correction information for correcting the motion error by comparing a posture corresponding to the motion information with at least one joint where the motion error occurred, outputs the posture correction information to the display, and the posture correction information includes information on the correction direction of the at least one joint where the motion error occurred. Claim 9 In claim 8, the electronic device, when generating the posture correction information, acquires the user's range of motion information based on the skeleton data and generates the posture correction information based on the range of motion information. Claim 10 delete
Citation Information
Patent Citations
Method for measuring and evaluating body performance of user
KR101959079B1
Apparatus and method for estimating the pose of the human body
KR1020220098895A
Non-face-to-face fitness training operation method and system
KR1020230040526A
Device and method for evaluating exercise ability, and record media for performing them
KR1020230133731A
