Method based on limb state monitoring

Abstract

The invention relates to the technical field of limb state monitoring, and particularly discloses a method based on limb state monitoring, which comprises the following steps: acquiring image data and depth data of a monitored object; performing limb state key point identification based on the image data and the depth data to obtain a key point sequence; performing normalization processing on the key point sequence to obtain a standard key point sequence; obtaining a key point time sequence; matching the standard key point sequence with the key point time sequence by using a dynamic time warping algorithm to obtain a matching result; based on a matching result, carrying out limb state key point deviation calculation to obtain limb state deviation; and performing deviation analysis based on the limb state deviation to obtain a limb state monitoring result. By means of the method, the efficiency and safety of exercise and rehabilitation training can be remarkably improved, accurate posture recognition and real-time guidance are achieved through the A I technology, dependence on professionals is reduced, and meanwhile individuation and interactivity are improved.

Description

Technical Field

[0001] The present invention relates to the technical field of limb state monitoring, and particularly relates to a method based on limb state monitoring. Background Art

[0002] In the fields of rehabilitation and fitness, in order to restore, maintain, and enhance physical fitness, it is often necessary to perform targeted postures and corresponding rehabilitation training according to the guiding steps. Currently, most rehabilitation trainings require the presence of professionals for in-person guidance, which is difficult to meet the needs of dispersed exercise participants, impromptu exercise, and self-help exercise.

[0003] With the continuous development of AI and machine vision technologies, using a camera, combined with the human pose key point algorithm, the head, body, and limb joints of the human body are captured in real time, and it is analyzed whether each joint point moves to the position required by the rehabilitation training, so as to judge whether the movement is in place and whether the recovery is normal. By converting the professional training guiding steps into an AI program, the needs of impromptu exercise at different regions and different times can be realized, which has a certain promoting significance for national health and medical rehabilitation.

[0004] Therefore, it is necessary to provide a method based on limb state monitoring. Summary of the Invention

[0005] The purpose of the present invention is to provide a method based on limb state monitoring, and solve the following technical problems: using the human key point recognition technology, combined with the requirements of the rehabilitation exercise posture (standard posture), writing a limb state monitoring algorithm, and combining with the voice function according to the monitoring situation to provide real-time interactive prompts, so as to facilitate the rehabilitation exercise personnel to adjust the exercise posture in time and move closer to the exercise standard.

[0006] The purpose of the present invention can be achieved through the following technical solutions:

[0007] A method based on limb state monitoring, the method includes the following steps:

[0008] Obtain the image data and depth data of the monitoring object;

[0009] Based on the image data and the depth data, perform limb state key point recognition to obtain a key point sequence;

[0010] Perform normalization processing on the key point sequence to obtain a standard key point sequence;

[0011] Obtain a key point time series; wherein, the key point time series is obtained by storing the key point sequence of the standard limb state as a time series;

[0012] Use the dynamic time warping algorithm to match the standard key point sequence and the key point time series to obtain a matching result;

[0013] Based on the matching result, calculate the deviation of the key points of the limb state to obtain the limb state deviation;

[0014] Based on the limb state deviation, perform deviation analysis to obtain the limb state monitoring result.

[0015] As a further solution of the present invention, the identifying the key points of the limb state based on the image data and the depth data to obtain a sequence of key points includes:

[0016] Perform image recognition based on the image data to obtain a first sequence of key points of the limb state;

[0017] Convert the depth value of each pixel in the depth data into a coordinate value;

[0018] Based on the coordinate values, determine a second sequence of key points of the limb state;

[0019] Fuse the first sequence of key points of the limb state and the second sequence of key points of the limb state based on the Kalman filtering algorithm to obtain the sequence of key points.

[0020] As a further solution of the present invention, the method further includes:

[0021] Obtain inertial sensor data of the monitoring object; the inertial sensor data includes joint angle and acceleration information;

[0022] Estimate the posture and motion trajectory based on the joint angle and the acceleration information;

[0023] Calculate the joint angle change information based on the result of the posture and motion trajectory estimation;

[0024] Determine a third sequence of key points of the limb state based on the joint angle change information;

[0025] Fuse the first sequence of key points of the limb state, the second sequence of key points of the limb state and the third sequence of key points of the limb state based on the Kalman filtering algorithm to obtain the sequence of key points.

[0026] As a further solution of the present invention, the calculating the deviation of the key points of the limb state based on the matching result to obtain the limb state deviation includes:

[0027] Calculate the deviation of each key point of the limb state based on the matching result;

[0028] Group the key points of the limb state based on the human joint division; wherein, each group corresponds to a joint part;

[0029] Determine the limb state deviation of the local joint based on the deviation of the limb state key points within each group;

[0030] Determine the overall limb state deviation through the spatial relationship between the human joints.

[0031] As a further solution of the present invention, the deviation analysis based on the limb state deviation to obtain the limb state monitoring result includes:

[0032] Combine the limb state deviation of the local joint and the overall limb state deviation to obtain a multi-scale comprehensive deviation;

[0033] Determine the limb state monitoring result based on the multi-scale comprehensive deviation.

[0034] As a further solution of the present invention, the limb state deviation of the local joint reflects the difference in joint angles and is represented by the following formula:

[0035]

[0036] where E1 i represents the limb state deviation of the i-th local joint, is the calculated angle of the i-th local joint, is the angle of the i-th local joint in the standard limb state;

[0037] The overall limb state deviation is represented by the following formula:

[0038]

[0039] where E2j is the overall limb state deviation, represents the distance between the j-th limb state key point and the k-th limb state key point, represents the distance between the j-th limb state key point and the k-th limb state key point in the standard limb state;

[0040] The multi-scale comprehensive deviation is represented by the following formula:

[0041] E = α·E 1i +(1 - α)·E 2j

[0042] where E represents the multi-scale comprehensive deviation, and α is a weight coefficient between 0 and 1, used to balance the relative influence of the limb state deviation of the local joint and the overall limb state deviation.

[0043] As a further solution of the present invention, the method further includes:

[0044] Based on the limb state monitoring result, issue a posture adjustment prompt to the monitored object; the adjustment prompt includes a voice prompt and an interface prompt.

[0045] A system based on limb state monitoring, the system includes:

[0046] A first acquisition module for acquiring image data and depth data of a monitored object;

[0047] A key point sequence acquisition module for performing limb state key point recognition based on the image data and the depth data to obtain a key point sequence;

[0048] A normalization module for performing normalization processing on the key point sequence to obtain a standard key point sequence;

[0049] A key point time series acquisition module for acquiring a key point time series; wherein, the key point time series is obtained by storing the key point sequence of the standard limb state as a time series;

[0050] A matching module for using the dynamic time warping algorithm to match the standard key point sequence and the key point time series to obtain a matching result;

[0051] A deviation calculation module for performing limb state key point deviation calculation based on the matching result to obtain a limb state deviation;

[0052] A deviation analysis module for performing deviation analysis based on the limb state deviation to obtain a limb state monitoring result.

[0053] Advantages of the present invention: (1) Through real-time limb state monitoring, users can quickly identify and adjust non-standard movement postures, thereby avoiding unnecessary mistakes and improving the exercise effect. The system can automatically evaluate and provide instant feedback to help users optimize the execution of each action. (2) It can reduce the dependence on professional guidance. Using AI technology and a key point recognition system, users can exercise independently without the presence of professional personnel. The system can independently judge the standardness of the posture and provide guidance, which is suitable for scattered, impromptu, and self-service exercise scenarios. Description of the Drawings

[0054] The present invention will be further described below with reference to the accompanying drawings.

[0055] Figure 1 is a flowchart of the method for limb state monitoring according to the present invention. Detailed Embodiments

[0056] The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the protection scope of the present invention.

[0057] The method for monitoring limb states disclosed in this application can be implemented in the following ways. Way 1: Use a desktop computer + display screen + voice player, plus a camera; Way 2: Bring a notebook with a built-in camera and voice function; among them, both Way 1 and Way 2 also need to be configured with a set of AI exercise software. After the AI exercise software is started, each step in the method for monitoring limb states disclosed in this application can be executed. In some embodiments, a depth sensor and an inertial sensor can be additionally equipped to obtain more data and improve the accuracy of key point recognition. Step 1: Ensure that the camera and voice device are properly connected to the computer; Step 2: Install the AI exercise software on the computer; Step 3: Double-click to start the software; Step 4: Complete the exercise according to the exercise steps; Step 5: View the exercise results. During implementation, a person's picture can be obtained in real time through a camera, and the picture is transmitted to the human key point algorithm. The algorithm identifies 33 key points of the human body, such as eyes, ears, nose, mouth, shoulders, elbows, wrists, etc., and then judges whether the corresponding joint points reach the corresponding positions required by the training. When they reach, the next exercise requirement is prompted through voice and the display screen. When they do not reach, voice prompts are given to continue to complete this step of the exercise. When the exercise is completed, the time used for this exercise is displayed and announced. When no exercise is needed, the screen end button can be moved in the air with the hand to achieve touchless start and stop, and there is no need to wear external devices for exercise.

[0058] Please refer to Figure 1 As shown, the present invention is a method for monitoring limb states, Figure 1 which is a schematic flowchart of the method for monitoring limb states of the present invention. In some embodiments, Figure 1 the flowchart 100 shown can be executed by a processing device (for example, a processor). The processing device can be an electronic device with computing functions. As Figure 1 shown, the flowchart 100 can include the following operations.

[0059] Step 101, obtain the image data and depth data of the monitoring object.

[0060] The image data refers to the two-dimensional images of the monitoring object (such as the human body) obtained through imaging devices (such as cameras, depth cameras, etc.). These images are used for human key point recognition to help the system determine the position of the monitoring object in space.

[0061] Depth data is the distance information of each pixel obtained through depth sensors (such as Kinect, LiDAR). Depth data is used to provide the three-dimensional spatial coordinates of the monitored object, which can effectively eliminate the errors caused by perspective changes in two-dimensional images.

[0062] In some embodiments, an imaging device and a depth sensor can be used to simultaneously collect image data and depth data. The obtained image data can be an RGB image, and the depth data is the distance information of each pixel generated by the depth sensor.

[0063] The obtained depth data can be used as an information supplement to the image data, which can improve the accuracy of subsequent human key point recognition.

[0064] Step 102, perform limb state key point recognition based on the image data and the depth data to obtain a key point sequence.

[0065] Limb state key point recognition refers to extracting the key points of the human body (such as shoulders, elbows, knees, hips, etc.) and determining their positions in space with the assistance of image processing techniques and depth data.

[0066] The key point sequence refers to the arrangement of the position coordinates of all the recognized key points. The obtained limb key point sequence may include the two-dimensional or three-dimensional coordinates of parts such as shoulders, elbows, and knees.

[0067] In some embodiments, according to the image data and the depth data, a key point recognition algorithm can be used to extract the three-dimensional coordinates of each key point to form a complete key point sequence.

[0068] In some embodiments, the performing limb state key point recognition based on the image data and the depth data to obtain a key point sequence may include the following operations.

[0069] S10, perform image recognition based on the image data to obtain a first limb state key point sequence.

[0070] Image recognition refers to processing an image through computer vision algorithms to identify the positions of human key points in it. Human pose estimation can be performed through neural network models (such as OpenPose, MediaPipe, etc.). For example, through an AI image recognition algorithm, key points such as shoulders, knees, and ankle joints can be recognized from a photo of a squat and their positions on the two-dimensional plane can be marked.

[0071] The first limb state key point sequence refers to the coordinate sequence of each key point of the human body recognized from the image data. Since the image itself is two-dimensional, these coordinates are usually planar coordinates (x, y). For example, in image recognition, obtaining the coordinates of the shoulder (150, 200), the coordinates of the knee (180, 320), etc., forms the first limb state key point sequence. In subsequent steps, the two-dimensional planar coordinates of the depth data can be combined and converted into three-dimensional space coordinates.

[0072] In some embodiments, by inputting the image data into an image recognition algorithm, the positions of the key points of the human body can be extracted. The result after algorithm processing will output the two-dimensional coordinates of each key point, forming the first limb state key point sequence.

[0073] S11, convert the depth value of each pixel in the depth data into a coordinate value.

[0074] The depth data is the distance information between each pixel point provided by the depth sensor and the camera. The depth value of each pixel represents the physical distance from this point to the camera. For example, the depth sensor can provide a depth map, where the value of each pixel represents the distance from this point to the sensor. For example, the depth value of a certain point is 2.5 meters, indicating that this point is 2.5 meters away from the sensor.

[0075] Combining the depth value of each pixel in the depth map with the position (x, y) of this pixel in the image and performing geometric transformation based on the camera internal parameters (focal length, principal point, etc.), the three-dimensional space coordinates (x, y, z) corresponding to this pixel, that is, the coordinate value, can be obtained. For example, through the internal parameters of the camera and the depth value, the depth value (for example, 2.5 meters) of a pixel point (for example, the coordinate in the image is (100, 150)) is converted into the actual position coordinates in three-dimensional space (for example, (1.2, 1.5, 2.5)).

[0076] For each pixel point in the depth map, by combining the two-dimensional position and the depth value of this point and using the internal parameter formula of the camera for conversion, the three-dimensional coordinates of each pixel point can be obtained.

[0077] S12, based on the coordinate value, determine the second limb state key point sequence.

[0078] The second limb state key point sequence is the coordinate sequence of each key point of the human body in three-dimensional space obtained after the conversion of the depth data. The coordinates of these key points are three-dimensional coordinates (x, y, z), which can reflect the actual spatial posture of the human body. For example, through the conversion of the depth data, the three-dimensional coordinates of the shoulder (1.2, 1.5, 2.5), the three-dimensional coordinates of the knee (1.3, 2.0, 3.0), etc. can be obtained, forming the second limb state key point sequence.

[0079] In some embodiments, the three-dimensional positions of each key point can be identified based on the three-dimensional coordinates converted from depth data. For example, by calculating the depth information of the knees and shoulders in the depth image, their three-dimensional coordinates are determined.

[0080] S13. Based on the Kalman filter algorithm, fuse the first limb state key point sequence and the second limb state key point sequence to obtain the key point sequence.

[0081] The Kalman filter algorithm is a recursive filter algorithm used to estimate the state of a dynamic system, which can fuse signals from multiple sources to improve the accuracy and stability of the results. In this method, the Kalman filter algorithm can combine the information extracted from image recognition (two-dimensional coordinates) and depth data (three-dimensional coordinates), and through filtering and smoothing processing, fuse the two to obtain a more accurate key point sequence.

[0082] The fusion of the key point sequence refers to combining the first limb state key point sequence (two-dimensional coordinates) and the second limb state key point sequence (three-dimensional coordinates) through the Kalman filter to generate a more accurate and stable final key point sequence. For example, if the position of a certain key point in the result of image recognition is unstable, the Kalman filter algorithm can correct its position based on the stable three-dimensional coordinates provided by the depth data, so that the final key point sequence is more accurate.

[0083] In some embodiments, through the Kalman filter algorithm, using the key point coordinate sequences provided by the image data and the depth data, according to the different noise characteristics of the first sequence and the second sequence, their weights can be dynamically adjusted, and finally an optimized key point sequence is output.

[0084] In this embodiment, through the combination of the image data and the depth data, the two-dimensional coordinates of the human body key points are extracted by using image recognition, and these coordinates are converted into three-dimensional space coordinates through the depth data. Then, based on the Kalman filter algorithm, the two are fused to obtain a more accurate and stable limb state key point sequence, which can improve the accuracy and robustness of key point recognition.

[0085] In some embodiments, the method further includes the following operations.

[0086] S20. Obtain the inertial sensor data of the monitoring object; the inertial sensor data includes joint angle and acceleration information.

[0087] Inertial sensor data refers to motion data obtained through an inertial measurement unit (IMU) or other inertial sensors. Inertial sensors include accelerometers, gyroscopes, and magnetometers, which can provide information such as acceleration, angular velocity, and direction of an object. For example, an IMU device worn on the user can obtain acceleration information (such as speed changes per second) and joint angles (the angle is determined by a combination of gyroscopes and accelerometers).

[0088] The joint angle indicates the relative angle of the joint in space, such as the bending angle of the knee joint; the acceleration information indicates the acceleration sensed by the acceleration sensor.

[0089] In some embodiments, inertial sensors worn on different parts of the user's body can be used to collect data and transmit it to the system. The system can obtain joint angle changes and acceleration data in real time to provide input for subsequent posture and motion trajectory estimation.

[0090] S21, performing posture and motion trajectory estimation based on the joint angle and the acceleration information.

[0091] Posture estimation refers to estimating the body posture of the monitored object through sensor data. The spatial posture of the object can be inferred based on the acceleration and angular velocity data of the sensor by combining accelerometers and gyroscopes.

[0092] Motion trajectory estimation refers to the process of calculating the motion path of an object through continuous inertial data (including acceleration and angular velocity information). The velocity can be obtained by integrating the acceleration data, and the position can be obtained by integrating the velocity data.

[0093] In some embodiments, the body posture and motion trajectory can be estimated based on the joint angle and acceleration data obtained by the inertial sensor through a mathematical model (such as Kalman filtering or other sensor fusion methods).

[0094] S22, calculating joint angle change information based on the results of the posture and motion trajectory estimation.

[0095] The joint angle change information indicates the angle change of the joint over a period of time. The change of the angle of each joint over time can be calculated from the result of posture estimation, reflecting the movement of the body parts. For example, when a person's arm extends from a bent 90° angle to 180°, the joint angle change information is 90°.

[0096] The pose estimation results can be analyzed to calculate the change in angle of each joint during the movement. For example, the system can use time series analysis to identify the increase or decrease in joint angle and calculate the corresponding change.

[0097] S23. Determine the third limb state key point sequence based on the joint angle change information.

[0098] The third limb state key point sequence is a key point sequence generated based on joint angle change information.

[0099] In some embodiments, the corresponding joint positions can be deduced according to the calculated joint angle changes, and the coordinates of the key points can be calibrated in a three-dimensional space to obtain the third limb state key point sequence.

[0100] S24. Based on the Kalman filtering algorithm, fuse the first limb state key point sequence, the second limb state key point sequence, and the third limb state key point sequence to obtain the key point sequence.

[0101] The Kalman filtering algorithm is used to fuse information from different data sources to obtain a more accurate and stable estimation result. In this step, it is used to fuse the key point sequences generated from different sensors (image data, depth data, inertial sensor data). For example, if the knee coordinates identified by the image data and the depth data are slightly different from the knee coordinates deduced by the inertial sensor, the Kalman filtering algorithm can combine this information to obtain a more accurate knee position.

[0102] Key point sequence fusion refers to fusing the key point sequences generated from multiple data sources through Kalman filtering to improve the accuracy of the final result.

[0103] In some embodiments, multiple key point sequences obtained through different technologies (image data, depth data, inertial sensor data) can be used as inputs, and the Kalman filtering algorithm is used for data fusion. Kalman filtering will perform weighted processing on each data source and output the key point sequence.

[0104] Step 103. Perform normalization processing on the key point sequence to obtain the standard key point sequence.

[0105] Normalization processing refers to converting the obtained key point sequence into a unified scale so that the coordinates of the key points can be compared under different conditions. For example, assuming that the three-dimensional coordinate positions of the detected shoulders and knees are different, the normalization operation can align all key point positions relative to a reference point (such as the waist or pelvis).

[0106] The standard key point sequence is the key point sequence after normalization processing. The normalized standard key point sequence can facilitate subsequent analysis and improve the accuracy of the analysis. For example, after normalization, the three-dimensional coordinates of the shoulders can be converted into coordinate values relative to the center point, enabling the postures of different users to be uniformly compared.

[0107] Step 104: Obtain the key-point time series. The key-point time series is obtained by storing the key-point sequence of the standard limb state as a time series.

[0108] The key-point time series is a sequence formed by arranging the key-point sequences of the standard limb state in chronological order. The standard key-point sequence at each time point reflects the change of the limb state. For example, if the monitored object is performing an action (such as a squat), the key-point sequence at each time point records the three-dimensional coordinates of each key point during the action.

[0109] In some embodiments, by continuously collecting video frames, the changes of human key points at different time points are tracked to form a key-point time series. The key-point sequence can describe the dynamic changes of key points during the entire movement process.

[0110] Step 105: Use the dynamic time warping algorithm to match the standard key-point sequence and the key-point time series to obtain a matching result.

[0111] The dynamic time warping (DTW) algorithm is an algorithm for calculating the similarity of two sets of time series data, which can handle the differences caused by different speeds in the time series. Through the DTW algorithm, the key-point sequence of the standard posture can be optimally aligned with the key-point time series of the monitored object.

[0112] For example, assume that there is a time deviation between the time series of the standard squat action and the actual squat action. The DTW algorithm will automatically adjust the time axes of the two to make the matching between the key-point sequences more accurate.

[0113] The standard key-point sequence refers to the key-point sequence under the standard limb state. The standard key-point sequence is also a time series associated with time. For example, it can be a set of key points of the standard limb state at multiple time points.

[0114] In some embodiments, the standard key-point sequence can be dynamically time-warped with the actually collected key-point time series, and the optimal match can be obtained by adjusting the time axis, thereby calculating the similarity between the two sequences.

[0115] Step 106: Based on the matching result, calculate the key-point deviation of the limb state to obtain the limb state deviation.

[0116] The key-point deviation refers to the difference between the standard key-point sequence and the actual key-point time series, which can be obtained by calculating the coordinate error of each key point. For example, in the standard squat action, the target position of the knee is (0.5, 0.2, 0.3), and the position of the knee in the actual squat is (0.55, 0.25, 0.32), then the deviation is the distance between the two coordinates.

[0117] In some embodiments, based on the results of matching by the DTW algorithm, the spatial coordinate differences of each key point can be calculated, and then the overall limb state deviation can be obtained.

[0118] In some embodiments, performing limb state key point deviation calculation based on the matching results to obtain the limb state deviation may include the following operations.

[0119] S30. Based on the matching results, calculate the deviation of each limb state key point.

[0120] The matching result refers to the comparison result between the actual key point data and the standard data obtained after analyzing the limb state of the monitoring object through human key point recognition technology.

[0121] The limb state key point deviation refers to the deviation between the monitored limb key points (such as joint and limb end positions) and the standard or expected positions.

[0122] In some embodiments, the actual key point positions can be compared with the standard (or preset) positions to calculate the deviation value of each key point. For example, calculate the actual and standard deviations of the knee angle, wrist angle, elbow position, etc.

[0123] S31. Based on the division of human joints, group the limb state key points; wherein, each group corresponds to a joint part.

[0124] The division of human joints refers to dividing the limbs into different joint regions according to the human body structure. Each joint region consists of several key points, and these joint regions correspond to different movement parts of the human body. For example, the knee joint, elbow joint, shoulder joint, etc.

[0125] The grouping of limb state key points refers to grouping the key points of the limb according to the joint parts. For example, group all the key points related to the shoulder (such as the shoulder, elbow, and wrist) into one group.

[0126] In some embodiments, based on human anatomy and kinematics knowledge, the position of each joint and its related key points can be defined. For example, the knee joint includes the knee, hip, ankle, etc. Classify the limb state key points according to the joint parts. For example, the shoulder joint group can include the key points of the shoulder, elbow, and wrist, and the knee joint group includes the key points of the knee, hip, and ankle.

[0127] S32. Based on the deviation of the limb state key points within each group, determine the limb state deviation of the local joint.

[0128] The limb state deviation of a local joint refers to the deviation of key points within a certain joint group relative to the standard position. For example, if each key point (such as the knee, hip, and ankle) in the knee joint group deviates from the standard position, the deviation value of this group is calculated.

[0129] In some embodiments, the deviation of the entire joint can be obtained based on the deviations of the key points within each group. For example, if the deviation values of the key points within the knee joint group are relatively large, the overall deviation of the knee joint is also relatively large.

[0130] In some embodiments, weighted average, maximum deviation, or other statistical methods can be used to calculate the deviation of each joint group.

[0131] S33. Determine the overall limb state deviation based on the spatial relationship between the human joints.

[0132] The spatial relationship between human joints refers to the relative position and movement relationship between different joints. For example, the relative position of the knee joint and the hip joint, and the relative movement of the wrist and the elbow.

[0133] The overall limb state deviation refers to the comprehensive influence of all joint deviations, manifested as the degree to which the entire limb deviates from the standard movement pattern during movement. For example, if the deviations of each joint are relatively large, the overall limb state deviation will also be relatively large.

[0134] In some embodiments, the overall limb state deviation can be judged through the movement correlation between joints. For example, the deviations of the knee joint and the hip joint need to consider their relative angle and position relationship. Based on the deviations of each joint and their spatial relationship, the overall limb state deviation is obtained. A comprehensive weighted algorithm or a spatial geometry model can be used to calculate the overall deviation. For example, if the deviations of the knee joint and the hip joint are relatively large and the angle difference between them is obvious, the overall limb deviation is relatively large. By calculating the deviations of each joint and combining the spatial relationship between joints (such as angle difference, distance difference, etc.), a quantitative value of the overall deviation can be obtained.

[0135] In this way, the state deviation of each limb part can be accurately calculated, and the exercise posture can be adjusted according to the overall deviation to improve the exercise effect.

[0136] Step 107. Perform deviation analysis based on the limb state deviation to obtain the limb state monitoring result.

[0137] Deviation analysis refers to statistically analyzing the key point deviations of the limb state to evaluate whether the limb state meets the preset standard requirements. If the deviation is too large, the user can be prompted that the posture is not standard. For example, if the deviation value of a certain key point (such as the knee) exceeds the preset tolerance (such as exceeding 5 cm), it can be determined that the posture does not meet the standard.

[0138] Based on the calculated deviation value, analyze the overall performance of the limb state to determine whether adjustment is needed. The system can give feedback according to the judgment result, such as reminding the user to correct the non-standard posture, etc.

[0139] In some embodiments, the deviation analysis based on the limb state deviation to obtain the limb state monitoring result includes: combining the limb state deviation of the local joint and the overall limb state deviation to obtain a multi-scale comprehensive deviation; and determining the limb state monitoring result based on the multi-scale comprehensive deviation.

[0140] The multi-scale comprehensive deviation refers to the integrated deviation evaluation formed by integrating the deviations at multiple scales (such as local and overall scales). It can be understood as synthesizing the local joint deviation and the overall deviation according to a certain weight to form a multi-level deviation result. For example, the local deviation can reflect the detailed problems of the joint, while the overall deviation reflects the overall problems of limb coordination, and the combination of the two gives a more comprehensive deviation assessment.

[0141] In some embodiments, according to the local deviation and the overall deviation of each joint, they can be combined according to a preset rule (such as weight or weighted average). For example, they can be combined by the weighted average method or the quantization synthesis method, etc.

[0142] In some embodiments, the limb state deviation of the local joint reflects the difference in joint angles and is represented by the following formula (1):

[0143]

[0144] where E1 i represents the limb state deviation of the i-th local joint, is the calculated angle of the i-th local joint, is the angle of the i-th local joint in the standard limb state;

[0145] The overall limb state deviation is represented by the following formula (2):

[0146]

[0147] where E2j is the overall limb state deviation, represents the distance between the j-th limb state key point and the k-th limb state key point, represents the distance between the j-th limb state key point and the k-th limb state key point in the standard limb state.

[0148] The multi-scale comprehensive deviation is represented by the following formula (3):

[0149] E = α·E 1i+(1 - α)·E 2j (3)

[0150] Wherein, E represents the multi - scale comprehensive deviation, and α is a weight coefficient between 0 and 1, which is used to balance the relative influence of the limb state deviation of local joints and the overall limb state deviation.

[0151] By combining the limb state deviation of local joints and the overall limb state deviation, the multi - scale comprehensive deviation is obtained. Then, based on this comprehensive deviation analysis, the limb state monitoring result is obtained, so as to accurately feedback whether the user's current posture meets the exercise or rehabilitation standard, and corresponding adjustment suggestions are put forward. This process is of great significance for the application of rehabilitation training and fitness guidance systems, which can help users adjust their exercise postures in real time, improve the training effect and reduce sports injuries.

[0152] In some embodiments, according to the limb state monitoring result, a posture adjustment prompt can be sent to the monitored object. The adjustment prompt includes a voice prompt and an interface prompt.

[0153] The limb state monitoring result refers to obtaining and analyzing the user's movement posture information through various sensors, cameras or human posture recognition systems. It can include data such as joint angles, key point positions, and the overall state of the limbs. For example, if the knee bending angle of the user is greater than the standard angle, the system will take this deviation as the limb state monitoring result.

[0154] The monitored object refers to a person or device that undergoes limb state monitoring. In the present invention, the monitored object can be an individual performing exercise or rehabilitation training.

[0155] The posture adjustment prompt refers to the adjustment suggestion sent by the system according to the limb state monitoring result to help the user correct the exercise posture to meet the standard requirements. The prompt can be in various forms such as voice, text, or graphics. For example, when the user's back posture is not straight, the system may send a voice prompt "Please straighten your back".

[0156] The working principle of the present invention: Through real - time limb state monitoring, it can help users quickly identify and adjust non - standard exercise postures, thus avoiding unnecessary mistakes and improving the exercise effect. The system can automatically evaluate and provide instant feedback to help users optimize the execution of each action.

[0157] The above has described a detailed description of an embodiment of the present invention, but the content described is only the preferred embodiment of the present invention and cannot be considered as used to limit the scope of implementation of the present invention. All equivalent changes and improvements made according to the scope of the application of the present invention should still fall within the scope covered by the patent of the present invention.

Claims

1. A method based on limb status monitoring, characterized in that: The following steps are involved: Acquire image data and depth data of the monitored object; Perform limb state key point recognition based on the image data and the depth data to obtain a key point sequence; Normalizing the key point sequence to obtain a standard key point sequence; Acquire a key point time series; wherein the key point time series is obtained by storing a key point sequence of a standard limb state as a time series; Using a dynamic time warping algorithm, matching the standard key point sequence with the key point time sequence to obtain a matching result; Based on the matching result, calculating the deviation of key points of limb state to obtain the limb state deviation; Deviation analysis is performed based on the limb state deviation to obtain a limb state monitoring result.

2. The method based on limb status monitoring according to claim 1, characterized in that: The step of performing limb state key point recognition based on the image data and the depth data to obtain a key point sequence includes: Perform image recognition based on the image data to obtain a first limb state key point sequence; Convert the depth value of each pixel in the depth data into a coordinate value; Based on the coordinate values, determining a second limb state key point sequence; The first limb state key point sequence and the second limb state key point sequence are fused based on a Kalman filter algorithm to obtain the key point sequence.

3. The method based on limb status monitoring according to claim 2, characterized in that: The method further comprises: Acquiring inertial sensor data of the monitored object; the inertial sensor data includes joint angle and acceleration information; Performing posture and motion trajectory estimation based on the joint angle and the acceleration information; Based on the results of the posture and motion trajectory estimation, calculate the joint angle change information; Based on the joint angle change information, determining a third limb state key point sequence; The first limb state key point sequence, the second limb state key point sequence and the third limb state key point sequence are fused based on a Kalman filter algorithm to obtain the key point sequence.

4. The method based on limb status monitoring according to claim 1, characterized in that: The step of calculating the limb state key point deviation based on the matching result to obtain the limb state deviation comprises: Based on the matching results, the deviation of each limb state key point is calculated; Based on the division of human joints, the key points of limb states are grouped; each group corresponds to a joint part; Determine the limb state deviation of the local joint based on the deviation of the limb state key points in each group; The overall limb state deviation is determined through the spatial relationship between the human body joints.

5. The method based on limb status monitoring according to claim 4, characterized in that: The performing deviation analysis based on the limb state deviation to obtain the limb state monitoring result includes: Combining the limb state deviation of the local joint and the overall limb state deviation to obtain a multi-scale comprehensive deviation; Based on the multi-scale comprehensive deviation, a limb state monitoring result is determined.

6. The method based on limb status monitoring according to claim 5, characterized in that: in, The limb state deviation of the local joint reflects the difference in joint angles and is expressed by the following formula: Among them, E 1i represents the limb state deviation of the i-th local joint, is the calculated angle of the i-th local joint, is the angle of the i-th local joint in the standard limb state; The overall limb state deviation is expressed by the following formula: Among them, E 2j Overall limb status deviation, represents the distance between the jth limb state key point and the kth limb state key point, Represents the distance between the jth limb state key point and the kth limb state key point in the standard limb state; The multi-scale integrated deviation is expressed by the following formula: E=α·E 1i +(1-a)·E 2j ; Among them, E represents the multi-scale comprehensive deviation, and α is a weight coefficient between 0 and 1, which is used to balance the relative influence of the limb state deviation of the local joint and the overall limb state deviation.

7. The method based on limb status monitoring according to claim 1, characterized in that: The method further comprises: According to the limb state monitoring result, a posture adjustment prompt is issued to the monitored object; the adjustment prompt includes a voice prompt and an interface prompt.

Images