A recognition method and system for motion and posture detection

Abstract

The present application discloses a recognition method for detecting actions and postures, comprising: real-time collection of human image information or receiving human image information from a mobile terminal; detecting facial key points and body key points of a target object based on the human image information; when the human image information is a picture, automatically generating an evaluation baseline based on the picture, and performing static detection of the target object based on the evaluation baseline; when the human image information is a video, dynamically detecting the target key points of the target object based on the changing trend of the position of the target key points and the follow-up changes of the associated key points associated with the target key points during the target object's completion of the specified action of the item to be tested in the video. The present application can not only realize on-site detection, but also complete action and posture detection through videos and pictures sent by the tester; and can eliminate the muscle compensation that is easy to occur when a person performs an action, making the detection result more accurate.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a recognition method and system for performing action and posture detection. Background Art

[0002] Poor posture negatively impacts both physical appearance and physical health. However, with the shift to modern lifestyles and the widespread use of devices like computers and mobile phones, the number of people suffering from poor posture, such as hunching when standing or sitting, is increasing. Furthermore, it's difficult for people to assess or correct their own posture, making accurate and convenient posture detection a significant challenge.

[0003] Most existing testing systems rely on on-site testing and analysis, requiring participants to visit specialized facilities for testing, which is quite inconvenient. Furthermore, under normal circumstances, participants should perform a specific movement while maintaining all joints except those involved in the test. However, in reality, people experience muscle compensation to varying degrees during movement, and this is more pronounced in those with impaired motor skills. Ignoring muscle compensation and directly assessing the participant's posture based on test results can lead to significant discrepancies between the results and their actual physical condition, negating the significance of the test. Summary of the Invention

[0004] The purpose of this application is to solve the inconvenience caused to the tester by the need for on-site detection in the prior art, and to provide a detection system and method that can complete action and posture detection through videos and pictures.

[0005] The present invention is achieved through the following technical measures: a recognition system for action and posture detection, including: real-time collection of human body image information or receiving human body image information from a mobile terminal; performing facial key point detection and body key point detection on a target object based on the human body image information; when the human body image information is a picture, automatically generating an evaluation baseline based on the picture, and performing static detection on the target object based on the evaluation baseline; when the human body image information is a video, dynamically detecting the target key points of the target object based on the changing trend of the target key point position and the follow-up changes of the associated key points associated with the target key points during the target object completing the specified action of the item to be tested in the video; the target key points are some key points among the facial key points and the body key points.

[0006] Preferably, the human body image information is input into a human body key point detection model to obtain facial key points and body key points.

[0007] Preferably, the dynamic detection of the target object includes: defining associated key points related to the target key points according to the target key points, and positioning each key point by three-dimensional coordinates; sequentially connecting the target key points and the associated key points in each frame, drawing a plurality of lines connecting the target key points and the associated key points, wherein the lines at least include line segments of the target key points along two different directions, extracting the angle formed by the lines at the target key points, and obtaining an angle set ρ = {ρ1, ρ2, ..., ρ i}, where i is the number of video frames; determine the maximum value of the angle ρ within a given time period max and the minimum value ρ min , ρ max =max{ρ1,ρ2,...,ρ i},ρ min =min{ρ1,ρ2,...,ρ i}, that is, the joint activity detection result of the target key point is obtained, and the range of activity is ρ min ~ρ max .

[0008] Preferably, the target key points of the dynamic detection include joint mobility detection at the shoulders, elbows, wrists, neck, chest, sacrum, hips, knees and ankles.

[0009] Preferably, the static detection of the target object includes: extracting target key points in the image, connecting the target key points in a preset connection order, obtaining key point connection lines, and comparing the key point connection lines with the automatically generated evaluation baseline to obtain static detection results.

[0010] Preferably, the static test includes a leg length test, a limb tilt test, a head, neck and spine curvature test, and a head, neck and spine scoliosis test.

[0011] Preferably, during dynamic detection, a data list is generated for the target key points in each frame of the video screen, wherein the data list includes the three-dimensional coordinate value, timestamp and GRAY value of the target key points, and n is the total number of facial key point detections and body key points, that is, one frame of the picture can form a dictionary of n data list sets; finally, a dictionary of m groups of data list sets is formed, where m is the total number of frames of the video, and the detection result is corrected according to the change amplitude of the associated key points by identifying the changes in the associated key point data lists in the dictionary.

[0012] Preferably, the facial key points include nose, inner corner of left eye, pupil of left eye, outer corner of left eye, inner corner of right eye, pupil of right eye, outer corner of right eye, left ear, right ear, left corner of mouth and right corner of mouth; the body key points include left shoulder, right shoulder, left elbow, right elbow, left wrist, right wrist, left hip, right hip, left knee, right knee, left ankle, right ankle, left heel, right heel, left toe and right toe.

[0013] The present application also discloses an identification system for performing action and posture detection, which is used to implement the identification method, including: a human body image information acquisition unit; a judgment unit, used to judge whether the human body image information is a picture or a video; a static detection unit, used to automatically generate an evaluation baseline based on the picture when the human body image information is a picture, and perform static detection on the target object based on the evaluation baseline; a dynamic detection unit, used to perform dynamic detection on the target key points of the target object based on the changing trend of the target key point position and the follow-up changes of the associated key points associated with the target key points during the target object completing the specified action of the item to be tested in the video when the human body image information is a video.

[0014] The beneficial effects of the present application are as follows: (1) the present application can not only realize on-site detection, but also complete action and posture detection through videos and pictures sent by the tester, so that the tester can complete the detection more conveniently without having to go to a special institution to queue up for detection, which saves time and effort and is more convenient; (2) the present application can not only complete static detection but also complete dynamic detection according to the joint range of motion of the target key point; (3) the present application identifies the associated key points of the target key point by analyzing the coordinate changes of other key points with the target key point, monitors the change amplitude of the associated key point coordinates caused by the target object completing the specified action of the item to be tested in the associated data list, and corrects the detection result according to the said change amplitude, so as to eliminate the muscle compensation that is easy to occur when people perform actions, making the detection result more accurate. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] The accompanying drawings are used to provide a further understanding of the present application and constitute a part of the specification. Together with the embodiments of the present application, they are used to explain the present application and do not constitute a limitation of the present application. In the accompanying drawings:

[0016] Figure 1 A block diagram of the recognition method for action and posture detection;

[0017] Figure 2 Block diagram of the recognition system for action and posture detection. DETAILED DESCRIPTION

[0018] Exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although exemplary embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments set forth herein. Rather, these embodiments are provided to enable a more thorough understanding of the present disclosure and to fully convey the scope of the present disclosure to those skilled in the art.

[0019] A recognition method for performing motion and posture detection, used for detecting motion detection of target objects, such as Figure 1 As shown, including:

[0020] S11, real-time collection of human body image information or receiving human body image information from a mobile terminal;

[0021] The tester can record a video of themselves performing the specified action for the test item in front of a mobile phone camera or video camera, or take a photo of a specific pose and send it to the human body image information acquisition unit of the recognition system. Alternatively, they can complete real-time acquisition of the specified action or pose in front of the human body image information acquisition unit (including the camera device) of the recognition system. Specifically, for real-time acquisition, the test environment includes a detection area, a binocular camera located in front of the detection area, and the recognition system connected to the binocular camera. Large amounts of human body data are acquired in advance, processed, and then subjected to machine learning to generate a human key point detection model. When the target object enters the detection area, the human key point detection model is invoked to detect facial and body key points. Similarly, video information from the mobile device is input into the human key point detection model and then invoked to detect facial and body key points. Each key point in each frame corresponds to a set of three-dimensional data, including coordinate values ​​in the length, height, and depth directions, represented by Ci(xi,yi,zi).

[0022] S12, performing facial key point detection and body key point detection on the target object according to the human body image information;

[0023] The human body image information is input into the human body key point detection model. Common human body key point detection models include PoseNet, openpose, and mediapipe models. In this embodiment, the PoseNet model, a commonly used model currently used to obtain human body key points, is adopted, and facial key points and body key points are obtained according to the PoseNet model. Among them, facial key points include nose, inner corner of left eye, pupil of left eye, outer corner of left eye, inner corner of right eye, pupil of right eye, outer corner of right eye, left ear, right ear, left corner of mouth, and right corner of mouth; body key points include left shoulder, right shoulder, left elbow, right elbow, left wrist, right wrist, left hip, right hip, left knee, right knee, left ankle, right ankle, left heel, right heel, left toe, and right toe. In this embodiment, they are represented by the serial numbers in Table 1 respectively:

[0024] Table 1

[0025]

[0026]

[0027] S13: When the human body image information is a picture, an evaluation baseline is automatically generated according to the picture, and a static detection is performed on the target object according to the evaluation baseline.

[0028] Here, a human static posture detection model is used: First, images of the human body in different postures are acquired and annotated to generate a sample set, which is then split into a training set and a test set. Joint points in the human posture sample images in the training and test sets are identified and connected to obtain joint lines for the human posture sample images. An initial human posture detection model is trained based on the joint lines and annotation information of the human posture sample images in the training set. Finally, the initial human posture detection model is tested based on the joint lines of the human posture sample images in the test set using the stochastic gradient descent method to obtain a static human posture detection model. Static detection includes leg length and discrepancy testing, limb tilt detection, head and neck and spinal curvature detection, and head and neck and scoliosis detection.

[0029] Furthermore, static detection of the target object includes extracting target key points from the image, connecting the target key points in a predetermined connection order to obtain a key point connection line, and comparing the key point connection line with a corresponding evaluation baseline to obtain a static detection result. It should be noted that generating an evaluation baseline is a common method in image recognition.

[0030] Taking "head, neck and spinal curvature detection" as an example, the target key points are the nose C1, left shoulder C12, and right shoulder C13. The body key point data C1(x1, y1, z1), C12(x12, y12, z12), and C13(x13, y13, z13) of the nose C1, left shoulder C12, and right shoulder C13 need to be obtained. The center point position C0(x0, y0, z0) of the line connecting the left and right shoulders is further calculated. The nose C1 and the center point C0 of the line connecting the left and right shoulders are connected to obtain the relative angle A with the vertical midline of the human body, which is the target detection result, the head, neck and spinal curvature angle A.

[0031] In addition, the target key points for the leg length test include the left hip C18, left knee C20, left ankle C22, right hip C19, right knee C21, and right ankle C23. By comparing the lengths of the hip-knee-ankle line on both sides of the body, the presence of leg length discrepancy is identified. The target key points for limb tilt detection include the left ear C8, right ear C9, left shoulder C12, right shoulder C13, left hip C18, right hip C19, left knee C20, and right knee C21. The degree of limb tilt is identified by detecting the angle between the connection of the two ears, two shoulders, two hips, and two knees and the horizontal line. The head, neck, and spine scoliosis detection identifies the degree of scoliosis of the head, neck, and spine by measuring the angle between the frontal head, neck, and shoulder line and the vertical midline of the human body.

[0032] S14, when the human body image information is a video, the target key points of the target object are dynamically detected according to the changing trend of the target key point position and the follow-up changes of the associated key points associated with the target key points during the target object completes the specified action of the item to be tested in the video; the target key points are some key points among the facial key points and the body key points.

[0033] The dynamic detection includes joint range of motion detection of the shoulder, elbow, wrist, neck, thoracic, sacral, hip, knee and ankle joints, and segmental range of motion detection, including abduction, adduction, external rotation, internal rotation, etc. The target key points referred to in this embodiment refer to some or all of the 27 key points listed in Table 1.

[0034] Furthermore, dynamic detection of the target object is performed, including:

[0035] (11) Define the associated key points related to the target key point, and locate each key point using three-dimensional coordinates. Specifically, during dynamic detection, a data list is generated for the target key point in each frame of the video. The data list includes the three-dimensional coordinate value, timestamp, and GRAY value of the target key point. n is the total number of facial key point detections and body key points, that is, a frame can form a dictionary of n data list sets. Finally, a dictionary consisting of m groups of data list sets is formed, where m is the total number of frames in the video. During dynamic detection, each target key point has at least two associated key points involved in its movement.

[0036] (12) Sequentially connect the target key points and the associated key points in each frame, and draw multiple lines connecting the target key points and the associated key points, wherein the lines at least include line segments along two different directions of the target key points, extract the angles formed by the lines at the target key points, and obtain the angle set ρ = {ρ1, ρ2, ..., ρ i}, where i is the number of video frames.

[0037] (13) Determine the maximum value of the angle ρ within a given time period max and the minimum value ρmin , ρ max =max{ρ1,ρ2,...,ρ i},ρ min =min{ρ1,ρ2,...,ρ i}, that is, the joint activity detection result of the target key point is obtained, and the range of activity is ρ min ~ρ max .

[0038] Taking "right shoulder joint range of motion detection" as an example, the target key point is the right shoulder. According to the pre-defined matrix diagram analysis, the associated key points are the right elbow and the right hip. Obtain the body key point data C13 (x13, y13, z13), C15 (x15, y15, z15), and C19 (x19, y19, z19) of the right shoulder C13, right elbow C15, and right hip C19, and connect the right shoulder C13 and the right elbow C15, and the right shoulder C13 and the right hip C19 to obtain the right shoulder joint motion angle B. Assuming that the established time duration is 90s and 30 frames per second, 2700 frames of the target object during the detection process can be obtained, and the data set of the right shoulder joint motion angle ρ (0-2699) can be obtained, and the right shoulder joint motion angle ρ can be screened to determine the right shoulder joint motion angle ρ. min and ρ max , which is the target detection result right shoulder joint range of motion ρ min ~ρ max .

[0039] (14) By identifying changes in the associated key point data list in the dictionary, the detection result is corrected according to the changes in the associated key points. The change range of the associated key point coordinates in the associated data list caused by the target object completing the specified action of the item to be tested is monitored, and the detection result is corrected according to the change range.

[0040] It's worth noting that, under normal circumstances, when a target subject performs a specified action, all key points related to the item being tested should be active, while other unrelated key points should remain inactive. However, in reality, when people perform actions, muscle compensation occurs to a greater or lesser extent, and this is more pronounced in people with impaired motor skills. Therefore, while extracting the target key points, it's important to pay attention to changes in the associated key points associated with them, and to adjust the test results accordingly.

[0041] Still taking the "right shoulder joint range of motion detection" as an example, in an ideal state, when the shoulder joint moves, other joints are in a static state. However, people with impaired or weak motor ability are prone to hip joint muscle compensation. Therefore, the key point data of the right knee C21 (x21, y21, z21) is obtained, and the angle D between the shoulder-hip line C13C19 and the hip-knee line C19C21 is further obtained. When the angle D changes, it is actively corrected to the target detection result of the test item to be tested, the right shoulder joint range of motion. For example, the right shoulder joint range of motion range ρ min ~ρ max is 0-180°, and the hip joint compensation angle D is 20°, then the user's right shoulder joint range of motion ρ min ~ρ max 0~160°.

[0042] A recognition system for motion and posture detection is used to implement the above recognition method, such as Figure 2 As shown, it includes: a human body image information acquisition unit 100; a judgment unit 200, used to judge whether the human body image information is a picture or a video; a static detection unit 300, used to automatically generate an evaluation baseline according to the picture when the human body image information is a picture, and perform static detection on the target object according to the evaluation baseline; a dynamic detection unit 400, used to perform dynamic detection on the target key points of the target object according to the change trend of the target key point position and the follow-up change of the associated key points associated with the target key points during the target object completes the specified action of the item to be tested in the video when the human body image information is a video.

[0043] Obviously, those skilled in the art may make various changes and modifications to this application without departing from the spirit and scope of this application. Thus, if these modifications and variations of this application fall within the scope of the claims of this application and their equivalents, this application is intended to include these modifications and variations.

Claims

1. A recognition method for motion and posture detection, characterized in that: include: Real-time collection of human image information or receiving human image information from a mobile terminal; Performing facial key point detection and body key point detection on the target object based on the human body image information; When the human body image information is a picture, an evaluation baseline is automatically generated according to the picture, and a static detection is performed on the target object according to the evaluation baseline; When the human body image information is a video, the target key points of the target object are dynamically detected based on the changing trend of the position of the target key points and the follow-up changes of the associated key points associated with the target key points during the target object completes the specified action of the test item in the video, so as to obtain the joint mobility detection result of the target key points; When performing dynamic detection, each target key point action involves at least 2 associated key points; Monitor the change in the coordinates of the associated key points caused by the target object completing the specified action of the item to be tested, and correct the detection result according to the change.

2. A recognition method for motion and posture detection according to claim 1, characterized in that: The human body image information is input into a human body key point detection model to obtain facial key points and body key points.

3. The method for detecting movement and posture according to claim 1, wherein: The dynamic detection of the target object includes: Define the associated key points related to the target key point, and locate each key point through three-dimensional coordinates; The target key points and associated key points in each frame are sequentially connected to obtain multiple lines connecting the target key points and associated key points, wherein the lines at least include line segments along two different directions of the target key points, and the angles formed by the lines at the target key points are extracted to obtain the angle set. , where i is the maximum value of the angle determined by the number of video frames within a given time period and minimum value , , that is, the joint activity detection results of the target key points are obtained, and the range of motion is .

4. The method for detecting movement and posture according to claim 3, wherein: The target key points of the dynamic detection include joint mobility detection at the shoulders, elbows, wrists, neck, chest, sacrum, hips, knees and ankles.

5. The method for detecting movement and posture according to claim 3, wherein: During dynamic detection, a data list is generated for the target key points in each frame of the video. The data list includes the three-dimensional coordinate value, timestamp and gray value of the target key points. n is the total number of facial key point detections and body key points, that is, one frame of the picture can form a dictionary of n data list sets; finally, a dictionary of m groups of data list sets is formed, where m is the total number of frames of the video. By identifying the changes in the associated key point data lists in the dictionary, the detection results are corrected according to the change amplitude of the associated key points.

6. The method for detecting movement and posture according to claim 1, wherein: The static detection of the target object includes: Extract the target key points in the image, connect the target key points according to the preset connection order, obtain the key point connection line, compare the key point connection line with the automatically generated evaluation baseline, and obtain the static detection result.

7. A recognition method for motion and posture detection according to claim 6, characterized in that: The static tests include leg length test, limb tilt test, head neck and spine curvature test and head neck and spine scoliosis test.

8. The recognition method for motion and posture detection according to claim 1, characterized in that: The facial key points include nose, inner corner of left eye, pupil of left eye, outer corner of left eye, inner corner of right eye, pupil of right eye, outer corner of right eye, left ear, right ear, left corner of mouth and right corner of mouth; the body key points include left shoulder, right shoulder, left elbow, right elbow, left wrist, right wrist, left hip, right hip, left knee, right knee, left ankle, right ankle, left heel, right heel, left toe and right toe.

9. A recognition system for performing motion and posture detection, used to implement the recognition method according to any one of claims 1 to 8, characterized in that: include: Human body image information acquisition unit; a judging unit, configured to judge whether the human body image information is a picture or a video; a static detection unit, configured to automatically generate an evaluation baseline based on a picture when the human body image information is a picture, and perform static detection on the target object based on the evaluation baseline; a dynamic detection unit for dynamically detecting target key points of the target object based on a change trend of the position of the target key points and a follow-up change of associated key points associated with the target key points during the target object completing a specified action of the test item in the video when the human body image information is in the form of a video; Get the joint mobility detection results of the target key points; During dynamic detection, each target key point involves at least two associated key points when it moves; the change range of the associated key point coordinates caused by the target object completing the specified action of the item to be tested is monitored, and the detection result is corrected according to the change range.

Images