A running recognition method

Abstract

The present application provides a running recognition method, including the steps of: collecting an initial image during exercise; detecting a rectangular area where the human body is located in the initial image; intercepting the rectangular area to detect human key points, and connecting the human key points according to the positional relationship to form a human skeleton diagram; obtaining the human body perspective in the rectangular area and adjusting the aspect ratio of the human skeleton diagram; performing running recognition by analyzing the distribution relationship of the adjusted human skeleton diagram. The running recognition method of the present application detects human key points by intercepting the rectangular area, connects the human key points according to the positional relationship to form a human skeleton diagram, obtains the human body perspective in the rectangular area, adjusts the aspect ratio of the human skeleton diagram, converts the perspective of the human skeleton posture, and then analyzes the distribution relationship of the adjusted human skeleton diagram for running recognition, which not only improves the adaptability of the running recognition algorithm and can improve the running recognition accuracy from the two-dimensional image level, but also reduces the requirement for computing power.

Description

Technical Field

[0001] This application belongs to the technical field of image recognition, and more specifically, relates to a running recognition method. Background Art

[0002] The classification and recognition of sports fitness movements are widely applied to scenarios such as the evaluation of fitness activities and the assistance of fitness training. Running is a convenient daily physical exercise method and an effective way of aerobic respiration.

[0003] In the early days, the running posture was judged manually by coaches. In the prior art, the running posture of the human body can be detected through an image recognition algorithm. However, since the running postures of the human body obtained from different perspectives are different, the existing running recognition algorithm can only recognize the human body posture at a fixed perspective, including the human body orientation and the pitch angle. Thus, this running recognition algorithm can only be applied to occasions such as treadmills with fixed positions and perspectives, and the running recognition rate is insufficient in complex outdoor scenarios, thereby limiting the wide application of this technology. Summary of the Invention

[0004] The purpose of the embodiments of this application is to provide a running recognition method to solve the technical problems of insufficient adaptability and poor recognition accuracy existing in the running recognition process of the prior art.

[0005] To achieve the above purpose, the technical solution adopted by this application is: to provide a running recognition method, including the following steps:

[0006] Collect the initial image during the movement process;

[0007] Detect the rectangular area where the human body is located in the initial image;

[0008] Intercept the rectangular area to detect the key points of the human body, and connect the key points of the human body according to the positional relationship to form a human skeleton diagram;

[0009] Obtain the human body perspective in the rectangular area and adjust the aspect ratio of the human skeleton diagram;

[0010] Perform running recognition by analyzing the distribution relationship of the adjusted human skeleton diagram.

[0011] Preferably, the method for obtaining the human body perspective in the rectangular area and adjusting the aspect ratio of the human skeleton diagram includes the following steps:

[0012] Identify the face orientation in the rectangular area;

[0013] Horizontally adjust the width of the human skeleton diagram according to the face orientation;

[0014] Obtain a side human skeleton diagram.

[0015] Preferably, the method for horizontally adjusting the width of the human body skeleton diagram according to the face orientation includes:

[0016] Set the direction of the human frontal face as 0 degrees, the face orientation in the current rectangular area as θ, the width of the rectangular area as w, and the width of the extended human body skeleton diagram as W. Then the formula for W is:

[0017]

[0018] Preferably, after obtaining the side human body skeleton diagram, the method further includes the steps of:

[0019] Obtain the width-to-height ratio of the side human body skeleton diagram;

[0020] Stretch or compress the height of the side human body skeleton diagram so that the width-to-height ratio of the side human body skeleton diagram is within the set width-to-height ratio threshold range.

[0021] Preferably, the method for running recognition by analyzing the distribution relationship of the adjusted human body skeleton diagram includes the following steps:

[0022] Analyze the geometric relationship between the coordinate positions of the human key points;

[0023] Judge whether the geometric relationship conforms to the geometric threshold.

[0024] Preferably, the method for analyzing the geometric relationship between the coordinate positions of the human key points includes the following steps:

[0025] Obtain the length a of the upper arm, the length b of the forearm, and the distance c from the shoulder to the wrist of the same hand in the human body skeleton diagram respectively;

[0026] Let α be the angle between the upper arm and the forearm, then

[0027]

[0028] Judge whether α conforms to the arm angle threshold.

[0029] Preferably, the method for analyzing the geometric relationship between the coordinate positions of the human key points includes the following steps:

[0030] Obtain the length d of the thigh, the length e of the calf, and the distance f from the hip to the foot of the same leg in the human body skeleton diagram respectively;

[0031] Let β be the angle between the thigh and the calf, then

[0032]

[0033] Judge whether β conforms to the leg angle threshold.

[0034] Preferably, the method for running recognition by analyzing the distribution relationship of the adjusted human skeleton diagram includes the following steps:

[0035] Analyze the characteristic components corresponding to the coordinate set of key points;

[0036] Judge whether the characteristic components meet the set threshold.

[0037] Preferably, the method for analyzing the characteristic components corresponding to the coordinate set of key points includes the following steps:

[0038] Let the elbow key point of the same arm be A, the shoulder key point be B, the wrist key point be C, and W be the characteristic component corresponding to this arm; then

[0039]

[0040] Judge whether W meets the leg angle threshold.

[0041] Compared with the prior art, the running recognition method provided by this application detects human key points by intercepting a rectangular area, connects the human key points according to the position relationship to form a human skeleton diagram, obtains the human perspective in the rectangular area, adjusts the aspect ratio of the human skeleton diagram, converts the perspective of the human skeleton posture, and then analyzes the distribution relationship of the adjusted human skeleton diagram for running recognition. This not only improves the adaptability of the running recognition algorithm and can improve the running recognition accuracy from the two-dimensional image level, but also reduces the requirement for computing power. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following will briefly introduce the drawings required for use in the embodiments or the description of the prior art. Obviously, the following drawings are only some embodiments of the present application. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0043] Figure 1 It is a schematic flowchart of the running recognition method provided by the embodiment of the present application;

[0044] Figure 2 For Figure 1 the effect schematic diagram when the rectangular area where the human body is located in the initial image is detected by the running recognition method in

[0045] Figure 3 For Figure 2 the effect schematic diagram obtained by intercepting the rectangular area from the initial image in

[0046] Figure 4 It is a schematic diagram of the distribution of 18 key points of the human body detected by using openpose;

[0047] Figure 5 It is a schematic diagram of a human skeleton diagram obtained based on the Figure 3 rectangular region in;

[0048] Figure 6 It is a schematic diagram of a side human skeleton diagram obtained by stretching the width of the human skeleton diagram based on the Figure 5 in;

[0049] Figure 7 It is a human skeleton diagram obtained by compressing the height of the side human skeleton diagram based on the Figure 6 in;

[0050] Figure 8 It is a schematic diagram of a human skeleton diagram marked with a, b, c, d, e, f, α, β based on the Figure 7 in;

[0051] Figure 9 It is a schematic diagram of a human skeleton diagram marked with A, B, C based on the Figure 7 in. Detailed implementation manners

[0052] In order to make the technical problems, technical solutions and beneficial effects to be solved by the present application clearer and more understandable, the present application will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0053] It should be noted that when an element is referred to as being "fixed to" or "disposed on" another element, it can be directly on the other element or indirectly on the other element. When an element is referred to as being "connected to" another element, it can be directly connected to the other element or indirectly connected to the other element.

[0054] It should be understood that the orientation or positional relationship indicated by the terms "length", "width", "upper", "lower", "front", "rear", "left", "right", "vertical", "horizontal", "top", "bottom", "inner", "outer", etc. is based on the orientation or positional relationship shown in the accompanying drawings, and is only for the convenience of describing the present application and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore should not be construed as a limitation to the present application.

[0055] In addition, the terms "first" and "second" are only used for descriptive purposes and cannot be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Thus, the features defined with "first" and "second" may explicitly or implicitly include one or more of such features. In the description of the present application, "a plurality of" means two or more unless otherwise specifically defined.

[0056] Please refer to Figure 1 simultaneously. Now, the running recognition method provided by the embodiments of this application will be described. The running recognition method includes the following steps:

[0057] Step S1, collect the initial image during the exercise process;

[0058] Step S2, detect the rectangular area where the human body is located in the initial image;

[0059] Step S3, intercept the human body key points in the rectangular area and connect the human body key points according to the positional relationship to form a human body skeleton diagram;

[0060] Step S4, obtain the human body perspective in the rectangular area and adjust the aspect ratio of the human body skeleton diagram;

[0061] Step S5, perform running recognition by analyzing the distribution relationship of the adjusted human body skeleton diagram.

[0062] It can be understood that in step S2, please refer to Figure 2 and Figure 3 simultaneously. For the detection of the rectangular area where the human body is located in the initial image, the yolox algorithm can be used to detect the rectangular area where the human body is located. When there are multiple human body targets in the picture, multiple rectangular areas may be detected, and running recognition can be performed on all rectangular areas simultaneously. Compared with the traditional heat map human body detection method, this application can not only collect in real time, but also perform running recognition on any frame in the video. In addition, it does not require a thermal imager, which can save costs.

[0063] In step S3, please refer to Figure 3 and Figure 4 simultaneously. For the method of detecting human body key points, the openpose can be used to detect 18 key points of the human body. The 18 key points include the left eye, right eye, left ear, right ear, nose, left shoulder, right shoulder, neck, left elbow, right elbow, left wrist, right wrist, left thigh, right thigh, left knee, right knee, left foot and right foot, and the key points are connected according to the positional relationship to form a human body skeleton diagram.

[0064] In steps S4-5, please refer to Figure 6 and Figure 7, since the geometric relationships among the coordinate positions of human key points obtained by analyzing the human body in different postures are all different, and the characteristic components corresponding to the coordinate sets of the key points have fixed characteristics. Therefore, a threshold for the running state of the human body can be preset, and the geometric relationships among the coordinate positions of human key points and whether the characteristic components corresponding to the coordinate sets of the key points meet the set threshold are analyzed. If the threshold is met, it is recognized and judged as the running state; if the threshold is not met, it is recognized and judged as the non-running state. This calculation method of the ratio can effectively avoid the influence on the extraction result of the feature vector caused by the change in the body shape of the person due to the distance of the person. However, since the camera can only capture the image of one side, in the two-dimensional image, the human skeleton diagram obtained according to step S3 does not include depth. Therefore, the human skeleton diagrams obtained from different human perspectives will be different. Therefore, in step S3, by first obtaining the human perspective in the rectangular area and correspondingly adjusting the aspect ratio of the human skeleton diagram according to the human perspective, the perspective of the human skeleton diagram is converted, and the technical problem that needs to be solved at the three-dimensional level is solved by two-dimensional technology. Then, the distribution relationship of the adjusted human skeleton diagram is analyzed through step S5 for running recognition.

[0065] The running recognition method provided by this application, compared with the prior art, detects human key points by intercepting a rectangular area, connects the human key points according to the position relationship to form a human skeleton diagram, obtains the human perspective in the rectangular area, adjusts the aspect ratio of the human skeleton diagram, converts the perspective of the human skeleton posture, and then analyzes the distribution relationship of the adjusted human skeleton diagram for running recognition. This not only improves the adaptability of the running recognition algorithm and can improve the running recognition accuracy from the two-dimensional image level, but also reduces the requirement for computing power.

[0066] In another embodiment of this application, please refer to Figures 5 to 6 , in step S4, the method of obtaining the human perspective in the rectangular area and adjusting the aspect ratio of the human skeleton diagram includes the following steps:

[0067] Identify the face orientation in the rectangular area;

[0068] Horizontally adjust the width of the human skeleton diagram according to the face orientation;

[0069] Obtain a side human skeleton diagram.

[0070] It can be understood that to identify the face orientation in the rectangular area, a face orientation recognition model based on an artificial neural network can be used. This face orientation recognition model can be a multi-classification model, that is, the face orientation is divided into multiple categories according to the angle range. After the rectangular area image is input into this face orientation recognition model, the category corresponding to the face orientation can be output to achieve the purpose of obtaining the face orientation in the rectangular area. Since the side human skeleton diagram can more vividly reflect the running posture of the human body, and the limb skeleton postures of the human body during movement can be reflected in the side human skeleton diagram in the two-dimensional space, it is necessary to horizontally adjust the width of the human skeleton diagram according to the face orientation to obtain the side human skeleton diagram.

[0071] The method for horizontally adjusting the width of the human skeleton diagram according to the face orientation includes: setting the front face orientation direction of the human body as 0 degrees, the face orientation in the current rectangular area as θ, the width of the rectangular area as w, and the width of the extended human skeleton diagram as W. Then the formula for W is:

[0072]

[0073] For example, when the face orientation in the identified rectangular area is 30°, the width of the human skeleton diagram is horizontally adjusted by 2 times according to the face orientation to achieve the perspective of the human skeleton posture after conversion. The perspective of the human skeleton posture after conversion is to obtain the side human skeleton diagram, and the running is recognized by analyzing the distribution relationship of the side human skeleton diagram to improve the accuracy of running recognition.

[0074] Furthermore, please refer to Figures 6 to 7 together. After obtaining the side human skeleton diagram in step S4, the following steps are also included:

[0075] Obtain the aspect ratio of the width to the height of the side human skeleton diagram;

[0076] Stretch or compress the height of the side human skeleton diagram so that the aspect ratio of the width to the height of the side human skeleton diagram is within the set aspect ratio threshold range.

[0077] It can be understood that since the pitch angle during image acquisition is uncertain, in a two-dimensional image, an image taken with a level view of the human body is required to reflect the distribution relationship of the real human body skeleton diagram. However, in a top-down view, the height of the human body becomes smaller, but the body width changes relatively little. In a bottom-up view, the height of the human body becomes larger, and the body width also changes relatively little. Thus, by obtaining the aspect ratio of the width and height of the side human body skeleton diagram and stretching or compressing the height of the side human body skeleton diagram, the aspect ratio of the side human body skeleton diagram is made to be within the set aspect ratio threshold range. For example, if the aspect ratio of the side human body skeleton diagram is less than the minimum value of the aspect ratio threshold range, compress the height of the side human body skeleton diagram; if the aspect ratio of the side human body skeleton diagram is greater than the maximum value of the aspect ratio threshold range, stretch the height of the side human body skeleton diagram; if the aspect ratio of the side human body skeleton diagram is within the aspect ratio threshold range, maintain the aspect ratio of the side human body skeleton diagram.

[0078] In another embodiment of the present application, please also refer to Figure 8 , in step S5, a method for running recognition by analyzing the distribution relationship of the adjusted human body skeleton diagram includes the following steps:

[0079] Analyze the geometric relationship between the coordinate positions of the human body key points;

[0080] Judge whether the geometric relationship conforms to the geometric threshold.

[0081] It can be understood that since the geometric relationships between the coordinate positions of the human body key points obtained by analyzing the human running posture all have fixed characteristics, therefore, a geometric threshold between the coordinate positions of the human body key points in the running state can be preset. If it conforms to the threshold, it is recognized and judged as the running state; if it does not conform to the threshold, it is recognized and judged as the non-running state.

[0082] Specifically, please also refer to Figure 8 , in step S5, the method for analyzing the geometric relationship between the coordinate positions of the human body key points includes the following steps:

[0083] Respectively obtain the length a of the upper arm, the length b of the forearm, and the distance c from the shoulder to the wrist of the same hand in the human body skeleton diagram;

[0084] Let α be the included angle between the upper arm and the forearm, then

[0085] Judge whether α conforms to the arm included angle threshold.

[0086] It can be understood that when calculating the left hand, the shoulder can be understood as the left shoulder. Similarly, when calculating the right hand, the shoulder can be understood as the right shoulder. The same applies to the thigh, foot, wrist, and arm mentioned later. Since the upper arm and forearm of a person will form a certain angle in the running state, for example, the arm angle threshold can be 60° to 120°, that is, if 60° ≤ α ≤ 120°, it is determined that the person is running; otherwise, it is determined that the person is not in the running state.

[0087] Specifically, please refer to Figure 8 simultaneously. In step S5, the method for analyzing the geometric relationship between the coordinate positions of human key points includes the following steps:

[0088] Obtain the length d of the thigh, the length e of the lower leg, and the distance f from the thigh to the foot of the same leg in the human skeleton diagram respectively;

[0089] Let β be the angle between the thigh and the lower leg, then

[0090]

[0091] Determine whether β meets the leg angle threshold.

[0092] It can be understood that since the thigh and the lower leg of a person will form a certain angle in the running state, for example, the leg angle threshold can be 45° to 140°, that is, if 45° ≤ α ≤ 140°, it is determined that the person is running; otherwise, it is determined that the person is not in the running state.

[0093] In another embodiment of the present application, please refer to Figure 9 simultaneously. In step S5, the method for running recognition by analyzing the distribution relationship of the adjusted human skeleton diagram includes the following steps:

[0094] Analyze the characteristic components corresponding to the coordinate set of key points;

[0095] Determine whether the characteristic components meet the set threshold.

[0096] It can be understood that since the characteristic components corresponding to the coordinate set of key points obtained by analyzing the human running posture all have fixed characteristics, therefore, the threshold of the characteristic components corresponding to the coordinate set of key points of the human body in the running state can be preset. If it meets the threshold, it is recognized and determined as the running state; if it does not meet the threshold, it is recognized and determined as the non-running state. This calculation method of characteristic components can effectively avoid the influence on the extraction result of the characteristic vector caused by the change of the human body shape due to the distance of the person.

[0097] Specifically, please refer to Figure 9 simultaneously. In step S5, the method for analyzing the characteristic components corresponding to the coordinate set of key points includes the following steps:

[0098] Let the elbow key point of the same arm be A, the shoulder key point be B, the wrist key point be C, and W be the feature component corresponding to this arm; then

[0099]

[0100] Judge whether W meets the leg angle threshold.

[0101] It can be understood that for the two arms, the human key point relationship features are extracted to obtain the corresponding feature components. Here, first, the three key points on the arm are represented by two vectors respectively to represent the torso between the key points, and the relationship between the two vectors is obtained. Further, the feature component corresponding to this relationship feature is calculated. In the above formula, it is actually equivalent to calculating the projection length of vector AB on vector AC, and then calculating its ratio to vector BC, so that W can be used as the relationship feature component between the forearm and the upper arm. If W meets the relationship feature component threshold, it is recognized as the running state, otherwise it is recognized as the non-running state.

[0102] Similarly, by using the method of analyzing the feature components corresponding to the coordinate sets of the key points as described above, the human key point relationship features of the two legs can also be extracted to obtain the corresponding feature components.

[0103] The above are only the preferred embodiments of the present application and are not intended to limit the present application. Any modifications, equivalent replacements, and improvements made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A running recognition method, characterized in that, It includes the following steps: Collect the initial images during the movement process; Detect the rectangular area where the human body is located in the initial image; Crop the rectangular area to detect the human body key points, and connect the human body key points according to the positional relationship to form a human body skeleton diagram; Obtain the human body perspective in the rectangular area and adjust the aspect ratio of the human body skeleton diagram; The method of obtaining the human body perspective in the rectangular area and adjusting the aspect ratio of the human body skeleton diagram includes the following steps: Identify the face orientation in the rectangular area; Horizontally adjust the width of the human body skeleton diagram according to the face orientation; Obtain the side view human body skeleton diagram; After obtaining the side view human body skeleton diagram, it further includes the steps: Obtain the aspect ratio of the side view human body skeleton diagram; Stretch or compress the height of the side view human body skeleton diagram so that the aspect ratio of the side view human body skeleton diagram is within the set aspect ratio threshold range; Perform running recognition by analyzing the distribution relationship of the adjusted human body skeleton diagram.

2. The running recognition method according to claim 1, wherein The method of horizontally adjusting the width of the human body skeleton diagram according to the face orientation includes: Set the front face orientation direction of the human body to 0 degrees, the face orientation in the current rectangular area to θ, the width of the rectangular area to w, and the width of the stretched human body skeleton diagram to W. Then the formula for W is:

3. The running recognition method according to claim 1, wherein The method of making the aspect ratio of the side view human body skeleton diagram within the set aspect ratio threshold range includes: If the aspect ratio of the side view human body skeleton diagram is less than the minimum value of the aspect ratio threshold range, compress the height of the side view human body skeleton diagram. If the aspect ratio of the side view human body skeleton diagram is greater than the maximum value of the aspect ratio threshold range, stretch the height of the side view human body skeleton diagram. If the aspect ratio of the side view human body skeleton diagram is within the aspect ratio threshold range, keep the aspect ratio of the side view human body skeleton diagram.

4. The running recognition method according to any one of claims 1 to 3, characterized in that, The method of performing running recognition by analyzing the distribution relationship of the adjusted human body skeleton diagram includes the following steps: Analyze the geometric relationship between the coordinate positions of the human body key points; Judge whether the geometric relationship meets the geometric threshold.

5. The running recognition method according to claim 4, wherein The method of analyzing the geometric relationship between the coordinate positions of the human body key points includes the following steps: Respectively obtain the length a of the upper arm, the length b of the lower arm, and the distance c from the shoulder to the wrist of the same hand in the human body skeleton diagram; Let α be the angle between the upper arm and the lower arm, then Judge whether α meets the arm angle threshold.

6. The running recognition method according to claim 4, wherein, The method of analyzing the geometric relationship between the coordinate positions of the human body key points includes the following steps: Respectively obtain the length d of the thigh, the length e of the lower leg, and the distance f from the hip to the foot of the same leg in the human body skeleton diagram; Let β be the angle between the thigh and the lower leg, then Judge whether β meets the leg angle threshold.

7. The running recognition method according to any one of claims 1 to 3, characterized in that, The method of performing running recognition by analyzing the distribution relationship of the adjusted human body skeleton diagram includes the following steps: Analyze the characteristic components corresponding to the coordinate set of the key points; Judge whether the characteristic components meet the set threshold.

8. The running recognition method according to claim 7, characterized in that The method of analyzing the characteristic components corresponding to the coordinate set of the key points includes the following steps: Let the elbow key point of the same arm be A, the shoulder key point be B, and the wrist key point be C. Let W be the characteristic component corresponding to this arm. Then Judge whether W meets the leg angle threshold.

Images