A method for human key point detection of humanoid robot assisted fitness

Abstract

The application relates to the technical field of robots, in particular to a humanoid robot assisted fitness method for human key point detection, which comprises the following steps: collecting real-time motion videos of a person; using a MobileNetv3 network as a backbone of a YOLOV7 model, improving a Conv module of the YOLOV7 network model by using a Ghost Conv module to obtain an improved YOLOV7 network model, and performing target detection on the motion videos to obtain a human position; tracking the human target by using a Centriod Tracking algorithm and calculating a center of mass coordinate of the human; performing real-time detection on the human by using an improved Openpose network to obtain the position and number of human key points; calculating the angle, distance and displacement between the human key points to determine whether the action is standard; and the application solves the problem that the existing method cannot simultaneously consider the key point detection accuracy, real-time performance and light weight.

Description

Technical Field

[0001] This invention relates to the field of robotics, and more particularly to a method for humanoid robot-assisted fitness involving the detection of key points on the human body. Background Technology

[0002] In recent years, with the rise of fitness activities, a new type of device, the fitness mirror, has become popular. When turned off, the fitness mirror is no different from a dressing mirror we use every day. When turned on, a virtual coach image appears on the mirror. When the user follows the virtual coach to perform various fitness or rehabilitation movements, the fitness mirror will detect and capture the user's movements through its built-in camera and display them on the fitness mirror. The user's movements are then compared with the virtual coach's image movements to give an intelligent score. The user can also judge whether their movements are standard in this process.

[0003] However, fitness mirrors have two major pain points in the market. First, fitness mirrors have errors in detecting user movements, especially when the user's movements are very complex or when there is overlap of body parts. Second, the user's movements are often displayed on the fitness mirror as a very simple flat image, making it difficult for the user to intuitively feel the bending or angle of the movement at that time, which will affect the effect of fitness exercise. Summary of the Invention

[0004] To address the shortcomings of existing methods, this invention solves the problem that existing methods cannot simultaneously achieve high accuracy, real-time performance, and lightweight design in key point detection.

[0005] The technical solution adopted in this invention is: a method for humanoid robot-assisted fitness with human body key point detection, comprising the following steps:

[0006] Step 1: Collect real-time motion video of the person;

[0007] Step 2: Using the MobileNetv3 network as the backbone, the YOLOv7 network model is improved by using the Ghost Conv module to obtain the improved YOLOv7 network model. Then, the target detection is performed on the motion video to obtain the human body position.

[0008] Furthermore, the GhostConv operation process includes:

[0009] First, some features are obtained through ordinary convolution;

[0010] The output Y of a regular convolution is:

[0011] Y = X × f + b (1)

[0012] Where b is the bias constant, X is the input video frame data, and f is the convolution kernel;

[0013] Secondly, some features are subjected to inexpensive convolutional linear exchange to obtain the ghost feature map;

[0014] This can be expressed by the formula:

[0015] Y'=X*f'

[0016]

[0017] Where f' is the derivative of f, and Y'∈R h '×w' ×m * indicates a convolution operation, m is the number of channels for a regular convolution, and ψ i,j To perform linearly transform convolution operations, y' i To perform a linearly transformed convolution operation on Y'.

[0018] Step 3: Use the Centriole Tracking algorithm to track the human target and calculate the centroid coordinates of the human body;

[0019] Step 4: Improve the Openpose network to perform real-time human body detection and obtain the location and number of key points on the human body;

[0020] Furthermore, the Openpose network is improved by adopting the VGG network architecture, and the 7×7 convolution in the Openpose network is modified into three cascaded 3×3 depthwise separable convolution kernels, with a skip connection consisting of a 1×1 convolution kernel added between every three convolution kernels.

[0021] Furthermore, key points on the human body include: left eye, right eye, nose, neck, left shoulder, right shoulder, left elbow, right elbow, left wrist, right wrist, left hip, right hip, left knee, right knee, left ankle, and right ankle.

[0022] Step 5: Calculate the angles, distances, and displacements between key points to determine if the action is standard.

[0023] Furthermore, the formula for calculating the angle is:

[0024]

[0025] θ = arccos(cosθ)

[0026] Here, (x1,y1) and (x2,y2) are the coordinates of two adjacent key points.

[0027] Furthermore, the formula for distance is:

[0028]

[0029] Here, (x1,y1) and (x2,y2) are the coordinates of two adjacent key points.

[0030] Furthermore, the formula for displacement is:

[0031] W = VU = (x2 - x1, y2 - y1)

[0032] Here, (x1,y1) and (x2,y2) are the coordinates of two adjacent key points.

[0033] The beneficial effects of this invention are:

[0034] 1. The backbone network of the YOLOv7 object detection algorithm is replaced with the MobileNetv3 network, and the Conv module in YOLOv7 is improved with the GhostConv module to reduce the computing power requirements;

[0035] 2. An improved Openpose human keypoint detection network is adopted using depthwise separable convolution, resulting in better real-time detection performance;

[0036] 3. By calculating the angles, distances, and displacements between key points, it is determined whether the action is standard. To address the issue of correcting non-standard user actions, a humanoid robot is used to directly demonstrate standard actions, which is more intuitive. Attached Figure Description

[0037] Figure 1 This is a flowchart of the humanoid robot-assisted fitness method for detecting key human body points according to the present invention;

[0038] Figure 2 This is a schematic diagram of the GhostConv processing procedure;

[0039] Figure 3 This is a schematic diagram of the depthwise separable convolution used in this invention to replace the 7×7 convolution;

[0040] Figure 4 This is a schematic diagram of the depthwise separable convolution operation process;

[0041] Figure 5 These are the specific locations and numbers of the 16 key human body points used in this invention. Detailed Implementation

[0042] The present invention will be further described below with reference to the accompanying drawings and embodiments. The drawings are simplified schematic diagrams, which only illustrate the basic structure of the present invention in a schematic manner, and therefore only show the components related to the present invention.

[0043] like Figure 1 As shown, a method for humanoid robot-assisted fitness involving human key point detection includes the following steps:

[0044] Step 1: Collect real-time motion video of the person;

[0045] Step 2: Replace the backbone network of the YOLOv7 model with the MobileNetv3 network, and improve the Conv module in the YOLOv7 network model with the GhostConv module; use the improved YOLOv7 network to perform target detection on the motion video and obtain the bounding boxes of human body positions.

[0046] YOLOv7's object detection algorithm boasts high speed and a small parameter count. It consists of three parts: input, backbone network, and head network. The input part preprocesses the input image using methods such as data augmentation and adaptive anchor box calculation. The backbone network comprises several CBS modules, an ELAN layer, and an MP layer. The head network uses a Rep layer and a CBM module.

[0047] To enable target detection and human recognition on mobile devices, this invention replaces the original DarkNet53 backbone network of YOLOv7 with the lightweight backbone network MobileNetv3. This not only improves the recognition accuracy of the algorithm but also effectively reduces computing power. MobileNetv3 is an improvement on MobileNetv2, which can significantly reduce the number of model parameters while improving the model's feature representation ability.

[0048] Meanwhile, in order to improve the real-time performance of detection, this invention uses the GhostConv module to replace the convolution module in YOLOv7, so that the model can be compressed again. There would be a lot of redundancy when using traditional convolution for feature extraction.

[0049] Figure 2 This is a schematic diagram of the GhostConv processing procedure. First, a portion of the features are obtained through ordinary convolution. The specific process is as follows: For given data X∈R c×k×n Where c is the number of input channels, and h and w are the height and width of the input data, respectively; Y∈R h '×w' ×n The output feature map has a convolution kernel of f∈R. c×k×n h' and w' are the height and width of the output feature map, n is the number of channels, and k×k is the size of the convolutional kernel f. The operation of any convolutional layer used to generate n feature maps can be described by the formula:

[0050] Y = X × f + b (1)

[0051] Where b is the bias constant.

[0052] Then, the output feature Y is subjected to inexpensive convolutional linear exchange ψ. k The ghost feature map is obtained as shown in the following formula:

[0053] Y'=X*f' (2)

[0054]

[0055] In the formula, f' is the derivative of f, and Y'∈R h '×w' ×m * indicates a convolution operation, m is the number of channels for a regular convolution, and ψ i,j To perform linearly transform convolution operations, y' i To perform linearly transformed convolution operations on Y'; inexpensive convolution operations consume less computational power, making the network more lightweight.

[0056] Step 3: Using the Centriole Tracking algorithm, calculate the centroid coordinates of the human body based on the bounding box of the human body's position, and then track the human body.

[0057] Centroid Tracking is a real-time centroid tracking algorithm. It works by calculating the position change of the centroid of each target in each frame to track the target's motion trajectory. An improved YOLOv7 model is used to detect the initial frames of the motion video until a human figure appears. After determining the human's position, the centroid tracking algorithm calculates the human's centroid coordinates, and then uses the centroid's position information to match between adjacent frames, thus achieving continuous tracking of the target.

[0058] Target detection only occurs in the initial frame before the human body appears. Subsequent human body tracking is performed using the centroid tracking algorithm. Since the computational cost of the centroid tracking algorithm is much less than that of the improved YOLOv7 target detection algorithm, this method of combining the target detection algorithm with the centroid tracking algorithm can reduce computational overhead and optimize running speed.

[0059] The calculated centroid coordinates of the human body are used as its unique identifier. Human bodies (with unique identifiers) from the previous frame are matched with those in the current frame. The distance between the centroids of the previous and current frames is calculated to determine if they belong to the same human body, and the human body's position and centroid coordinates are updated accordingly. Simultaneously, human body keypoint detection is performed. If a human body with a unique identifier is not detected in multiple consecutive frames, it is considered lost and removed from the target list. The robot's head servo motor (with the camera on the robot's head) is controlled to rotate, continuously detecting human bodies until one is detected, at which point human body keypoint detection is performed again.

[0060] Step 4: Improve the Openpose human keypoint detection network by using depthwise separable convolution, and perform real-time detection of user keypoints;

[0061] OpenPose is an end-to-end multi-person pose estimation algorithm that uses a convolutional neural network (VGG19) as its feature extraction network. The feature extraction network primarily employs the VGG network architecture, adding a branch network to the VGG model. This branch network, together with the VGG19 convolutional neural network, forms the OpenPose feature extraction network. The OpenPose structure can be divided into two main parts. First, features are extracted using the VGG19 convolutional neural network. Then, the resulting feature maps are input into two different branches: the Part Affinity Fields Branch and the Part Confidence Maps Branch. The Part Affinity Fields Branch focuses on predicting the connectivity information between keypoints, accurately reflecting the relative positional relationships between keypoints on the human body. The Part Confidence Maps Branch is specifically used to predict the existence of each keypoint, and its output represents the confidence level of a particular keypoint's presence.

[0062] The 7×7 convolution in the original Openpose network structure can obtain a larger receptive field, but this also requires a lot of computing power. In order to improve the real-time performance of key point detection, this invention changes the 7×7 convolution kernel in the original Openpose network structure to three cascaded 3×3 depth-separable convolution kernels, and adds a 1×1 convolution kernel to form a skip connection between every three convolution kernels. Figure 3 This is a schematic diagram of the depth-separable convolution used in this invention to replace the 7×7 convolution.

[0063] Depthwise separable convolution can be divided into two parts: depthwise convolution and pointwise convolution. It breaks down the relationship between spatial dimension and channel dimension into two steps. Figure 4 This is a schematic diagram of a depthwise separable convolution operation. If the size of the input feature map is represented as S1×S1×T, the size of the convolution kernel is S2×S2×T, and the size of the output feature map is S3×S3×T, then for traditional convolution operations, each convolution kernel is convolved with each corresponding input feature map layer. Therefore, the formula for calculating the number of parameters is shown below.

[0064] L1=(S1×S1×T)×I

[0065] For depthwise convolution, multiple layers of convolution kernels convolve the input features separately to obtain a new feature map with the same number of input features. The number of parameters is shown below.

[0066] L dw = (S1×S1×1)×T

[0067] Raster convolution, on the other hand, uses a certain number of 1×1 convolution kernels to fuse intermediate features based on the previous step, and finally obtains the input feature map. The number of parameters is shown below.

[0068] L pw = (1×1×T)×I

[0069] The ratio of the total number of parameters between traditional convolution operations and depthwise separable convolution is shown below.

[0070]

[0071] Since the 7×7 convolution kernel is replaced, the amount of convolution operations will be reduced to 1 / 9 of the original, reducing the number of parameters and making the model more lightweight.

[0072] An improved OpenPose detection algorithm is used to perform real-time user detection. Considering that some detected human body key points are not of great use to the overall function of this invention, only 16 key points are retained, from top to bottom: left eye, right eye, nose, neck, left shoulder, right shoulder, left elbow, right elbow, left wrist, right wrist, left hip, right hip, left knee, right knee, left ankle, and right ankle. The specific location and numbering information of the 16 key points are as follows: Figure 5 As shown.

[0073] Step 5: Calculate the angles, distances, and displacements between the corresponding key points, and determine whether the action is standard.

[0074] Based on the key point data obtained in the previous step, calculate the angle, distance, and displacement indices. When calculating the indices, at least three key point numbers and location information need to be selected to obtain the angle value; two key point numbers and location information need to be selected to obtain the distance data; and at least one key point number and location information need to be selected to obtain the displacement.

[0075] Calculation of angle θ: Taking the shoulder, elbow, and wrist as examples, points P, A, and B are the identified key points of the left shoulder, left elbow, and left wrist. Starting from point P, form an angle with points A and B respectively. Calculate the angle using the following formula, ensuring it is between 0 and 180°. The calculation formula is as follows:

[0076]

[0077] θ = arccos(cosθ)

[0078] Here, (x1,y1) and (x2,y2) are the coordinates of two adjacent key points.

[0079] Calculation of distance D: Let the coordinates of two key points be M(x1,y1) and N(x2,y2). The formula for calculating the distance between the two key points is as follows:

[0080]

[0081] Calculation of displacement W: Displacement W is used to calculate the displacement data of key points corresponding to two frames before and after the user's action. Let the key point in the previous frame be U(x1,y1) and the same key point in the next frame be V(x2,y2). The displacement calculation formula is as follows:

[0082] W = VU = (x2 - x1, y2 - y1)

[0083] Determine if the action is standard; if not, use PID control to make the robot's servo motor perform the standard action.

[0084] Based on the formulas for angle, distance, and displacement mentioned above, real-time data of the user is calculated. This real-time data is then compared with pre-set standard movement data for fitness or rehabilitation exercises. Since it is difficult for the user's movements to perfectly match the standard movements, a reasonable threshold needs to be set. If the difference is less than the threshold, the movement is considered qualified; otherwise, a deviation is displayed, and the humanoid robot demonstrates the standard movement, calling up the pre-set movement group and using PID control of the robot's servo motor to perform the standard movement.

[0085] Based on the above-described preferred embodiments of the present invention, and through the foregoing description, those skilled in the art can make various changes and modifications without departing from the inventive concept. The technical scope of this invention is not limited to the contents of the specification, but must be determined according to the scope of the claims.

Claims

1. A method for humanoid robot-assisted fitness involving the detection of key human body points, characterized in that, Includes the following steps: Step 1: Collect real-time motion video of the person; Step 2: Using the MobileNetv3 network as the backbone, a YOLOv7 network model is adopted. The Ghost Conv module is used to improve the Conv module of the YOLOv7 network model to obtain an improved YOLOv7 network model. Target detection is then performed on the motion video to obtain the human body position. Target detection only exists in the initial frames before the human body appears. Step 3: Use the Centriole Tracking algorithm to track the human target and calculate the coordinates of the human's centroid; subsequent human tracking is completed using the centroid tracking algorithm. Step 4: Improve the Openpose network to perform real-time human body detection and obtain the location and number of key points on the human body; The improved Openpose network adopts the VGG network architecture, and the 7×7 convolution in the Openpose network is modified into three cascaded 3×3 depthwise separable convolution kernels, with a skip connection consisting of a 1×1 convolution kernel added between every three convolution kernels. Step 5: Calculate the angles, distances, and displacements between key points on the human body to determine if the movement is standard.

2. The method for humanoid robot-assisted fitness based on human key point detection according to claim 1, characterized in that, The GhostConv operation process includes: First, some features are obtained through ordinary convolution; Output of ordinary convolution for: （1） Where b is the bias constant, X Input video frame data, f For convolution kernel; Secondly, some features are subjected to inexpensive convolutional linear exchange to obtain the ghost feature map; This can be expressed by the formula: in, f’ for f Differentiating, , For convolution operations, m The number of channels for performing a regular convolution. To perform linearly transforming convolution operations, To Y Perform a linear transformation convolution operation.

3. The method for humanoid robot-assisted fitness based on human key point detection according to claim 1, characterized in that, Key points on the human body include: left eye, right eye, nose, neck, left shoulder, right shoulder, left elbow, right elbow, left wrist, right wrist, left hip, right hip, left knee, right knee, left ankle, and right ankle.

4. The method for humanoid robot-assisted fitness based on human key point detection according to claim 1, characterized in that, The formula for calculating the angle is: in,( , )and( , ) represents the coordinates of two adjacent key points.

5. The method for humanoid robot-assisted fitness based on human key point detection according to claim 1, characterized in that, The formula for distance is: in,( , )and( , ) represents the coordinates of two adjacent key points.

6. The method for humanoid robot-assisted fitness based on human body key point detection according to claim 1, characterized in that, The formula for displacement is: in,( , )and( , ) represents the coordinates of two adjacent key points.

Images