A pedestrian foot recognition method applied to a walking rehabilitation robot

Abstract

The application discloses a kind of pedestrian foot recognition methods applied to walking rehabilitation robot, when the low-cost RGB-D camera of walking rehabilitation robot only gathers the foot posture of lower half of body, the present application only utilizes the feature of pedestrian foot to carry out foot recognition and pose analysis, designs a kind of foot recognition method using based on ellipse fitting, increases the constraint of the ratio of major axis and minor axis of ellipse, area and color similarity, and designs a kind of left and right foot classification method.Compared with the calibrated pose, calculate error, effectively judge and identify to meet engineering needs.The pedestrian foot recognition method applied to walking rehabilitation robot proposed in the application can be applied to image recognition of pedestrian foot of walking rehabilitation robot, and improves the real-time and accuracy of image recognition.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of image recognition of walking rehabilitation robots, and particularly relates to a walking person foot recognition method applied to a walking rehabilitation robot. BACKGROUND

[0002] In recent years, robot technology has been widely applied in the field of life and health, especially intelligent walking aids, such as providing auxiliary support, navigation and fall protection services for the elderly. Research and development of walking rehabilitation robots suitable for current production practice not only can alleviate the vacancy of nursing personnel to a great extent, but also have broad market application prospects.

[0003] For walking rehabilitation robots, real-time and accurate target recognition and extraction is a key technology of the robot vision system. At present, robots can detect human lower limbs through visual sensors, laser sensors and multi-sensor. Kinect is a depth camera developed by Microsoft Corporation, which has powerful gesture recognition, gesture recognition and other functions, can detect corner and edge information, and the price is also moderate. Laser sensors can detect obstacles for collision avoidance and can also detect the distance between the user's legs and the robot, and record the movement of the user's legs, such as leg detection method based on neighborhood distance support vector domain description (NDSVDD), but the sensor price is high. Therefore, in actual research, visual sensors have obvious advantages, on the one hand, they can obtain corner and edge information in three-dimensional space, on the other hand, cameras are also widely concerned because of their better sampling frequency and low price.

[0004] In the past few decades, there have been two main methods for detecting the lower limbs of walking rehabilitation robots. One is the laser-based leg scanning detection method, and the other is to identify and estimate the key points of the human body. The method of identifying and estimating the key points of the human body, such as OpenPose, is a human key point detection source project developed by Carnegie Mellon University. It uses a caffe deep learning framework based on convolutional neural networks and supervised learning, and through a bottom-up detection method, it identifies and estimates the key points of the human body, facial key points, facial key points, facial key points, facial key points, and hand key points. It has strong robustness and is the world's first real-time multi-person two-dimensional pose estimation algorithm based on deep learning, but in actual application, if only the foot key points of the lower limbs appear in the field of view, similar openpose algorithms cannot identify the human foot.

[0005] Therefore, the application considers the traditional target detection algorithm least square ellipse fitting algorithm for the foot recognition of the walking rehabilitation robot user, and the image edge is fitted. Due to the specificity of the ellipse, the ellipse fitting algorithm can accurately give the result in any noise or occlusion, and finally realize the foot recognition under different scenes and light. SUMMARY

[0006] In view of the fact that the low-cost RGB-D camera of the walking rehabilitation robot only collects the foot posture of the lower body of a human being, the human body skeleton point method based on openpose cannot effectively recognize the foot of a pedestrian, in order to overcome the shortcomings of the existing method, the application proposes a pedestrian foot recognition method of a walking rehabilitation robot based on a low-cost RGB-D camera. In the high and low threshold design of the canny edge detection operator, the improved adaptive gradient threshold is used instead of the manual preset requiring prior knowledge, and the detected edge is obtained. The area and long and short axis constraints are increased, two fitting ellipses with the most similar colors are screened out as two feet, a left and right foot classification method is designed, and finally the foot recognition under different scenes and light is realized, so that the foot posture accuracy and real-time performance are improved. The effectiveness of the method is proved by experiments, and the experiments show that the method has high accuracy and real-time performance.

[0007] The technical scheme of the application is as follows:

[0008] A pedestrian foot recognition method applied to a walking rehabilitation robot, comprising the following steps:

[0009] Step 1) image Gaussian filtering:

[0010] Firstly, a kinectV1 camera on the walking rehabilitation robot is used to take pictures in real time to obtain an image matrix, and a filter is used to remove image noise, that is, the filter is convolved with the image.

[0011] Step 2) calculating pixel gradient intensity and direction:

[0012] The first order derivatives of the horizontal and vertical directions of each pixel are calculated, and then the gradient size G and direction θ of each pixel are calculated through the following two formulas.

[0013]

[0014]

[0015] Where G x The horizontal gradient G y is the vertical gradient, and the calculation process is as follows: assuming that the image window A has a center pixel e, A is convolved with the horizontal and vertical templates of Sobel respectively, and then the horizontal gradient G xand vertical gradient G y For

[0016]

[0017]

[0018] Step 3) NMS non-maximum suppression:

[0019] Find the local gradient maximum value of the pixel along the pixel gradient direction, set the pixel gray value corresponding to the local non-maximum value to 0, remove the false edge, and refine and accurately locate the image edge.

[0020] Step 4) Adaptive gradient double threshold selection:

[0021] Suppose a gradient threshold is selected in the range (0, 360) as threshold, the number of pixels less than the gradient threshold threshold is n0, and the ratio of the total number of pixels in the picture is w0, which is expressed as follows:

[0022]

[0023] Where a is the width of the picture, and b is the height of the picture.

[0024] The number of pixels greater than the gradient threshold threshold is n1, and the ratio of the total number of pixels in the picture is w1, which is expressed as follows:

[0025]

[0026] The average gray value of all pixels less than the gradient threshold threshold is u0; the average gray value of all pixels greater than the gradient threshold threshold is u1; and the total average gradient u is expressed as follows:

[0027] u = w0*u0 + w1*u1 (1.7)

[0028] The inter-class variance sigma is expressed as follows:

[0029] Sigma = w0*(u0-u)*(u0-u) + w1*(u1-u)*(u1-u) (1.8)

[0030] When the inter-class variance is maximum, the corresponding gradient threshold threshold is the best gradient threshold, that is, the high threshold maxVal, and the low threshold (minVal) is set to

[0031] Step 5) Apply double threshold to determine edge

[0032] Step four gets two thresholds: maxVal and minVal, and filters the local maximum pixel points x generated in non-maximum suppression, to achieve the purpose of noise reduction and keeping edge continuity.

[0033] ① If the gradient amplitude of the current edge pixel is greater than or equal to the high threshold, the current edge pixel is marked as a strong edge (to be retained).

[0034] ② If the gradient amplitude of the current edge pixel is between the high threshold and the low threshold, the current edge pixel is marked as a virtual edge (to be retained).

[0035] Generally, for a virtual edge:

[0036] If connected with a strong edge, the edge is processed as an edge.

[0037] If not connected with a strong edge, the edge is a weak edge, which is suppressed

[0038] ③ If the gradient amplitude of the current edge pixel is less than or equal to the low threshold, the current edge pixel is suppressed (to be discarded).

[0039] Step 6) Perform dilation operation on the picture edge to get clearer contour and draw the two groups of contours with the most boundary points.

[0040] Step 7) Least square ellipse fitting algorithm is used to fit the edge:

[0041] The ellipse equation is x 2 +axy+by 2 +cx+dy+e=0 (1.9) For n data points, (1.9) can be rewritten in matrix form

[0042] WX=Y (1.10)

[0043] Where X=[a b c d e] T

[0044] And rank(W)=5

[0045] We get

[0046] X=(W T W) -1 W T Y (1.11)

[0047] Thus the coefficients a, b, c, d, e in equation (1.9) are solved.

[0048] Step 8) screen the ellipse within a certain area threshold and the ratio of the major and minor axes of the ellipse within a certain threshold after fitting.

[0049] Step 9) if the number of ellipses meeting the conditions is greater than 2, screen the two ellipses with the closest color within a certain threshold, and identify as feet.

[0050] Step 10) draw two fitted ellipses, the center of the ellipse is the midpoint of the feet, and the end point of the major axis of the ellipse is the toe or heel. The coordinates of the midpoint of the two feet in the pixel coordinate system are (x1 y1), (x2 y2); if x1 >= x2, (x1 y1) is the left foot midpoint coordinate; otherwise (x1 y1) is the right foot midpoint coordinate.

[0051] The beneficial effects of the present application are as follows:

[0052] 1) The present application is aimed at the low-cost RGB-D camera of the walking rehabilitation robot, which can only collect the foot posture of the lower body of the human body, and the openpose-based human skeleton point method cannot effectively identify the pedestrian feet. Therefore, a pedestrian foot recognition method applied to a walking rehabilitation robot is designed based on a low-cost RGB-D camera.

[0053] 2) The adaptive gradient threshold method is used in the high and low threshold design of the canny edge detection operator, which has high-quality real-time detection of different feet in different environments. In addition, due to the specificity of the ellipse, the ellipse fitting algorithm can accurately give the result in any noise or occlusion, making the foot image recognition result more accurate.

[0054] 3) The present application increases the area and major and minor axis constraints, and screens out the two ellipses with the most similar color as the feet, and designs a left and right foot classification method. Compared with the calibrated pose, the error is calculated, which meets the engineering needs. BRIEF DESCRIPTION OF DRAWINGS

[0055] Fig. 1 The present application is based on the selection of adaptive gradient threshold value;

[0056] Fig. 2 The present application is a pedestrian foot recognition flowchart applied to a walking rehabilitation robot. DETAILED DESCRIPTION

[0057] The present application will be further described in detail below with reference to the accompanying drawings.

[0058] Reference Figs. 1-2 A pedestrian foot recognition method applied to a walking rehabilitation robot, the algorithm implementation process of the present patent is as follows:

[0059] Step 1) Image Gaussian filtering: Firstly, the kinectV1 camera reads the picture in real time to get the image matrix, and the image is filtered using Gaussian filtering.

[0060] Step 2) Calculate the pixel gradient strength and direction: The pixel gradient strength and direction are calculated as shown in formulas (1.1) and (1.2).

[0061] Step 3) NMS non-maximum suppression: NMS non-maximum suppression is used to remove false edges, refine and accurately locate the image edges.

[0062] Step 4) Adaptive gradient double threshold selection:

[0063] Calculate the number of pixels below a certain gradient threshold threshold∈(0,360) n0 and the ratio of the picture pixels w0 as shown in formula (1.5); calculate the number of pixels above the gradient threshold n1 and the ratio of the picture pixels w1 as shown in formula (1.6); calculate the average gray of all pixels below the threshold u0; calculate the average gray of the pixels above the threshold u1, calculate the total average gradient u as shown in formula (1.7); get the inter-class variance sigma as shown in formula (1.8), calculate the maximum inter-class variance. The threshold corresponding to the maximum inter-class variance is the best gradient threshold. The best gradient threshold threshold is the high threshold maxVal, and the low threshold is set to

[0064] Step 5) Apply double threshold to determine edges:

[0065] Step 4 gets two thresholds: high threshold (maxVal) and low threshold (minVal), and filters the local maximum pixel points x generated in the non-maximum suppression to achieve the purpose of noise reduction and edge continuity.

[0066] ① If the gradient amplitude of the current edge pixel is greater than or equal to the high threshold, mark the current edge pixel as a strong edge (need to be retained).

[0067] ② If the gradient amplitude of the current edge pixel is between the high threshold and the low threshold, mark the current edge pixel as a virtual edge (need to be retained).

[0068] Generally, for a virtual edge:

[0069] If connected with a strong edge, the edge is processed as an edge.

[0070] If there is no connection with a strong edge, the edge is a weak edge, which is suppressed

[0071] ③ If the gradient amplitude of the current edge pixel is less than or equal to the low threshold, suppress the current edge pixel (need to be discarded).

[0072] Step 6) Perform an inflation operation on the picture edge to obtain a clearer contour and use the cv2.findContours function in opencv to draw the boundary points, and only draw the two groups of contours with the most boundary points.

[0073] Step 7) Use the cv2.fitEllipse function in opencv to fit the two groups of contour points on the picture into two ellipses, so that as many sample points of the contours as possible are on the ellipses.

[0074] Step 8) Screen the ellipses within a certain area threshold after fitting and the ratio of the major and minor axes of the ellipses within a certain threshold.

[0075] Step 9) If the number of ellipses meeting the conditions is greater than 2, screen the two ellipses with the closest color within a certain threshold to identify as feet.

[0076] Step 10) Draw the two fitted ellipses, and the ellipse center is the midpoint of the feet, and the ellipse major axis end point is the toe or heel. Obtain the coordinates of the midpoint of the two feet in the pixel coordinate system as (x1 y1), (x2 y2); if x1≥x2, then (x1 y1) is the left foot midpoint coordinate; otherwise (x1 y1) is the right foot midpoint coordinate.

[0077] The above embodiment only represents one embodiment of the present application and cannot be understood as limiting the scope of the present application. It should be noted that those skilled in the art can make several deformation designs without departing from the concept of the present application, which are all within the protection scope of the present application.

Claims

1. A pedestrian foot recognition method applied to a walking rehabilitation robot, characterized by, The method comprises the following steps: Step 1) image Gaussian filtering: using a kinectV1 camera to take pictures in real time to obtain an image matrix, and using a filter to remove image noise, that is, using a filter to convolve with the image; Step 2) calculating pixel gradient intensity and direction: calculating the first-order derivative of each pixel of the image in the horizontal and vertical directions, and then calculating the gradient size G and direction of each pixel; Step 3) NMS non-maximum suppression: finding the local gradient maximum value of the pixel point along the pixel gradient direction, setting the gray value of the pixel point corresponding to the local non-maximum value to 0, removing false edges, and refining and accurately positioning the image edge; Step 4) adaptive gradient double-threshold selection: In the range of (0, 360), a gradient threshold is selected as threshold, the number of pixels less than the gradient threshold threshold is n0, and the ratio of the total number of pixels is w0, which is represented as follows: Where a is the width of the picture and b is the height of the picture; The number of pixels greater than the gradient threshold threshold is n1, and the ratio of the total number of pixels is w1, which is represented as follows: The average gray value of all pixels less than the gradient threshold threshold is u0; the average gray value of all pixels greater than the gradient threshold threshold is u1; and the total average gradient u is represented as follows: u = w0*u0 + w1*u1 (1.7) The inter-class variance sigma is represented as follows: Sigma = w0*(u0-u)*(u0-u) + w1*(u1-u)*(u1-u) (1.8) When the inter-class variance is maximum, the corresponding gradient threshold threshold is the optimal gradient threshold, that is, the high threshold maxVal, and the low threshold minVal is set as Step 5) applying double thresholds to determine edges: Two thresholds, a high threshold maxVal and a low threshold minVal, are obtained through step 4), and the local maximum value pixel x generated in the non-maximum suppression is filtered to achieve the purpose of noise reduction and edge continuity; Step 6) performing an inflation operation on the picture edge and drawing all group contours; Step 7) fitting the edge using a least square ellipse fitting algorithm; Step 8) screening the ellipse within the set area threshold after fitting and the ellipse within the set threshold ratio of major and minor axes; Step 9) if the number of ellipses meeting the conditions is greater than 2, screening the two ellipses with the closest color within the set threshold, and identifying them as feet; Step 10) drawing the two fitted ellipses, and the ellipse center is the midpoint of the feet, and the ellipse major axis end point is the toe or heel.

2. The pedestrian foot recognition method for a walking rehabilitation robot according to claim 1, wherein The specific process of step 2) is as follows: where G x horizontal gradient, G y vertical gradient, the calculation process is shown as follows: set the image window A, the center pixel is e, and A is respectively convolved with the horizontal and vertical templates of Sobel, so as to obtain the horizontal gradient G x and the vertical gradient G y of the center pixel point e: 3.The pedestrian foot recognition method applied to a walking rehabilitation robot according to claim 1, wherein, The specific process of step 5) is as follows: a. If the gradient amplitude of the current edge pixel is greater than or equal to the high threshold, the current edge pixel is marked as a strong edge; b. If the gradient amplitude of the current edge pixel is between the high threshold and the low threshold, the current edge pixel is marked as a virtual edge; For a virtual edge: b1. If connected with a strong edge, the edge is processed as an edge; b2. If not connected with a strong edge, the edge is a weak edge and is suppressed; c. If the gradient amplitude of the current edge pixel is less than or equal to the low threshold, the current edge pixel is suppressed.

4. The pedestrian foot recognition method for a walking rehabilitation robot according to claim 1, wherein The specific process of step 7) is as follows: The ellipse equation is x 2 + a x y + b y 2 + c x + d y + e = 0 (1.9) For n data points, rewrite (1.9) in matrix form: WX = Y (1.10) Where: X = [a b c d e] T And rank(W) = 5, we have: WX = Y (1.10) X = (W T W) -1 W T Y (1.11) Thus, the coefficients a, b, c, d, e in formula (1.9) are solved.

5. The pedestrian foot recognition method for a walking rehabilitation robot according to claim 1, wherein The specific process of the step 10) is as follows: The coordinates of the midpoint of the two feet in the pixel coordinate system are (x1 y1) and (x2 y2). If x1 is greater than x2, (x1 y1) is the coordinate of the left foot midpoint; otherwise, (x1 y1) is the coordinate of the right foot midpoint.

Images