A method for a robot to recognize and locate a human mouth

Abstract

The present invention belongs to the field of automation control technology, and relates to a method for a robot to identify and locate a human mouth. The present invention locates key feature points of a face through depth color image recognition, and then calculates and establishes a human mouth coordinate system, and then calculates the optimal observation position of the camera relative to the face for active perception, and finally accurately locates the spatial position of the oral cavity. The specific steps are: obtaining a depth color image of the camera; inputting the color image into an integrated regression tree to calculate the coordinates of the facial feature points; coupling the color image with the depth image, and performing coordinate transformation to obtain the coordinates of the feature points in the camera system; extracting key feature points to calculate and establish a human mouth coordinate system; and moving the robotic arm according to the optimal observation position of the human mouth coordinates. This method improves the positioning accuracy of the depth camera, and provides a solid foundation for the robot to perform position judgment and data collection based on the depth camera.

Description

Technical Field

[0001] The present invention belongs to the field of automation control technology and relates to a method for a robot to identify and locate a human mouth, and in particular to an active perception method for a robot to identify and locate a human mouth based on a depth camera. Background Art

[0002] The traditional method of using depth cameras to locate faces often focuses on improving the accuracy of data processing methods, but ignores the fact that the accuracy and precision of its measurements are often affected by environmental factors such as the position and angle of the camera relative to the face. In actual operation, this error is often not quantifiable, resulting in increased risks in the actual measurement process. For example, patent CN108764089B "Method for identifying the midline of the human back spine based on a depth camera" uses cubic spline interpolation to obtain curvature coordinates; but the accuracy of data point acquisition depends on the observation angle and position. In many scenarios, especially application scenarios with very high requirements for measurement accuracy and precision, the risks caused by this error are unacceptable. In addition to using traditional image recognition methods to improve accuracy, the present application can also autonomously move to the optimal observation position, improving the effectiveness of the depth camera from both the data itself and the processing method. Summary of the Invention

[0003] The purpose of this invention is to propose a robot-based method for identifying and locating the human mouth, so as to overcome the shortcomings of traditional technologies. In various application scenarios with high precision requirements, it can autonomously find the optimal shooting position to minimize the error in face recognition, and effectively improve the accuracy of the camera's active perception and identification positioning.

[0004] The robot's method for identifying and locating a human mouth proposed in the present invention first obtains a depth color image from a camera, inputs the color image into an integrated regression tree, calculates the coordinates of facial feature points, couples the color image with the depth image, obtains the coordinates of the feature points in the camera system through coordinate transformation, extracts key feature points, calculates and establishes a human mouth coordinate system, and moves the robotic arm according to the optimal observation position of the human mouth coordinates.

[0005] The advantages of the robot's method for identifying and locating a human mouth proposed in the present invention are:

[0006] The robot's mouth recognition and positioning method uses depth color image recognition to locate key facial features, then calculates and establishes a mouth coordinate system. This method then actively senses the optimal camera position relative to the face, ultimately accurately locating the oral cavity's spatial position. This method effectively addresses the accuracy issues associated with facial recognition and positioning, effectively reducing depth camera measurement errors and laying a solid foundation for medical robot system construction, data acquisition, and experimental analysis. BRIEF DESCRIPTION OF THE DRAWINGS

[0007] Figure 1 This is a flowchart of the method for robot to identify and locate human mouth proposed by the present invention.

[0008] Figure 2 It is a schematic diagram of the coordinate transformation from the depth color image to the camera coordinate system. DETAILED DESCRIPTION

[0009] The robot's method for identifying and locating a human mouth proposed in the present invention first obtains a depth color image from a camera, inputs the color image into an integrated regression tree, calculates the coordinates of facial feature points, couples the color image with the depth image, obtains the coordinates of the feature points in the camera system through coordinate transformation, extracts key feature points, calculates and establishes a human mouth coordinate system, and moves the robotic arm according to the optimal observation position of the human mouth coordinates.

[0010] The flowchart of the above identification and positioning method is as follows: Figure 1 As shown, the specific steps include:

[0011] (1) Use a depth camera to obtain a depth color image of the face (abbreviated as RGBD image), where the depth coordinate of the depth color image is z;

[0012] (2) Using the gradient boosting tree algorithm, an integrated regression tree is learned;

[0013] (3) Input the color image of step (1) into the integrated regression tree of step (2) and calculate the two-dimensional coordinates (x v ,y v );

[0014] (4) The two-dimensional coordinates (x v ,y v ) and the corresponding depth coordinate z of step (1) is converted into the three-dimensional coordinate (x, y, z) in the depth camera coordinate system, where is the focal length of the depth camera (in pixels), h is the pixel height of the color image, and θ is the field of view of the depth camera, as shown in Figure 2 As shown;

[0015] (5) Select the facial feature points F1, F2, F3, and F4 from the facial feature points of step (3), and then obtain the line F1F2 between the left corner of the mouth feature point F1 and the right corner of the mouth feature point F2, and the line F3F4 between the upper lip midpoint F3 and the lower lip midpoint F4. The common perpendicular line between the lines F1F2 and F3F4 is used as the z-axis on the mouth coordinate system, and the direction vector of the z-axis is the unit vector of the cross product of the above lines. The position of the intersection of the vertical line passing through the left and right corners of the mouth in the depth camera coordinate system is in, is the coordinate of the left corner of the mouth, coefficient is the coordinate of the midpoint of the upper lip in the camera coordinate system, is the vector of the line connecting the left and right corners of the mouth in the camera coordinate system, and the intersection is denoted as O, which is the origin of the mouth coordinate system. The line F1F2 is the x-axis of the mouth coordinate system, and the direction vector of the x-axis is The direction of the cross product of the line F1F2 and the z-axis is obtained according to the right-hand principle to obtain the y-axis on the human mouth coordinate system. The direction vector of the y-axis is Get the three-dimensional coordinate system of the human mouth

[0016] (6) Using the following formula, the position of the origin of the human mouth coordinate system in step (5) in the depth camera coordinate system is Calculate the current position coordinates of the depth camera relative to the human mouth coordinate system (x c ,y c ,z c ):

[0017]

[0018] (7) According to the optimal position of the depth camera to observe the human face, a coordinate difference threshold is set, and the optimal position coordinates are compared with the current position coordinates of step (6). If the difference between the optimal position coordinates and the current position coordinates is greater than the coordinate difference threshold, the robot arm moves to the optimal position according to the difference, and repeats steps (1) to (6). If the difference between the optimal position coordinates and the current position coordinates is less than or equal to the coordinate difference threshold, the difference between the two position coordinates is less than the threshold, then the depth camera reaches the optimal observation position, and the robot completes the recognition and positioning of the human mouth.

Claims

1. A method for robot to identify and locate human mouth, characterized in that: The method first obtains the depth color image of the camera, inputs the color image into the integrated regression tree, calculates the coordinates of the facial feature points, couples the color image with the depth image, obtains the coordinates of the feature points in the camera system through coordinate transformation, extracts the key feature points, calculates and establishes the human mouth coordinate system, and moves the robotic arm according to the optimal observation position of the human mouth coordinates; The method specifically comprises the following steps: (1) Use a depth camera to obtain a depth color image of the face, where the depth coordinate of the depth color image is z; (2) Using the gradient boosting tree algorithm, an integrated regression tree is learned; (3) Input the color image of step (1) into the integrated regression tree of step (2) and calculate the two-dimensional coordinates (x v ,y v ); (4) The two-dimensional coordinates (x v ,y v ) and the corresponding depth coordinate z of step (1) is converted into the three-dimensional coordinate (x, y, z) in the depth camera coordinate system, where is the focal length of the depth camera, h is the pixel height of the color image, and θ is the field of view angle of the depth camera; (5) Select the facial feature points F1, F2, F3, and F4 from the facial feature points of step (3), and then obtain the line F1F2 between the left corner of the mouth feature point F1 and the right corner of the mouth feature point F2, and the line F3F4 between the upper lip midpoint F3 and the lower lip midpoint F4. The common perpendicular line between the lines F1F2 and F3F4 is used as the z-axis on the mouth coordinate system, and the direction vector of the z-axis is the unit vector of the cross product of the above lines. The position of the intersection of the vertical line passing through the left and right corners of the mouth in the depth camera coordinate system is in, is the coordinate of the left corner of the mouth, coefficient is the coordinate of the midpoint of the upper lip in the camera coordinate system, is the vector of the line connecting the left and right corners of the mouth in the camera coordinate system, and the intersection is denoted as O, which is the origin of the mouth coordinate system. The line F1F2 is the x-axis of the mouth coordinate system, and the direction vector of the x-axis is The direction of the cross product of the line F1F2 and the z-axis is obtained according to the right-hand principle to obtain the y-axis on the human mouth coordinate system. The direction vector of the y-axis is Get the three-dimensional coordinate system of the human mouth (6) Using the following formula, the position of the origin of the human mouth coordinate system in step (5) in the depth camera coordinate system is Calculate the current position coordinates of the depth camera relative to the human mouth coordinate system (x c ,y c ,z c ): (7) According to the optimal position of the depth camera to observe the human face, a coordinate difference threshold is set, and the optimal position coordinates are compared with the current position coordinates of step (6). If the difference between the optimal position coordinates and the current position coordinates is greater than the coordinate difference threshold, the robot arm moves to the optimal position according to the difference, and repeats steps (1) to (6). If the difference between the optimal position coordinates and the current position coordinates is less than or equal to the coordinate difference threshold, the difference between the two position coordinates is less than the threshold, then the depth camera reaches the optimal observation position, and the robot completes the recognition and positioning of the human mouth.

Images