A gesture recognition method, apparatus, device and medium

Abstract

The application discloses a gesture recognition method, device, equipment and medium. The application obtains the coordinates of the skeleton points at the elbows and wrists in a to-be-recognized image, predicts a maximum hand prediction box and a minimum hand prediction box according to the coordinates of the skeleton points, and the gloves worn by a user have multiple colors. According to a preset seed point, an image is segmented by using a color-based region growing algorithm to obtain each segmentation candidate region, the region edge of which is located between the minimum hand prediction box and the maximum hand prediction box. Each segmentation candidate region is a contour region of one color of the glove. The hand contour region is determined by using a convex hull detection method. The region image of the hand contour region is filled with a preset pixel value to remove the influence of the glove color on gesture recognition. The region image filled with the skin color is input into a pre-stored gesture recognition model to obtain an output gesture recognition result, so that gestures can be universally recognized when the user wears different gloves.

Description

[0001] This application claims priority to the Chinese Patent Application No. 202211715130.4, filed on December 29, 2022, and entitled "Gesture Recognition Method, Device, Equipment and Medium", the entire content of which is incorporated herein by reference. TECHNICAL FIELD

[0002] The present application relates to the technical field of gesture recognition, and in particular to a gesture recognition method, device, equipment and medium. BACKGROUND

[0003] In winter, high-voltage work environment, etc., workers need to wear gloves to work, so how to realize the gesture recognition of wearing gloves has become a technical problem to be solved.

[0004] The existing gesture recognition method is to collect the gesture action of the construction personnel wearing gloves as a sample in real time, and make a gesture action video sequence; the gesture action features in the video sequence are extracted through a graph convolutional neural network; the gesture action classification model is trained using the gesture action features, and a trained gesture action classification model is obtained; the trained gesture action classification model is used to judge and recognize the gesture action of the construction personnel in real time.

[0005] However, the technical solution of the prior art can only successfully recognize the gesture when wearing gloves of a specific style, and when wearing gloves of other styles, the graph convolutional neural network cannot recognize the hand region and extract the gesture action features, resulting in failure to successfully recognize the gesture. Therefore, the gesture recognition method of the prior art has poor universality. SUMMARY

[0006] The present application provides a gesture recognition method, device, equipment and medium to solve the problem of poor universality of the gesture recognition method of the prior art.

[0007] In a first aspect, the present application provides a gesture recognition method, comprising:

[0008] obtaining the coordinates of the elbow and wrist points in the to-be-recognized image;

[0009] According to the coordinates of the skeleton points, the maximum hand prediction box and the minimum hand prediction box are predicted; a region growing algorithm based on color is used to perform image segmentation in the to-be-recognized image according to a preset seed point, to obtain each segmentation candidate region whose region edge is located between the minimum hand prediction box and the maximum hand prediction box;

[0010] According to the pixel set of each segmentation candidate region, a hand contour region is detected by using a convex hull detection method, a region image of the hand contour region in the image to be recognized is filled with a preset pixel value, and the region image filled with the skin color is input into a pre-stored gesture recognition model to obtain an output gesture recognition result.

[0011] Further, the maximum hand prediction box and the minimum hand prediction box are predicted according to the coordinates of the skeleton points, and the maximum hand prediction box and the minimum hand prediction box are predicted according to the coordinates of the skeleton points.

[0012] According to the coordinates of the skeleton points, a small arm length between the elbow and the wrist is determined, a first side length of the maximum hand prediction box is determined according to the small arm length, and a second side length of the minimum hand prediction box is determined according to a ratio of the small arm length to a preset value.

[0013] According to the coordinates of the skeleton points, the small arm length, and an angle value determination function, a normal line perpendicular to a plane where a camera is located and an angle value of an included angle between the normal line and an extension line of the elbow to the wrist are determined, and according to a sine function value of the angle value, the first side length, and the second side length, a half value of a first product value of the sine function value and the first side length and a half value of a second product value of the sine function value and the second side length are determined.

[0014] According to the extension line of the elbow to the wrist, a square region with a center located on the extension line and a distance from the center to the wrist being the half value of the first product value and a side length being the first side length is determined as the maximum hand prediction box, and a square region with a center located on the extension line and a distance from the center to the wrist being the half value of the second product value and a side length being the second side length is determined as the minimum hand prediction box.

[0015] Further, the image segmentation in the image to be recognized is performed according to a preset seed point by using a color-based region growing algorithm to obtain each segmentation candidate region with a region edge located between the minimum hand prediction box and the maximum hand prediction box.

[0016] According to a pixel value of a preset seed point and a pixel value of each first pixel point in a preset neighborhood range of the preset seed point, a target first pixel point with a color similarity greater than a preset color similarity threshold value is determined in each first pixel point, the preset seed point and the target first pixel point are saved in a pixel set, the target first pixel point is taken as a target seed point for the next round of region growing, and until no target seed point is determined, the segmentation region composed of the pixel points in the pixel set is determined to stop growing.

[0017] determining whether the segmentation region reaches the edge of the minimum hand prediction frame according to the segmentation region and the minimum hand prediction frame; if the segmentation region does not reach the edge of the minimum hand prediction frame, determining an adjacent point of an intersection point of a line extending from the elbow to the wrist and the segmentation region, and taking the adjacent point as a target seed point for region growing in the next round until the segmentation region stops growing;

[0018] if the segmentation region reaches the edge of the minimum hand prediction frame, determining the segmentation region as a segmentation candidate region, determining each second pixel point in a preset neighborhood range of the target seed point of the last round of region growing according to the target seed point of the last round of region growing of the segmentation region, taking a target second pixel point not located in the segmentation candidate region as an updated preset seed point for next segmentation candidate region growing, and performing region growing based on the updated preset seed point to obtain a next segmentation candidate region until the region edge reaches the edge of the maximum hand prediction frame to obtain a last segmentation candidate region.

[0019] Further, after the segmentation region is determined as a segmentation candidate region, before the determination of each second pixel point in a preset neighborhood range of the target seed point of the last round of region growing according to the target seed point of the last round of region growing of the segmentation region, the method further comprises:

[0020] if the number of pixel points in the pixel point set of the segmentation candidate region is less than a preset number threshold, merging the pixel point set of the segmentation candidate region with the pixel point set of the adjacent previous segmentation candidate region to determine an updated segmentation candidate region; and / or

[0021] if the difference between the first average pixel value of the pixel points in the pixel point set of the segmentation candidate region and the second average pixel value of the pixel points in the pixel point set of the adjacent previous segmentation candidate region is less than a preset pixel value threshold, determining that the color difference between the segmentation candidate region and the adjacent previous segmentation candidate region is small, merging the pixel point set of the segmentation candidate region with the pixel point set of the adjacent previous segmentation candidate region to determine an updated segmentation candidate region.

[0022] Further, the determining of the hand contour region by the method of convex hull detection according to the pixel point set of each segmentation candidate region comprises:

[0023] performing convex hull detection on the pixel points in the pixel point set of each segmentation candidate region to detect each convex vertex and each convex hull defect point;

[0024] For each convex hull defect point, based on the convex hull defect point and the convex vertices on both sides, it is determined whether the convex hull defect depth of the convex hull defect point meets the first preset defect depth condition of clenching a fist or the second preset defect depth condition of extending the fingers. If the convex hull defect depth of each convex hull defect point meets the first preset defect depth condition or the second preset defect depth condition, the segmentation candidate region is determined as the hand contour region.

[0025] Further, determining whether the convex hull defect depth of the convex hull defect point satisfies the first preset defect depth condition for a clenched fist, based on the convex hull defect point and the convex vertices on both sides, includes:

[0026] If the first angle value of the first included angle formed by the convex hull defect point and the convex vertices on both sides is greater than the first preset angle value threshold, then the convex hull defect depth of the convex hull defect point is determined to be small.

[0027] Further, determining whether the convex hull defect depth of the convex hull defect point satisfies the second preset defect depth condition for finger extension based on the convex hull defect point and the adjacent two convex vertices includes:

[0028] The length of the first line connecting the convex hump defect point to the bone point of the wrist, and the length of the second line connecting the convex vertex of the adjacent side of the convex hump defect point to the bone point of the wrist are determined. If the minimum value of the second line length and the first line length is within a preset value range, and the second angle value of each second included angle formed by the bone point of the wrist, the convex hump defect point, and the convex vertex of each side is less than a second preset angle value threshold, and the ratio of the minimum to the maximum value of the first line length and the second line length is not greater than a preset value threshold, and the ratio of the maximum to the minimum value of the two line lengths connecting the convex hump defect point to the convex vertex of each side is not less than the preset value threshold, then the convex hump defect depth of the convex hump defect point satisfies the second preset defect depth condition for finger extension.

[0029] Secondly, this application provides a gesture recognition device, the device comprising:

[0030] The acquisition module is used to obtain the coordinates of the skeletal points at the elbow and wrist in the image to be identified;

[0031] determining module, configured to: predict a maximum hand prediction box and a minimum hand prediction box according to coordinates of the skeleton points; perform image segmentation in the to-be-identified image by using a color-based region growing algorithm based on a preset seed point to obtain each segmentation candidate region whose region edge is located between the minimum hand prediction box and the maximum hand prediction box; and determine a hand contour region detected by using a convex hull detection method according to a pixel point set of each segmentation candidate region;

[0032] The recognition module is configured to perform skin color filling on a region image of the hand contour region in the to-be-identified image by using a preset pixel value, and input the region image after the skin color filling into a pre-stored gesture recognition model to obtain an output gesture recognition result.

[0033] Further, the determining module is specifically configured to: determine a small arm length between the elbow and the wrist according to the coordinates of the skeleton points; determine a first side length of the maximum hand prediction box according to the small arm length; determine a second side length of the minimum hand prediction box according to a ratio of the small arm length to a preset value; determine a function according to the coordinates of the skeleton points, the small arm length, and an angle value; determine a normal line perpendicular to a plane where the camera is located and the angle value of an included angle between the normal line and an extension line of the elbow to the wrist; determine a first product value of the sine function value of the angle value and the first side length and a second product value of the sine function value of the angle value and the second side length; and determine that a square region with a center located on the extension line of the elbow to the wrist, a distance from the center to the wrist being a half of the first product value, and a side length being the first side length is the maximum hand prediction box, and a square region with a center located on the extension line of the elbow to the wrist, a distance from the center to the wrist being a half of the second product value, and a side length being the second side length is the minimum hand prediction box.

[0034] Further, the determining module is specifically configured to: determine a target first pixel point in each first pixel point in a preset neighborhood range of the preset seed point and having a color similarity greater than a preset color similarity threshold value to the preset seed point according to a pixel value of the preset seed point and a pixel value of each first pixel point in the preset neighborhood range of the preset seed point; save the preset seed point and the target first pixel point in a pixel point set; take the target first pixel point as a target seed point for a next round of region growing; and stop growing when it is determined that there is no target seed point.

[0035] determining whether the segmentation region reaches the edge of the minimum hand prediction frame; if the segmentation region does not reach the edge of the minimum hand prediction frame, determining an adjacent point of an intersection point of a line extending from the elbow to the wrist and the segmentation region, and taking the adjacent point as a target seed point for region growing in the next round until the segmentation region stops growing;

[0036] if the segmentation region reaches the edge of the minimum hand prediction frame, determining the segmentation region as a segmentation candidate region, determining each second pixel point in a preset neighborhood range of the target seed point of the last round of region growing according to the target seed point of the last round of region growing of the segmentation region, taking a target second pixel point not located in the segmentation candidate region as an updated preset seed point for next segmentation candidate region growing, and performing region growing based on the updated preset seed point to obtain a next segmentation candidate region until the region edge reaches the edge of the maximum hand prediction frame to obtain a last segmentation candidate region.

[0037] Further, the determining module is further configured to, after determining the segmentation region as a segmentation candidate region, before determining each second pixel point in a preset neighborhood range of the target seed point of the last round of region growing according to the target seed point of the last round of region growing of the segmentation region, if the number of pixel points in the pixel point set of the segmentation candidate region is less than a preset number threshold, merging the pixel point set of the segmentation candidate region with a pixel point set of a previous adjacent segmentation candidate region to determine an updated segmentation candidate region; and / or if a difference between a first average pixel value of the pixel points in the pixel point set of the segmentation candidate region and a second average pixel value of the pixel points in the pixel point set of the previous adjacent segmentation candidate region is less than a preset pixel value threshold, determining that the color difference between the segmentation candidate region and the previous adjacent segmentation candidate region is small, and merging the pixel point set of the segmentation candidate region with the pixel point set of the previous adjacent segmentation candidate region to determine the updated segmentation candidate region.

[0038] Further, the determining module is specifically configured to, for each segmentation candidate region, perform convex hull detection on the pixel points in the pixel point set of the segmentation candidate region to detect each convex vertex and each convex hull defect point; and for each convex hull defect point, determine whether a convex hull defect depth of the convex hull defect point satisfies a first preset defect depth condition of a fist clenching or a second preset defect depth condition of a finger stretching according to the convex hull defect point and the convex vertices on two adjacent sides, and if the convex hull defect depth of each convex hull defect point satisfies the first preset defect depth condition or the second preset defect depth condition, determine the segmentation candidate region as a hand contour region.

[0039] Further, the determining module is specifically configured to determine that the convex defect depth of the convex defect point satisfies a first preset defect depth condition of making a fist if a first angle value of a first included angle formed by the convex defect point and two adjacent convex vertices on two sides of the convex defect point is greater than a first preset angle value threshold.

[0040] Further, the determining module is specifically configured to determine the first line length of the convex defect point and a second line length of a wrist skeletal point, and the second line length is a line length of the wrist skeletal point and a convex vertex on any one side of the convex defect point, and determine that the convex defect depth of the convex defect point satisfies a second preset defect depth condition of stretching fingers if a minimum value of the first line length and the second line length is located in a preset numerical range, each second angle value of each second included angle formed by the wrist skeletal point and the convex defect point and the convex vertex on each side is less than a second preset angle value threshold, a ratio of a minimum value and a maximum value of the first line length and the second line length is not greater than a preset numerical threshold, and a ratio of a maximum value and a minimum value of two line lengths of the convex defect point and the convex vertex on each side is not less than the preset numerical threshold.

[0041] In a third aspect, the present application provides an electronic device, comprising: a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory complete mutual communication through the communication bus.

[0042] The memory stores a computer program, and when the program is executed by the processor, the processor implements the steps of any one of the above gesture recognition methods.

[0043] In a fourth aspect, the present application provides a computer readable storage medium, which stores a computer program, and when the computer program is executed by a processor, the steps of any one of the above gesture recognition methods are implemented.

[0044] The application provides a gesture recognition method, device, equipment and medium. In the application, the coordinates of the elbow and wrist bone points in a to-be-recognized image are acquired, the maximum hand prediction box and the minimum hand prediction box are predicted according to the coordinates of the bone points, the gloves worn by the user have multiple colors, each segmentation candidate region whose region edge is located between the minimum hand prediction box and the maximum hand prediction box is obtained by image segmentation through a color-based region growing algorithm according to a preset seed point, each segmentation candidate region is a contour region of one color of the glove, the hand contour region of the glove edge is determined through a convex hull detection method, which may be a fist state or an extended finger state, the region image of the hand contour region is filled with a preset pixel value to remove the influence of the glove color on gesture recognition, and the region image filled with the skin color is input into a pre-stored gesture recognition model to obtain an output gesture recognition result, so that gestures can be recognized universally when the user wears different gloves. BRIEF DESCRIPTION OF DRAWINGS

[0045] In order to more clearly illustrate the technical solutions in the application, the drawings needed in the embodiments will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the application, and other drawings can be obtained by those skilled in the art without creative labor.

[0046] Figure 1 A process schematic diagram of a gesture recognition method provided by the application;

[0047] Figure 2 A schematic diagram of a to-be-recognized image provided by the application;

[0048] Figure 3 A schematic diagram of a to-be-recognized image processed by a 3D human pose estimation algorithm provided by the application;

[0049] Figure 4 A hand schematic diagram provided by the application;

[0050] Figure 5 A schematic diagram of a hand prediction box provided by the application;

[0051] Figure 6 A schematic diagram of a hand contour region after convex hull detection provided by the application;

[0052] Figure 7 A schematic diagram of a region image of a hand contour region filled with skin color provided by the application;

[0053] Figure 8 A schematic diagram of a hand prediction box provided by the application;

[0054] Figure 9 A schematic diagram of a maximum hand prediction frame provided for an embodiment of the present application;

[0055] Figure 10 A schematic diagram of a segmentation region provided for an embodiment of the present application;

[0056] Figure 11 A schematic diagram of an adjacent point of an intersection point of an extension line of an elbow to a wrist and a segmentation region when a segmentation region does not reach an edge of a minimum hand prediction frame provided for the present application;

[0057] Figure 12 A schematic diagram of a hand contour region provided for the present application;

[0058] Figure 13 A schematic diagram of determining a hand contour region provided for the present application;

[0059] Figure 14 A structural schematic diagram of a gesture recognition device provided for the present application;

[0060] Figure 15 A structural schematic diagram of an electronic device provided for the present application. DETAILED DESCRIPTION

[0061] In order to make the purposes, technical solutions and advantages of the present application clearer, the present application will be further described in detail below with reference to the drawings. Obviously, the described embodiments are only a part of the embodiments of the present application, but not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by a person of ordinary skill in the art without creative work fall within the scope of protection of the present application.

[0062] In order to improve the universality of the gesture recognition method for wearing gloves, the present application provides a gesture recognition method, device, equipment and medium.

[0063] Figure 1 A process schematic diagram of a gesture recognition method provided for the present application, the process includes the following steps:

[0064] S101: Obtain the coordinates of the skeletal points at the elbow and wrist in the image to be recognized.

[0065] In order to improve the universality of the gesture recognition method for wearing gloves, the gesture recognition method disclosed by the present application is applied to an electronic device, wherein the electronic device can be a host, a tablet computer, a notebook computer, a smart phone and the like, or a server, wherein the server can be a local server or a cloud server, and the embodiments of the present application do not limit this.

[0066] The electronic device obtains a to-be-recognized image. Specifically, the to-be-recognized image can be collected by an image collection device of the electronic device itself, or can be received from another electronic device connected to the electronic device. The image collection device can be an RGB camera, and the to-be-recognized image is a color image. The image collection device can also be an infrared camera, and the to-be-recognized image is a black-and-white image. Figure 2 A to-be-recognized image provided for the present application is shown in FIG. 1. Figure 2 As shown in FIG. 1, a user in the to-be-recognized image wears black gloves.

[0067] The obtained to-be-recognized image is recognized to determine the coordinates of the skeletal points at the elbow and wrist. Specifically, the three-dimensional coordinates (x, y, z) of each joint point are estimated from the to-be-recognized image by a 3D human pose estimation algorithm, and the coordinates of the skeletal points at the elbow and wrist are determined from the three-dimensional coordinates of each joint point. The elbow includes the left elbow and the right elbow, and the wrist includes the left wrist and the right wrist.

[0068] The 3D human pose estimation is essentially a regression problem, which is a task of converting the motion characteristics of a human body in a 2D image into the motion of the human body in a corresponding 3D space by analyzing the motion characteristics of the human body in the 2D image. After completing the 2D human pose estimation, prior knowledge of human body mechanics is introduced, a human body parameter constraint function item is designed, and the continuity of the motion of the skeletal points between consecutive frames is used for 3D human pose estimation. For example, the 3D human pose estimation algorithm can be VideoPose3D.

[0069] Figure 3 A to-be-recognized image processed by a 3D human pose estimation algorithm provided for the present application is shown in FIG. 2. Figure 3 As shown in FIG. 2, the recognized joint points include the wrist, elbow, shoulder, and hip.

[0070] S102: According to the coordinates of the skeletal points, a maximum hand prediction box and a minimum hand prediction box are predicted. A color-based region growing algorithm is used to perform image segmentation in the to-be-recognized image according to a preset seed point, and each segmentation candidate region whose region edge is located between the minimum hand prediction box and the maximum hand prediction box is obtained.

[0071] The electronic device predicts the hand region according to the coordinates of the skeletal points. Since the wrist can be rotated backward / forward, flexed / extended, and abducted / adducted, a total of three degrees of freedom of the inclination pitch and the deflection, the hand freedom is an inherent property of the wrist. Based on the hand freedom, the hand prediction box (HROI) can be predicted according to the length of the lower arm and the direction of the lower arm.

[0072] Figure 4 A hand diagram provided for the present application is shown in FIG. 3. Figure 4As shown, when the forearm is at different angles with the camera, due to the limitation of the freedom of the hand, the area of the hand will have different performances, but when determining the hand prediction box in the image to be recognized, the hand in the image to be recognized is located on the extension line of the forearm.

[0073] The hand prediction box includes a maximum hand prediction box (HROI max ) and a minimum hand prediction box (HROI min ), the maximum hand prediction box is to guide the range of the following color-based region growing algorithm, and the minimum hand prediction box is to prevent the pattern on the back of the glove from affecting the selection of the segmentation candidate region.

[0074] Figure 5 A schematic diagram of predicting a hand prediction box in an image to be recognized is provided for the present application, as shown in Figure 5 , as shown in Figure 5 , there are two boxes in the left hand of the user, the large box is the maximum hand prediction box, and the small box is the minimum hand prediction box.

[0075] After the electronic device predicts the maximum hand prediction box and the minimum hand prediction box, it uses a color-based region growing algorithm to perform image segmentation in the image to be recognized according to a preset seed point, obtains a segmentation candidate region of each color, and determines that the segmentation candidate region is deleted when the coordinates of all the pixel points of the region edge reach the edge of the minimum hand prediction box and the coordinates of any pixel point of the region edge reach the edge of the maximum hand prediction box, and the remaining region edge is between the minimum hand prediction box and the maximum hand prediction box.

[0076] S103: According to the pixel point set of each segmentation candidate region, a hand contour region is detected by using a convex hull detection method, a region image of the hand contour region in the image to be recognized is filled with a preset pixel value, and the region image filled with skin color is input into a pre-stored gesture recognition model to obtain an output gesture recognition result.

[0077] According to the determined pixel point set of each segmentation candidate region, the electronic device detects the pixel point set of each segmentation candidate region by using a convex hull detection method, and detects a hand segmentation region that meets a preset condition as a hand contour region.

[0078] Figure 6 A schematic diagram of the hand contour region after convex hull detection is provided for the present application, Figure 6 , each dot in the figure is a vertex detected by the convex hull.

[0079] Since the color of the hand contour region is the color of the glove, the color of the glove will affect the accuracy of gesture recognition. In order to remove the influence of the color of the glove on gesture recognition, the electronic device fills the region image of the hand contour region in the to-be-recognized image with a preset pixel value, wherein the preset pixel value is a pixel value of the skin color of the skin, to obtain a region image after skin color filling.

[0080] Figure 7 A schematic diagram of a region image of a hand contour region after skin color filling provided in the present application is shown in Figure 7 As shown, the hand contour region after skin color filling is equivalent to a hand region without wearing a glove.

[0081] The region image after skin color filling is input into a pre-stored gesture recognition model to obtain an output gesture recognition result. The gesture recognition model can be a model pre-trained to recognize gestures without wearing a glove, or an existing gesture recognition model. The gesture recognition model can be a neural network model or a generator-discriminator network model, which is not limited in the present application.

[0082] In the present application, the coordinates of the elbow and wrist points in the to-be-recognized image are obtained, the maximum hand prediction box and the minimum hand prediction box are predicted according to the coordinates of the skeleton points, there are various colors on the glove worn by the user, and each segmentation candidate region whose region edge is located between the minimum hand prediction box and the maximum hand prediction box is obtained by image segmentation based on a color-based region growing algorithm according to a preset seed point. Each segmentation candidate region is a contour region of one color of the glove. The hand contour region of the glove edge is determined by a convex hull detection method. It can be in a fist state or an extended finger state. The region image of the hand contour region is filled with a preset pixel value for skin color, to remove the influence of the color of the glove on gesture recognition. The region image after skin color filling is input into a pre-stored gesture recognition model to obtain an output gesture recognition result, so as to universally recognize gestures when the user wears different gloves.

[0083] In order to predict the hand prediction box, on the basis of the above-mentioned embodiments, in the present application, the maximum hand prediction box and the minimum hand prediction box are predicted according to the coordinates of the skeleton points, which includes:

[0084] According to the coordinates of the skeleton points, the length of the small arm between the elbow and the wrist is determined, the first side length of the maximum hand prediction box is determined according to the length of the small arm, and the second side length of the minimum hand prediction box is determined according to the ratio of the length of the small arm to a preset value.

[0085] According to the coordinates of the bone points, the length of the forearm, and the angle value determination function, the angle value of the angle between the normal line perpendicular to the plane where the camera is located and the extension line of the elbow to the wrist is determined, and according to the sine function value of the angle value, the first side length and the second side length, the half value of the first product value of the sine function value and the first side length and the half value of the second product value of the sine function value and the second side length are determined.

[0086] According to the extension line of the elbow to the wrist, a square region with a center located on the extension line and a distance from the center to the wrist being the half value of the first product value and a side length being the first side length is determined as the maximum hand prediction frame, and a square region with a center located on the extension line and a distance from the center to the wrist being the half value of the second product value and a side length being the second side length is determined as the minimum hand prediction frame.

[0087] In order to predict the hand prediction frame, the electronic device determines the length of the forearm between the elbow and the wrist according to the coordinates of the bone points of the elbow and the wrist, for example, the coordinates of the bone point of the elbow are (x elbow , y elbow , z elbow ), the coordinates of the bone point of the wrist are (x wrist , y wrist , z wrist ), and the length of the forearm L is determined.

[0088] Wherein

[0089] According to the determined length of the forearm, the length of the forearm is determined as the first side length of the maximum hand prediction frame, that is, l max =L; according to the length of the forearm and a preset value, the ratio of the length of the forearm to the preset value is determined, wherein the preset value is not less than 2, for example, it can be 2, 3, 4 and the like, and the ratio is determined as the second side length of the minimum hand prediction frame, for example, l min =L / 3.

[0090] In order to determine the minimum hand prediction frame and the maximum hand prediction frame, the electronic device determines the angle value of the angle between the normal line perpendicular to the plane where the camera is located and the extension line of the elbow to the wrist according to the coordinates of the bone points of the elbow and the wrist, the length of the forearm and the angle value determination function Specifically, according to the absolute value of the difference between the coordinate values in the depth direction of the bone points of the elbow and the wrist, the ratio of the absolute value of the difference to the length of the forearm is determined, and the angle value is obtained by performing the inverse cosine function on the ratio, which is the angle value of the angle between the normal line perpendicular to the plane where the camera is located and the extension line of the elbow to the wrist.

[0091] Due to the limitations of hand freedom, the human hand cannot move between the elbow and wrist. However, due to the different relative angles between the camera and the arm, the center of the hand prediction bounding box may fall between the wrist and elbow when imaging in a 2D image. Therefore, the position of the center of the hand prediction bounding box is affected by the angle of the forearm, and a distance of sina*l / 2 needs to be added to the extension line of the forearm. When determining the maximum distance of the center of the hand prediction bounding box from the wrist, l is taken as l max When determining the distance from the center of the minimum hand prediction box to the wrist, l is set to l. min ∠a is usually in the range of [0°, 90°].

[0092] Based on the sine function value of the angle, the first side length, and the second side length, determine half of the first product of the sine function value and the first side length. Half of the first product is the distance from the center of the largest hand prediction box to the wrist. Also, determine half of the second product of the sine function value and the second side length. Half of the second product is the distance from the center of the smallest hand prediction box to the wrist.

[0093] Based on the extension line from the elbow to the wrist, a square region with its center on the extension line and a distance from the center to the wrist equal to half the first product value, and a side length equal to the first side length, is determined as the largest hand prediction box. A square region with its center on the extension line and a distance from the center to the wrist equal to half the second product value, and a side length equal to the second side length, is determined as the smallest hand prediction box.

[0094] Figure 8 A schematic diagram of a hand prediction frame provided in this application is shown below. Figure 8 As shown, the coordinates of the elbow are (x... elbow y elbow The coordinates of the wrist are (x) wrist y wrist The length of the forearm between the elbow and wrist is L. When ∠a is 90°, the right ( Figure 3 The larger bounding box on the left and right sides of the image is the largest predicted hand bounding box, and the smaller bounding box is the smallest predicted hand bounding box. The center dot of the larger bounding box is the center point of the largest predicted hand bounding box, and the center dot of the smaller bounding box is the center point of the smallest predicted hand bounding box.

[0095] To determine each segmentation candidate region, based on the above embodiments, in this application, the step of performing image segmentation on the image to be identified using a color-based region growing algorithm based on preset seed points to obtain each segmentation candidate region whose region edge is located between the minimum hand prediction box and the maximum hand prediction box includes:

[0096] According to the pixel value of the preset seed point and the pixel value of each first pixel point in the preset neighborhood range of the preset seed point, a target first pixel point in which the color similarity to the preset seed point is greater than a preset color similarity threshold is determined, the preset seed point and the target first pixel point are saved in a pixel point set, and the target first pixel point is taken as a target seed point for the next round of region growing until it is determined that there is no target seed point, and then the growth of the segmentation region composed of the pixel points in the pixel point set is stopped.

[0097] According to the segmentation region and the minimum hand prediction frame, if the segmentation region does not reach the edge of the minimum hand prediction frame, an adjacent point of the intersection point of the extension line of the elbow to the wrist and the segmentation region is determined, and the adjacent point is taken as a target seed point for the next round of region growing until the segmentation region stops growing.

[0098] If the segmentation region reaches the edge of the minimum hand prediction frame, the segmentation region is determined as a segmentation candidate region, each second pixel point in the preset neighborhood range of the target seed point for the last round of region growing of the segmentation region is determined, a target second pixel point in which each second pixel point is not located in the segmentation candidate region is taken as an updated preset seed point for the next segmentation candidate region growing, and region growing is performed based on the updated preset seed point to obtain a next segmentation candidate region until the region edge reaches the edge of the maximum hand prediction frame to obtain a last segmentation candidate region.

[0099] In order to determine each segmentation candidate region, the electronic device saves the coordinates of the preset seed point, wherein the preset seed point is set by the user in advance, for example, the preset seed point can be a skeletal point of the wrist, or a skeletal point near the wrist, Figure 9 A schematic diagram of a maximum hand prediction frame is provided for the embodiment of the present application, as Figure 9 shown, Figure 9 Any point in the line segment of the area where the wrist is located in the left (left in the figure) frame line of the middle can be a preset seed point.

[0100] The electronic device first performs color-based region growing from the preset seed point, specifically according to the pixel value of the preset seed point and the pixel value of each first pixel point in the preset neighborhood range of the preset seed point, wherein the preset neighborhood range can be a four-neighborhood range or an eight-neighborhood range, the preset seed point and each first pixel point in the preset neighborhood range are compared in color similarity, and the color similarity of each first pixel point to the preset seed point is determined.

[0101] In order to realize the color-based region growing, the electronic device further stores a preset color similarity threshold, wherein the preset color similarity threshold can be preset by a user, if it is desired to improve the accuracy of the determined segmentation candidate region, the preset color similarity threshold can be set to be smaller, if it is desired to improve the efficiency of determining the segmentation candidate region, the preset color similarity threshold can be set to be larger.

[0102] According to the preset color similarity threshold and the color similarity of each first pixel point and the preset seed point, the first pixel point with the color similarity greater than the preset color similarity threshold is determined as a target first pixel point, and the target first pixel point and the preset seed point are stored in the pixel point set, which can be the number of the pixel point or the coordinates of the pixel point.

[0103] The electronic device takes the target first pixel point as a target seed point for the next round of region growing, and performs region growing based on the target seed point, determines each first pixel point in the preset neighborhood range of the target seed point, determines the target first pixel point with the color similarity greater than the preset color similarity threshold from each first pixel point, saves it to the pixel point set and determines it as a new target seed point, until it is determined that there is no target seed point, and the segmentation region composed of the pixel points in the pixel point set is determined to stop growing.

[0104] Figure 10 A schematic diagram of a segmentation region provided by the embodiment of the present application is shown in Figure 10 As shown, the wrist bone point is taken as the preset seed point, the preset seed point is black, and each black pixel point determined after region growing based on the black color forms a segmentation region.

[0105] According to the determined segmentation region and the minimum hand prediction frame, the electronic device judges whether the segmentation region reaches the edge of the minimum hand prediction frame, which can be specifically that a line connecting the wrist bone point and the farthest pixel point of the region edge of the segmentation region is determined, and it is judged whether the line passes through any frame line of the minimum hand prediction frame, if yes, it is determined that the segmentation region reaches the edge of the minimum hand prediction frame, if not, it is determined that the segmentation region does not reach the edge of the minimum hand prediction frame.

[0106] If the segmented region does not reach the edge of the minimum hand prediction box, then the region growing of the segmented region composed of pixels in the pixel set continues. The adjacent points of the intersection of the extension line from the elbow to the wrist and the segmented region are determined, where the adjacent points are located outside the segmented region. The adjacent points are used as the target seed points for the next round of region growing. The region growing continues based on the target seed points until, for example, it is determined that there are no new target seed points, then the segmented region composed of pixels in the pixel set stops growing. If the segmented region reaches the edge of the minimum hand prediction box, then the segmented region is determined as a segmentation candidate region.

[0107] Figure 11 This application provides a schematic diagram illustrating the adjacent points of the intersection between the extension line from the elbow to the wrist and the segmented region when the segmented region does not reach the edge of the minimum hand prediction box. Figure 11 As shown, Figure 11 The oblique line in the middle is the extension of the line from the elbow to the wrist, located at... Figure 11 The area formed by the outline of the small box is the segmented area. The intersection of the extension line and the segmented area is adjacent to a dot in the direction of the extension line. This dot is the adjacent point of the intersection of the extension line from the elbow to the wrist and the segmented area.

[0108] Based on the target seed point of the last round of region growing in the segmented region, the electronic device determines each second pixel point within the preset neighborhood range of the target seed point. Based on the coordinates of each second pixel point, it determines the target second pixel point that is not located in the segmentation candidate region. The target second pixel point is used as the updated preset seed point for the next segmentation candidate region for region growing. Based on the updated preset seed point, the same region growing method is used to obtain the next segmentation candidate region until the edge of a certain segmentation candidate region reaches the edge of the maximum hand prediction box, and it is determined as the last segmentation candidate region.

[0109] To more accurately determine each segmentation candidate region, based on the above embodiments, in this application, after determining the segmentation region as a segmentation candidate region and before determining each second pixel point of the preset neighborhood range of the target seed point of the last round of region growing based on the target seed point of the last round of region growing of the segmentation region, the method further includes:

[0110] If the number of pixels in the pixel set of the segmentation candidate region is less than a preset threshold, then the pixel set of the segmentation candidate region is merged with the pixel set of the adjacent previous segmentation candidate region to determine the updated segmentation candidate region; and / or,

[0111] If a difference between the first average pixel value of the pixel points in the pixel point set of the segmentation candidate region and a second average pixel value of the pixel points in the pixel point set of the adjacent previous segmentation candidate region is less than a preset pixel value threshold, it is determined that the color difference between the segmentation candidate region and the adjacent previous segmentation candidate region is small, the pixel point set of the segmentation candidate region and the pixel point set of the adjacent previous segmentation candidate region are merged, and an updated segmentation candidate region is determined.

[0112] After the electronic device determines the segmentation region as a segmentation candidate region, in order to improve the accuracy of the determined segmentation candidate region, the electronic device performs merging of the segmentation candidate regions, specifically including two methods, the first method is to merge the segmentation candidate region with the adjacent previous segmentation candidate region when the number of pixel points in the pixel point set of the segmentation candidate region is small, and the second method is to merge two segmentation candidate regions with small color difference.

[0113] In the first method, the electronic device pre-stores a preset number threshold, wherein the preset number threshold is pre-set by the user, if it is desired to improve the accuracy of the determined segmentation candidate region, the preset number threshold can be set to be smaller, if it is desired to improve the efficiency of the determination of the hand contour region while ensuring the accuracy of the segmentation candidate region, the preset number threshold can be set to be larger, for example, the preset number threshold is L / 20, wherein L represents the length of the small arm, and the unit of L can be pixels.

[0114] According to the number of pixel points in the pixel point set of the segmentation candidate region and the preset number threshold, it is determined whether the number of pixel points is less than the preset number threshold, if it is determined that the number of pixel points is less than the preset number threshold, the pixel point set of the segmentation candidate region and the pixel point set of the adjacent previous segmentation candidate region are merged to obtain an updated segmentation candidate region composed of pixel points in the merged pixel point set; if it is determined that the number of pixel points is not less than the preset number threshold, no processing is performed on the segmentation candidate region.

[0115] In the second method, the electronic device determines a difference between the first average pixel value of the pixel points in the pixel point set of the segmentation candidate region and the second average pixel value of the pixel points in the pixel point set of the adjacent previous segmentation candidate region.

[0116] In order to merge pixel point sets with close colors, the electronic device also pre-stores a preset pixel value threshold, if it is desired to improve the accuracy of the determined segmentation candidate region, the preset pixel value threshold can be set to be smaller, if it is desired to improve the efficiency of the determination of the hand contour region while ensuring the accuracy of the segmentation candidate region, the preset number threshold can be set to be larger.

[0117] If the difference is less than the preset pixel value threshold, it is determined that the color difference between the segmentation candidate region and the adjacent previous segmentation candidate region is small, and the pixel point set of the segmentation candidate region and the pixel point set of the adjacent previous segmentation candidate region are merged to obtain an updated segmentation candidate region composed of pixel points in the merged pixel point set; if the difference is not less than the preset pixel value threshold, no processing is performed on the segmentation candidate region.

[0118] The electronic device can only implement the steps of the first method or the second method, or can implement the steps of the first method and the second method. Specifically, the second method can be implemented after the first method, or the first method can be implemented after the second method, or the first method and the second method can be implemented simultaneously, so as to obtain the updated segmentation candidate region. The application does not specifically limit the execution order of the first method and the second method, and the execution order can be flexibly set according to requirements.

[0119] In order to determine the hand contour region, on the basis of the above embodiments, in the present application, the method of detecting the hand contour region by using the convex hull detection according to the pixel point set of each segmentation candidate region comprises:

[0120] For each segmentation candidate region, the pixel points in the pixel point set of the segmentation candidate region are subjected to convex hull detection to detect each convex vertex and each convex hull defect point.

[0121] For each convex hull defect point, it is determined whether the convex hull defect depth of the convex hull defect point satisfies a first preset defect depth condition of a fist or a second preset defect depth condition of a finger stretching according to the convex hull defect point and the convex vertices on the adjacent two sides. If the convex hull defect depth of each convex hull defect point satisfies the first preset defect depth condition or the second preset defect depth condition, the segmentation candidate region is determined as a hand contour region.

[0122] In order to detect the hand contour region, the electronic device subjects the pixel points in the pixel point set of each segmentation candidate region to convex hull detection to detect each convex vertex and each convex hull defect point.

[0123] For each convex hull defect point, the electronic device determines whether the convex hull defect depth of the convex hull defect point satisfies a first preset defect depth condition of a fist or a first preset defect depth condition of a finger stretching according to the convex hull defect point and the convex vertices on the adjacent two sides.

[0124] If the convex defect depth of each convex defect point satisfies the first preset defect depth condition, the electronic device determines that the segmentation candidate region is a hand contour region. Figure 12 A hand contour region provided in the present application is shown in the following figure, Figure 12 The edge of the segmentation candidate region in the figure forms a hand contour.

[0125] In order to determine whether the convex defect depth of the convex defect point satisfies the first preset defect depth condition of the fist, in the embodiment of the present application, the step of determining whether the convex defect depth of the convex defect point satisfies the first preset defect depth condition of the fist according to the convex defect point and the convex vertices on the two adjacent sides includes:

[0126] If the first angle value of the first included angle at the convex defect point formed by the convex defect point and the convex vertices on the two adjacent sides is greater than the first preset angle value threshold, it is determined that the convex defect depth of the convex defect point satisfies the first preset defect depth condition of the fist.

[0127] The two edges connected to the convex defect point and the convex vertices on the two adjacent sides are taken as the two edges of the first included angle at the convex defect point, and the first included angle at the convex defect point is formed. If the first angle value of the first included angle is greater than the first preset angle value threshold, it is determined that the convex defect depth of the convex defect point is small, wherein the first preset angle value threshold can be 150 degrees, 121 degrees, 125 degrees, etc. The present application does not limit the comparison.

[0128] After comparing each convex defect point, if it is determined that the convex defect depth of each convex defect point is small, the electronic device determines that the detected vertex satisfies the first preset defect depth condition of the fist, and determines the segmentation candidate region as a hand contour region.

[0129] For example, the electronic device determines the convex defect point as point B, determines the convex vertices on the two adjacent sides of the convex defect point as points A and D respectively, and determines that the first included angle ∠ABD at point B is greater than 120 degrees. It is determined that the convex defect depth of the convex defect point satisfies the first preset defect depth condition of the fist.

[0130] In order to determine whether the convex defect depth of the convex defect point satisfies the second preset defect depth condition of the fist, in the embodiment of the present application, the step of determining whether the convex defect depth of the convex defect point satisfies the second preset defect depth condition of the fist according to the convex defect point and the convex vertices on the two adjacent sides includes:

[0131] determining a first line length of the convex hull defect point and the skeletal point of the wrist, and a second line length of the convex hull defect point and the skeletal point of the wrist, determining whether the minimum value of the second line length and the first line length is within a preset numerical range, if yes, determining that the first sub-condition is satisfied, if no, determining that the first sub-condition is not satisfied; wherein the preset numerical range is set by the user in advance, for example, the preset numerical range is 0.1R to 1.3R, wherein R is the average distance from the center of the hand segmentation candidate region to all convex vertices.

[0132] determining whether the minimum value of the second line length and the first line length is within a preset numerical range, if yes, determining that the first sub-condition is satisfied, if no, determining that the first sub-condition is not satisfied; wherein the preset numerical range is set by the user in advance, for example, the preset numerical range is 0.1R to 1.3R, wherein R is the average distance from the center of the hand segmentation candidate region to all convex vertices.

[0133] determining whether the minimum value of the second line length and the first line length is within a preset numerical range, if yes, determining that the first sub-condition is satisfied, if no, determining that the first sub-condition is not satisfied; wherein the preset numerical range is set by the user in advance, for example, the preset numerical range is 0.1R to 1.3R, wherein R is the average distance from the center of the hand segmentation candidate region to all convex vertices.

[0134] determining whether the minimum value of the second line length and the first line length is within a preset numerical range, if yes, determining that the first sub-condition is satisfied, if no, determining that the first sub-condition is not satisfied; wherein the preset numerical range is set by the user in advance, for example, the preset numerical range is 0.1R to 1.3R, wherein R is the average distance from the center of the hand segmentation candidate region to all convex vertices.

[0135] According to the first and second line lengths, the minimum and maximum of the first and second line lengths are determined, and a ratio of the minimum and maximum is determined. It is determined whether the ratio is not greater than a preset numerical threshold. If yes, it is determined that the third sub-condition is satisfied. If no, it is determined that the third sub-condition is not satisfied. The preset numerical threshold is set by a user in advance. The preset numerical threshold can be 0.8, 0.85, 0.7, etc. Preferably, the preset numerical threshold is 0.8.

[0136] According to the convex defect point and the convex vertex on each side of the convex defect point, the lengths of two lines connecting the convex defect point and the convex vertex on each side are determined. According to the lengths of the two lines, a ratio of the maximum and minimum is determined. It is determined whether the ratio is not less than a preset numerical threshold. If yes, it is determined that the fourth sub-condition is satisfied. If no, it is determined that the fourth sub-condition is not satisfied.

[0137] The electronic device determines each convex defect point. If it is determined that the convex defect depth of each convex defect point satisfies the second preset defect depth condition, it is determined that the segmentation candidate region is determined as a hand contour region.

[0138] Figure 13 A schematic diagram for determining a hand contour region is provided for the present application, as shown in Figure 13 The convex vertices A, D and the convex defect point B therebetween form a triangle ABD. A, B and the wrist point C form a triangle ABC. Point B is close to or on the circumference of the palm center circle. Since according to the characteristics of the stretched fingers, when AB and BD are too short, A and D are not the tip points. When AC and BC are close in length, A is not the tip point. When AB and BD are close in length, A and D are the tip points. Therefore, when the following conditions are met simultaneously, the convex defect point B is determined to satisfy the second defect depth condition of the stretched fingers: (1) when AC>BC, 0.1R≤BC≤1.3R; when AC<BC, 0.1R≤AC≤1.3R, where R is the average of the distances from the center of mass of the hand segmentation candidate region to all convex vertices; (2) ∠ACB<45° and ∠BCD<45°; (3) MIN(BC,AC) / MAX(BC,AC)≤0.8, where MIN(BC,AC) represents the minimum of BC and AC, and MAX(BC,AC) represents the maximum of BC and AC; (4) MAX(AB,BD) / MIN(AD,BD)≥0.8, where MAX(AB,BD) represents the maximum of AB and BD, and MIN(AD,BD) represents the minimum of AD and BD. When it is determined that each convex defect point satisfies the second defect depth condition, it is determined that the fingers are in the stretched state.

[0139] Figure 14 A structure schematic diagram of a gesture recognition device is provided for the present application, as shown inFigure 14 The device comprises:

[0140] The acquisition module 1401 is configured to acquire coordinates of skeletal points at elbows and wrists in an image to be recognized.

[0141] The determination module 1402 is configured to predict a maximum hand prediction box and a minimum hand prediction box according to the coordinates of the skeletal points, perform image segmentation in the image to be recognized by using a color-based region growing algorithm based on a preset seed point, obtain each segmentation candidate region whose region edge is located between the minimum hand prediction box and the maximum hand prediction box, and determine a hand contour region detected by using a convex hull detection method according to a pixel point set of each segmentation candidate region.

[0142] The recognition module 1403 is configured to perform skin color filling on a region image of the hand contour region in the image to be recognized by using a preset pixel value, and input the region image after skin color filling to a pre-stored gesture recognition model to obtain an output gesture recognition result.

[0143] Further, the determination module 1402 is specifically configured to determine a small arm length between the elbows and the wrists according to the coordinates of the skeletal points, determine a first side length of the maximum hand prediction box according to the small arm length, determine a second side length of the minimum hand prediction box according to a ratio of the small arm length to a preset value, determine a function according to the coordinates of the skeletal points, the small arm length, and an angle value, determine a normal line perpendicular to a plane where a camera is located and an angle value of an included angle of an extension line of the elbows to the wrists, determine a half value of a first product value of a sine function value of the angle value and the first side length and a half value of a second product value of the sine function value and the second side length according to the first side length and the second side length, and determine the maximum hand prediction box as a square region whose center is located on the extension line of the elbows to the wrists, whose distance from the center to the wrists is the half value of the first product value, and whose side length is the first side length, and determine the minimum hand prediction box as a square region whose center is located on the extension line and whose distance from the center to the wrists is the half value of the second product value and whose side length is the second side length.

[0144] Further, the determining module 1402 is specifically configured to: determine, according to a pixel value of a preset seed point and a pixel value of each first pixel point in a preset neighborhood range of the preset seed point, a target first pixel point in the each first pixel point, the target first pixel point having a color similarity greater than a preset color similarity threshold with the preset seed point, save the preset seed point and the target first pixel point in a pixel point set, take the target first pixel point as a target seed point for next round region growing, and stop growing when no target seed point is determined; and determine, according to the segmentation region and the minimum hand prediction box, whether the segmentation region reaches an edge of the minimum hand prediction box; if the segmentation region does not reach the edge of the minimum hand prediction box, determine an adjacent point of an intersection point of a line extending from the elbow to the wrist and the segmentation region, take the adjacent point as a target seed point for next round region growing, and stop growing when the segmentation region stops growing.

[0145] If the segmentation region reaches the edge of the minimum hand prediction box, determine that the segmentation region is a segmentation candidate region, determine each second pixel point in a preset neighborhood range of a target seed point for last round region growing of the segmentation candidate region, take a target second pixel point in the each second pixel point, which is not located in the segmentation candidate region, as an updated preset seed point for next segmentation candidate region growing, perform region growing based on the updated preset seed point to obtain a next segmentation candidate region, and obtain a last segmentation candidate region when a region edge reaches an edge of the maximum hand prediction box.

[0146] Further, the determining module 1402 is further configured to: before the determining each second pixel point in the preset neighborhood range of the target seed point for last round region growing of the segmentation candidate region, if a number of pixel points in the pixel point set of the segmentation candidate region is less than a preset number threshold, merge the pixel point set of the segmentation candidate region and a pixel point set of a previous segmentation candidate region adjacent to the segmentation candidate region to determine an updated segmentation candidate region; and / or if a difference between a first average pixel value of the pixel points in the pixel point set of the segmentation candidate region and a second average pixel value of the pixel points in the pixel point set of the previous segmentation candidate region adjacent to the segmentation candidate region is less than a preset pixel value threshold, determine that a color difference between the segmentation candidate region and the previous segmentation candidate region adjacent to the segmentation candidate region is small, merge the pixel point set of the segmentation candidate region and the pixel point set of the previous segmentation candidate region to determine the updated segmentation candidate region.

[0147] Further, the determination module 1402 is specifically configured to perform convex hull detection on the pixel points in the pixel point set of each segmentation candidate region, detect each convex vertex and each convex defect point; for each convex defect point, determine whether the convex defect depth of the convex defect point satisfies a first preset defect depth condition of a fist or a second preset defect depth condition of an extended finger according to the convex defect point and the convex vertices on the two adjacent sides, and if the convex defect depth of each convex defect point satisfies the first preset defect depth condition or the second preset defect point depth condition, determine the segmentation candidate region as a hand contour region.

[0148] Further, the determination module 1402 is specifically configured to determine that the convex defect depth of the convex defect point satisfies the first preset defect depth condition of the fist if a first angle value of a first included angle formed by the convex defect point and the convex vertices on the two adjacent sides is greater than a first preset angle value threshold.

[0149] Further, the determination module 1402 is specifically configured to determine a first line length of a first line connecting the convex defect point and a skeletal point of the wrist and a second line length of a second line connecting a convex vertex on any one side adjacent to the convex defect point and the skeletal point of the wrist, and if a minimum value of the second line length and the first line length is located in a preset numerical range, each second angle value of each second included angle formed by the skeletal point of the wrist, the convex defect point and the convex vertex on each side is less than a second preset angle value threshold, a ratio of a minimum value to a maximum value of the first line length and the second line length is not greater than a preset numerical threshold, and a ratio of a maximum value to a minimum value of two line lengths of the convex defect point and the convex vertex on each side is not less than the preset numerical threshold, determine that the convex defect depth of the convex defect point satisfies the second preset defect depth condition of the extended finger.

[0150] Figure 15 An electronic device structure schematic diagram is provided in the application, and on the basis of the above embodiments, the application further provides an electronic device, as shown in the figure, which comprises a processor 1501, a communication interface 1502, a memory 1503 and a communication bus 1504, wherein the processor 1501, the communication interface 1502 and the memory 1503 complete mutual communication through the communication bus 1504. Figure 15

[0151] The memory 1503 stores a computer program, and when the program is executed by the processor 1501, the processor 1501 executes the steps of the gesture recognition method of the above embodiments.

[0152] ​The communication bus mentioned in the above electronic device can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The communication bus can be divided into an address bus, a data bus, a control bus, etc. For the convenience of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus.

[0153] The communication interface 1502 is configured to communicate between the above electronic device and other devices.

[0154] The memory can include a Random Access Memory (RAM) and can also include a Non-Volatile Memory (NVM), such as at least one disk memory. Optionally, the memory can also be at least one storage device located away from the aforementioned processor.

[0155] The processor mentioned above can be a general-purpose processor, including a central processing unit, a network processor (NP), etc.; can also be a Digital Signal Processing (DSP), an application-specific integrated circuit, a field programmable gate array or other programmable logic device, a discrete gate or transistor logic device, a discrete hardware component, etc.

[0156] On the basis of the above embodiments, the present application further provides a computer readable storage medium, wherein the computer readable storage medium stores a computer program executable by a processor, and when the program runs on the processor, the processor executes the steps of the gesture recognition method of the above embodiments.

[0157] Those skilled in the art should understand that the embodiments of the present application can be provided as a method, a system, or a computer program product. Therefore, the present application can take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present application can take the form of a computer program product implemented on one or more computer usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer usable program code.

[0158] The computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart block or blocks. Figure 1 one or more flowcharts and / or blocks Figure 1 means for functionally implementing the

[0159] The computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart block or blocks. Figure 1 one or more flowcharts and / or blocks Figure 1 means for functionally implementing the

[0160] The computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart block or blocks. Figure 1 one or more flowcharts and / or blocks Figure 1 means for functionally implementing the

[0161] Obviously, numerous modifications and variations of the present application are possible in light of the above teachings. It is therefore to be understood that within the scope of the appended claims and their equivalents, the application can be practiced otherwise than as specifically described.

Claims

1. A gesture recognition method, characterized by, The method comprises: obtaining the coordinates of the skeleton points at the elbow and wrist in the image to be recognized; predicting a maximum hand prediction box and a minimum hand prediction box according to the coordinates of the skeleton points; performing image segmentation in the image to be recognized by using a color-based region growing algorithm based on a preset seed point to obtain each segmentation candidate region whose region edge is located between the minimum hand prediction box and the maximum hand prediction box, wherein each segmentation candidate region is a contour region of a color of a glove worn by a user, and the glove has multiple colors; determining a hand contour region by using a convex hull detection method according to a pixel point set of each segmentation candidate region; filling the skin color of a region image of the hand contour region in the image to be recognized by using a preset pixel value; and inputting the region image filled with the skin color into a pre-stored gesture recognition model to obtain an output gesture recognition result; wherein the image segmentation in the image to be recognized by using the color-based region growing algorithm based on the preset seed point to obtain each segmentation candidate region whose region edge is located between the minimum hand prediction box and the maximum hand prediction box comprises: determining a target first pixel point in each first pixel point of a preset neighborhood range of the preset seed point, which has a color similarity greater than a preset color similarity threshold value with the preset seed point, according to a pixel value of the preset seed point and a pixel value of each first pixel point of the preset neighborhood range of the preset seed point; saving the preset seed point and the target first pixel point in a pixel point set; taking the target first pixel point as a target seed point for the next round of region growing until no target seed point is determined, and then determining that the segmentation region composed of the pixel points in the pixel point set stops growing; judging whether the segmentation region reaches the edge of the minimum hand prediction box according to the segmentation region and the minimum hand prediction box; if the segmentation region does not reach the edge of the minimum hand prediction box, determining an adjacent point of the intersection point of the extension line of the elbow to the wrist and the segmentation region as a target seed point for the next round of region growing until the segmentation region stops growing; if the segmentation region reaches the edge of the minimum hand prediction box, determining the segmentation region as a segmentation candidate region; determining each second pixel point of a preset neighborhood range of the target seed point of the last round of region growing of the segmentation region; taking a target second pixel point in each second pixel point which is not located in the segmentation candidate region as an updated preset seed point for the next segmentation candidate region growing; and performing region growing based on the updated preset seed point to obtain the next segmentation candidate region until the region edge reaches the edge of the maximum hand prediction box to obtain the last segmentation candidate region; the method for determining the hand contour region by using the convex hull detection method according to the pixel point set of each segmentation candidate region comprises: performing convex hull detection on the pixel points in the pixel point set of each segmentation candidate region to detect each convex vertex and each convex hull defect point; For each convex hull defect point, whether the convex hull defect depth of the convex hull defect point satisfies a first preset defect depth condition of a fist clenching or a second preset defect depth condition of a hand stretching is determined according to the convex hull defect point and two adjacent convex vertices on both sides of the convex hull defect point, and if the convex hull defect depth of each convex hull defect point satisfies the first preset defect depth condition or the second preset defect depth condition, the segmentation candidate region is determined as a hand contour region.

2. The method of claim 1, wherein, The maximum hand prediction box and the minimum hand prediction box are predicted according to the coordinates of the skeleton points, and the method comprises the following steps: A small arm length between the elbow and the wrist is determined according to the coordinates of the skeleton points, a first side length of the maximum hand prediction box is determined according to the small arm length, and a second side length of the minimum hand prediction box is determined according to a ratio of the small arm length to a preset value; An angle value of an angle between a normal line perpendicular to a plane where the camera is located and an extension line of the elbow to the wrist is determined according to the coordinates of the skeleton points, the small arm length and an angle value determination function, a first product value of a sine function value of the angle value, the first side length and the second side length is determined, a half value of the first product value and a second product value of the sine function value, the first side length and the second side length are determined; According to the extension line of the elbow to the wrist, a square region with a center located on the extension line and a distance from the center to the wrist being the half value of the first product value and a side length being the first side length is determined as the maximum hand prediction box, and a square region with a center located on the extension line and a distance from the center to the wrist being the half value of the second product value and a side length being the second side length is determined as the minimum hand prediction box.

3. The method of claim 1, wherein, After the segmentation region is determined as the segmentation candidate region, before the target seed point of the last round of region growing is determined according to each second pixel point in a preset neighborhood range of the target seed point of the last round of region growing, the method further comprises the following steps: If the number of pixel points in the pixel point set of the segmentation candidate region is less than a preset number threshold, the pixel point set of the segmentation candidate region and the pixel point set of a previous segmentation candidate region adjacent to the segmentation candidate region are merged to determine an updated segmentation candidate region; and / or If a difference between a first average pixel value of the pixel points in the pixel point set of the segmentation candidate region and a second average pixel value of the pixel points in the pixel point set of the previous segmentation candidate region adjacent to the segmentation candidate region is less than a preset pixel value threshold, it is determined that the color difference between the segmentation candidate region and the previous segmentation candidate region adjacent to the segmentation candidate region is small, and the pixel point set of the segmentation candidate region and the pixel point set of the previous segmentation candidate region adjacent to the segmentation candidate region are merged to determine an updated segmentation candidate region.

4. The method of claim 1, wherein, The step of determining whether the convex hull defect depth of the convex hull defect point satisfies the first preset defect depth condition of the fist clenching or the second preset defect depth condition of the hand stretching according to the convex hull defect point and the two adjacent convex vertices on both sides of the convex hull defect point comprises the following steps: If a first angle value of a first included angle formed by the convex hull defect point and two adjacent convex vertices on two sides of the convex hull defect point is greater than a first preset angle value threshold, it is determined that the convex hull defect depth of the convex hull defect point satisfies a first preset defect depth condition of a fist.

5. The method of claim 1, wherein, The method further includes determining whether the convex hull defect depth of the convex hull defect point satisfies a second preset defect depth condition of a stretched finger, according to the convex hull defect point and the two adjacent convex vertices on two sides of the convex hull defect point. If a minimum value of the first connection length and the second connection length is within a preset numerical range, each second angle value of each second included angle formed by the wrist skeletal point and the convex hull defect point and each convex vertex on each side of the convex hull defect point is less than a second preset angle value threshold, a ratio of the minimum value and the maximum value of the first connection length and the second connection length is not greater than a preset numerical threshold, and a ratio of the maximum value and the minimum value of the two connection lengths between the convex hull defect point and each convex vertex on each side is not less than the preset numerical threshold, it is determined that the convex hull defect depth of the convex hull defect point satisfies the second preset defect depth condition of the stretched finger.

6. A gesture recognition apparatus, characterized by The device includes: an acquisition module configured to acquire coordinates of skeletal points at elbows and wrists in an image to be recognized; a determination module configured to predict a maximum hand prediction frame and a minimum hand prediction frame according to the coordinates of the skeletal points, perform image segmentation in the image to be recognized by using a color-based region growing algorithm based on a preset seed point, obtain each segmentation candidate region whose region edge is located between the minimum hand prediction frame and the maximum hand prediction frame, and determine a hand contour region detected by using a convex hull detection method according to a pixel point set of each segmentation candidate region; an identification module configured to perform skin color filling on a region image of the hand contour region in the image to be recognized by using a preset pixel value, input the region image filled with skin color to a pre-stored gesture recognition model, and obtain an output gesture recognition result. The determination module is specifically configured to: determine, according to a pixel value of a preset seed point and a pixel value of each first pixel point in a preset neighborhood range of the preset seed point, a target first pixel point in the each first pixel point, the target first pixel point having a color similarity greater than a preset color similarity threshold with the preset seed point, save the preset seed point and the target first pixel point in a pixel point set, take the target first pixel point as a target seed point for next round of region growing, and stop growing when no target seed point is determined, until a segmentation region composed of pixel points in the pixel point set is determined; determine, according to the segmentation region and the minimum hand prediction box, whether the segmentation region reaches an edge of the minimum hand prediction box; if the segmentation region does not reach the edge of the minimum hand prediction box, determine an adjacent point of an intersection point of a line extending from the elbow to the wrist and the segmentation region, take the adjacent point as a target seed point for next round of region growing, and stop growing until the segmentation region is stopped; if the segmentation region reaches the edge of the minimum hand prediction box, determine the segmentation region as a segmentation candidate region, determine, according to a target seed point of last round of region growing of the segmentation region, each second pixel point in a preset neighborhood range of the target seed point of last round of region growing, take a target second pixel point in the each second pixel point, which is not located in the segmentation candidate region, as an updated preset seed point for next segmentation candidate region growing, and perform region growing based on the updated preset seed point to obtain a next segmentation candidate region, until a region edge reaches an edge of the maximum hand prediction box to obtain a last segmentation candidate region. The determination module is specifically configured to: for each segmentation candidate region, perform convex hull detection on pixel points in a pixel point set of the segmentation candidate region to detect each convex vertex and each convex defect point; for each convex defect point, determine, according to the convex defect point and two adjacent convex vertices on two sides of the convex defect point, whether a convex defect depth of the convex defect point satisfies a first preset defect depth condition of a fist or a second preset defect depth condition of an extended finger, and if the convex defect depth of the each convex defect point satisfies the first preset defect depth condition or the second preset defect depth condition, determine the segmentation candidate region as a hand contour region.

7. An electronic device, comprising: It comprises: a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory complete mutual communication through the communication bus; The memory stores a computer program, and when the program is executed by the processor, the processor executes the steps of the gesture recognition method in any one of claims 1-5.

8. A computer-readable storage medium, characterized in that, It stores a computer program executable by the processor, and when the program runs on the processor, the processor executes the steps of the gesture recognition method in any one of claims 1-5.

Images