Hand joint point detection method and electronic equipment
By using the coordinates of the target hand joint node position in the hand image in the previous frame of hand image for hand detection and cropping, combined with the methods of rotational alignment and motion state determination, the instability problem of hand joint node detection in different scenes and postures is solved, and the accuracy and stability of the detection are improved.
Patent Information
- Application Number
- CN202411038471.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-30
- Publication Date
- 2025-03-07
Smart Images
Figure CN119580341A_ABST
Abstract
Description
Background Art
[0002] At present, the research on 3D (three dimensional) gesture interaction technology is gradually gaining popularity at home and abroad, and has become a key technology for AR (Augmented Reality) / VR (Virtual Reality) device interaction. The stability and real-time performance of 3D gesture interaction are the main factors affecting the user interaction experience. As the core technology of 3D gesture technology, hand joint detection is a key technology affecting interaction. Among them, the stability of the joint detection results directly affects the stability of the interactive experience and the actual experience effect.
[0003] Hand joint point detection will inevitably produce unstable output results in different scenes and different hand postures, that is, the accuracy of hand joint point detection results is low. To avoid this situation, the most commonly used method is to train the hand joint point detection model by adding relevant scenes and posture gesture data. To alleviate the problem of hand shaking. However, due to the high degree of hand freedom, the posture of the training samples is too rich, and it is necessary to take into account any scene and posture. Hand joint point detection inevitably uses a model with a large number of parameters, which in turn creates higher requirements for the platform computing power. Therefore, a new method for hand joint point detection is urgently needed. Summary of the invention
[0004] The present application provides a hand joint point detection method and electronic equipment for improving the accuracy of hand joint point detection.
[0005] In a first aspect, an embodiment of the present application provides a method for detecting hand joint points, the method comprising:
[0006] For any frame of hand image, hand detection is performed on the any frame of hand image according to the image position coordinates of the target hand joint points in the previous frame of hand image of the any frame of hand image, so as to obtain the position of the hand area in the any frame of hand image, wherein the target hand joint points are the joint points in the palm area;
[0007] Cropping the any frame of hand image according to the position of the hand region in the any frame of hand image to obtain a hand region image;
[0008] Performing joint point detection on the hand region image using a pre-trained hand joint point detection model to obtain image position coordinates of each hand joint point in the hand region image;
[0009] Determining the motion state of the hand in the hand area image based on the image position coordinates of the first joint point specified in the hand area image and the image position coordinates of the first joint point in each target hand image, wherein each target hand image is a hand image frame of a specified number of frames before any one frame of the hand image, and the motion state is used to indicate whether the hand is stationary;
[0010] Based on the movement state of the hand in the hand area image, the target position coordinates of each hand joint point in any frame of the hand image are obtained.
[0011] A second aspect of the present application provides an electronic device, comprising a processor and a memory, wherein the processor and the memory are connected via a bus.
[0012] The memory stores a computer program, and the processor is configured to perform the following operations based on the computer program:
[0013] For any frame of hand image, hand detection is performed on the any frame of hand image according to the image position coordinates of the target hand joint points in the previous frame of hand image of the any frame of hand image, so as to obtain the position of the hand area in the any frame of hand image, wherein the target hand joint points are the joint points in the palm area;
[0014] Cropping the any frame of hand image according to the position of the hand region in the any frame of hand image to obtain a hand region image;
[0015] Performing joint point detection on the hand region image using a pre-trained hand joint point detection model to obtain image position coordinates of each hand joint point in the hand region image;
[0016] Determining the motion state of the hand in the hand area image based on the image position coordinates of the first joint point specified in the hand area image and the image position coordinates of the first joint point in each target hand image, wherein each target hand image is a hand image frame of a specified number of frames before any one frame of the hand image, and the motion state is used to indicate whether the hand is stationary;
[0017] Based on the movement state of the hand in the hand area image, the target position coordinates of each hand joint point in any frame of the hand image are obtained.
[0018] According to a third aspect provided by an embodiment of the present invention, a computer storage medium is provided, wherein the computer storage medium stores a computer program, and the computer program is used to execute the method as described in the first aspect.
[0019] In the above-mentioned embodiment of the present application, the image position coordinates of each hand joint point in the previous frame of the hand image of any frame of the hand image are first used to perform hand detection on any frame of the hand image, and the position of the hand area in any frame of the hand image is obtained, and any frame of the hand image is cropped according to the position of the hand area in any frame of the hand image to obtain the hand area image. Therefore, the problem of sudden change in scale size of the hand area in each input image of the joint point detection model of the user's hand during the pinching process is avoided. The consistency of the input of the joint point detection model is guaranteed, and the accuracy of the detection result of the hand joint point is guaranteed. Then, based on the image position coordinates of the first joint point specified in the hand area image and the image position coordinates of the first joint point in each target hand image, the movement state of the hand in the hand area image is determined, and based on the movement state of the hand in the hand area image, the target position coordinates of each hand joint point in any frame of the hand image are obtained. It is ensured that the hand posture can be locked when the hand is stationary, and the problem of slight shaking when the hand is stationary is avoided, and the accuracy of the joint point detection result is improved. BRIEF DESCRIPTION OF THE DRAWINGS
[0020] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.
[0021] Figure 1 One of the flowcharts of the hand joint point detection method provided in the embodiment of the present application is exemplarily shown;
[0022] Figure 2 A schematic diagram of hand joint points provided by an embodiment of the present application is exemplarily shown;
[0023] Figure 3 The flowchart for determining the position of the hand region in any frame of hand image provided by the embodiment of the present application is exemplarily shown;
[0024] Figure 4 The schematic diagram of the hand area detection frame provided by the embodiment of the present application is exemplarily shown;
[0025] Figure 5 A schematic diagram of rotationally aligning a hand area image provided by an embodiment of the present application is exemplarily shown;
[0026] Figure 6 The flowchart of determining the rotation matrix of the hand region image provided by the embodiment of the present application is exemplarily shown;
[0027] Figure 7 A schematic diagram showing, by way of example, images of a hand region before and after rotation alignment provided by an embodiment of the present application;
[0028] Figure 8 The flowchart for determining the motion state of a hand in a hand region image provided by an embodiment of the present application is exemplarily shown;
[0029] Fig. 9 The second flowchart of the hand joint point detection method provided in the embodiment of the present application is exemplarily shown;
[0030] Fig.10 A schematic diagram of a hand joint point detection device provided by an embodiment of the present application is exemplarily shown;
[0031] Fig.11 The structural diagram of the electronic device provided by the embodiment of the present application is exemplified. DETAILED DESCRIPTION
[0032] In order to make the purpose, implementation mode and advantages of the present application clearer, the exemplary implementation mode of the present application will be clearly and completely described below in conjunction with the drawings in the exemplary embodiments of the present application. Obviously, the described exemplary embodiments are only part of the embodiments of the present application, rather than all the embodiments.
[0033] Based on the exemplary embodiments described in this application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the claims attached to this application. In addition, although the disclosure in this application is introduced according to one or several exemplary examples, it should be understood that each aspect of the disclosure can also constitute a complete implementation method separately.
[0034] It should be noted that the brief description of terms in this application is only for the convenience of understanding the embodiments described below, and is not intended to limit the embodiments of this application. Unless otherwise specified, these terms should be understood according to their common and usual meanings.
[0035] The terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the numbers used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover but not exclude inclusion, for example, a product or device comprising a series of components is not necessarily limited to those components clearly listed, but may include other components not clearly listed or inherent to these products or devices.
[0036] The term "module" as used in this application refers to any known or later developed hardware, software, firmware, artificial intelligence, fuzzy logic or combination of hardware and / or software code that is capable of performing the functions associated with that element.
[0037] The idea of the embodiments of the present application is summarized below.
[0038] Based on the existing technology, the hand joint point detection will inevitably have unstable output results in different scenes and different hand postures, that is, the accuracy of the hand joint point detection results is low. The embodiment of the present application provides a method for detecting hand joint points. First, the image position coordinates of each hand joint point in the previous frame of the hand image of any frame of hand image are used to perform hand detection on any frame of hand image to obtain the position of the hand area in any frame of hand image, and the any frame of hand image is cropped according to the position of the hand area in any frame of hand image to obtain a hand area image. Therefore, the problem of sudden change in scale size of the hand area in each input image of the joint point detection model of the user's hand during the kneading process is avoided. The consistency of the joint point detection model input is guaranteed, and the accuracy of the hand joint point detection results is guaranteed. Then, based on the image position coordinates of the first joint point specified in the hand region image and the image position coordinates of the first joint point in each target hand image, the motion state of the hand in the hand region image is determined, and based on the motion state of the hand in the hand region image, the target position coordinates of each hand joint point in any frame of the hand image are obtained. This ensures that the hand posture can be locked when the hand is stationary, avoids the problem of slight shaking when the hand is stationary, and improves the accuracy of the joint point detection results.
[0039] like Figure 1 FIG. 1 is a flow chart of a method for detecting hand joint points, which may include the following steps:
[0040] Step 101: for any frame of hand image, hand detection is performed on the any frame of hand image according to the image position coordinates of the target hand joint points in the previous frame of hand image of the any frame of hand image, so as to obtain the position of the hand area in the any frame of hand image, wherein the target hand joint points are the joint points in the palm area;
[0041] like Figure 2 As shown, this is a schematic diagram of the hand joints. Figure 2 In this example, the number of hand joints is 21. Assuming that the last frame of the hand image corresponds to the time t-1, the data of the image position coordinates of the 21 joints detected in the tracking state are Joints t-1={(u0, v0), (u1, v1),…(u 19 , v 19 ), (u 20 , v 20 )}.
[0042] Next, the specific method of determining the position of the hand region in any frame of the hand image in step 101 is introduced. Figure 3 As shown, it is a schematic diagram of a flow chart for determining the position of the hand region in any frame of hand image, which may specifically include the following steps:
[0043] Step 301: obtaining the position of the palm area in any one frame of the hand image according to the image position coordinates of each target hand joint point in the previous frame of the hand image;
[0044] The position of the palm area in the embodiment of the present application includes the coordinates of the center point of the palm area, the height of the palm area and the width of the palm area. The specific method of determining the position of the palm area in step 301 is described below:
[0045] (1) Coordinates of the center point of the palm area:
[0046] The horizontal coordinate of the center point of the palm area is obtained according to the horizontal coordinate with the largest value and the horizontal coordinate with the smallest value in the image position coordinates of each target joint point; and the vertical coordinate of the center point of the palm area is obtained according to the vertical coordinate with the largest value and the vertical coordinate with the smallest value in the image position coordinates of each target joint point.
[0047] In one embodiment, the horizontal coordinate of the center point of the palm area is obtained in the following manner:
[0048] The difference between the maximum horizontal coordinate and the minimum horizontal coordinate is divided by the specified value to obtain the horizontal coordinate of the center point of the palm area. The horizontal coordinate of the center point of the palm can be obtained by formula (1):
[0049]
[0050] Wherein, u0 is the horizontal coordinate of the center point of the palm, u max is the horizontal coordinate with the maximum value, u min is the horizontal coordinate with the minimum value, and r is the specified value.
[0051] In one embodiment, the ordinate of the center point of the palm area is obtained in the following manner:
[0052] The difference between the maximum ordinate and the minimum ordinate is divided by the specified value to obtain the ordinate of the center point of the palm area. The ordinate of the center point of the palm can be obtained by formula (2):
[0053]
[0054] Wherein, v0 is the ordinate of the center point of the palm, v max is the vertical coordinate with the maximum value, v min is the ordinate where the value is the smallest.
[0055] The specified numerical value in the embodiment of the present application is 2, but the specific value of the specified numerical value is not limited and can be set according to actual conditions.
[0056] (2) Width of the palm area:
[0057] Subtract the maximum horizontal coordinate from the minimum horizontal coordinate in the image position coordinates of each target joint point to obtain the width of the palm area. The width of the palm area can be obtained by formula (3):
[0058] w=u max -u min ......(3);
[0059] Wherein, w is the width of the palm area.
[0060] (3) Height of the palm area:
[0061] Subtract the ordinate with the largest value and the ordinate with the smallest value in the image position coordinates of each target joint point to obtain the height of the palm area. The height of the palm area can be obtained by formula (4):
[0062] h=v max -v min ......(4);
[0063] Wherein, h is the height of the palm area.
[0064] Step 302: enlarging the position of the palm region in the arbitrary frame of hand image to obtain the position of the hand region in the arbitrary frame of hand image.
[0065] The position of the hand area in the embodiment of the present application includes the position coordinates of the center point of the hand area, the height of the hand area, and the width of the hand area. The following is an explanation of the method for determining the position of the hand area in the embodiment of the present application:
[0066] (1) The position coordinates of the center point of the hand area:
[0067] The image position coordinates of the second joint point specified in the previous frame of hand image are determined as the position coordinates of the center point of the hand area.
[0068] The second joint point in the embodiment of the present application is the joint point where the index finger connects to the palm. Figure 2 However, the second joint point is not limited in the embodiment of the present application, and can be specifically set according to actual conditions.
[0069] (2) Width of the hand area:
[0070] The width of the palm area is enlarged by a specified multiple to obtain the width of the hand area.
[0071] (3) Height of the hand area:
[0072] The height of the palm area is enlarged by the specified multiple to obtain the height of the hand area.
[0073] It should be noted that the specified multiple in the embodiment of the present application is 2, but the specified multiple is not limited. The specified multiple in the embodiment of the present application can be set according to the specific actual situation.
[0074] like Figure 4 As shown, the left side is the detection frame corresponding to the hand area obtained by conventional hand area detection, and the right side is the detection frame of the hand area obtained by this solution. The hand area detection frame obtained in the embodiment of the present application can ensure that the hand size scale in the image input to the joint point detection model remains stable, and further improve the stability of specific gesture interaction processes such as pinching. Avoid the problem of sudden changes in the size of the hand area in the image input to the joint point detection model during the pinching process.
[0075] Step 101 in the embodiment of the present application can also be implemented as: inputting the image position coordinates of the target hand joint point in the previous frame of the hand image of any one frame of the hand image into a pre-trained hand area detection model to obtain the position of the hand area in the any one frame of the hand image.
[0076] Step 102: cropping the arbitrary frame of hand image according to the position of the hand region in the arbitrary frame of hand image to obtain a hand region image;
[0077] Since the hand skeleton is a non-rigid structure with a high degree of freedom, it is difficult to use the hand joint point detection model to perform high-precision detection of the joint points in the hand area image. In order to alleviate the detection difficulty of the hand joint point detection model and improve the detection accuracy. In the embodiment of the present application, it is necessary to perform rotation alignment processing on the hand area image to simplify the diversity problem of different perspectives and reduce the multi-perspective complexity of the input image of the model.
[0078] like Figure 5 As shown, it is a schematic diagram of the process of rotating and aligning the hand area image, which may specifically include the following steps:
[0079] Step 501: Obtaining a rotation matrix of the hand region image through the image position coordinates of each hand joint point in the previous frame of the hand image and a preset standard vector; wherein the standard vector is used to represent a vector that is perpendicular to the horizontal axis in the image coordinate system and has a direction opposite to that of the vertical axis;
[0080] Next, the method of determining the rotation matrix of the hand region image is described in detail. Figure 6 As shown, it is a schematic diagram of a process of determining the rotation matrix of the hand region image, which may specifically include the following steps:
[0081] Step 601: Obtain the direction vector corresponding to the target hand image according to the image position coordinates of the third joint point specified in each hand joint point in the previous frame of the hand image, wherein the third joint point includes the joint point where the middle finger and the palm are connected and the root node of the palm; wherein the direction vector corresponding to the target hand image can be obtained by formula (5):
[0082]
[0083] in, is the direction vector corresponding to the target hand image, u9 is the image position horizontal coordinate of the joint point where the middle finger and the palm are connected in the third joint, v9 is the image position vertical coordinate of the joint point where the middle finger and the palm are connected in the third joint, u0 is the image position horizontal coordinate of the root node of the palm in the third joint, and v0 is the image position vertical coordinate of the root node of the palm in the third joint.
[0084] The joint point where the middle finger and the palm are connected in the embodiment of the present application is Figure 2 The joint point with index 9 in the palm and the root node of the palm are Figure 2 The joint point with index 0 in .
[0085] Step 602: Based on the direction vector and the standard vector, obtain the vector angle between the direction vector and the standard vector; wherein the vector angle between the direction vector and the standard vector can be obtained by formula (6):
[0086]
[0087] Wherein, θ is the vector angle between the direction vector and the standard vector, is the standard vector.
[0088] Step 603: using the vector angle, obtaining rotation parameters of the target hand image, wherein the rotation parameters include a rotation angle and a rotation direction;
[0089] The rotation direction in the embodiment of the present application is counterclockwise.
[0090] In one embodiment, step 603 can be specifically implemented as follows: if the difference between the image position horizontal coordinate of the joint point where the middle finger and the palm are connected in the third joint point and the image position horizontal coordinate of the root node of the palm in the third joint point is not less than a preset value, then the vector angle is determined as the rotation angle. Or, if the difference between the image position horizontal coordinate of the joint point where the middle finger and the palm are connected in the third joint point and the image position horizontal coordinate of the root node of the palm in the third joint point is less than a preset value, then the difference between the specified angle and the vector angle is determined as the rotation angle.
[0091] The specified angle in the embodiment of the present application is 2π. The rotation angle can be obtained by formula (7):
[0092]
[0093] Among them, θ rot is the rotation angle.
[0094] Step 604: Obtain a rotation matrix of the target hand image using the rotation parameters of the target hand image.
[0095] In one embodiment, step 604 may be specifically implemented as follows: inputting the rotation parameters of the target hand image into the getRotationMatrix2D function of OpenCV to obtain the rotation matrix.
[0096] Step 502: Use the rotation matrix to rotate the hand area image to obtain a rotated hand area image, and determine the rotated hand area image as the hand area image.
[0097] In the embodiment of the present application, the warpAffine function of OpenCV is combined with the rotation matrix to rotate the hand area image. The hand area image is rotated to the state where the direction vector corresponding to the third joint point is vertically upward. The hand area image before and after the rotation alignment is as follows Figure 7 As shown, from Figure 7 It can be seen from the figure that the direction vector corresponding to the third joint point in the rotated hand area image is vertically upward.
[0098] Step 103: Perform joint point detection on the hand region image using a pre-trained hand joint point detection model to obtain image position coordinates of each hand joint point in the hand region image;
[0099] In the embodiment of the present application, the hand joint point detection model can be set according to the actual situation, and the embodiment of the present application does not limit the hand joint point detection model.
[0100] Step 104: determining the motion state of the hand in the hand region image based on the image position coordinates of the first joint point specified in the hand region image and the image position coordinates of the first joint point in each target hand image, wherein each target hand image is a hand image of a specified number of frames before any one frame of the hand image, and the motion state is used to indicate whether the hand is stationary;
[0101] Next, the method of determining the motion state of the hand in the hand region image in step 104 is introduced. Figure 8 As shown, it is a schematic diagram of a process of determining the motion state of a hand in a hand region image, which may specifically include the following steps:
[0102] Step 801: for any one frame of the target hand images, based on the image position coordinates of the first joint point in the any one frame of the target hand image and the image position coordinates of the first joint point in the hand region image, obtain the Euclidean distance between the hand region image and the any one frame of the target hand image;
[0103] The method of determining the Euclidean distance by the image position coordinates in the embodiment of the present application belongs to the prior art, and the embodiment of the present application will not be described in detail here.
[0104] Step 802: comparing the average value of the Euclidean distances between the hand region image and each of the target hand images with a preset threshold;
[0105] The preset threshold in the embodiment of the present application can be set according to the specific actual situation, and the embodiment of the present application does not limit the specific value of the preset threshold.
[0106] Step 803: Determine whether the average value is greater than the preset threshold value, if so, execute step 804, if not, execute step 805;
[0107] Step 804: determining that the motion state of the hand in the hand region image is a non-stationary state;
[0108] Step 805: Determine whether the movement state of the hand in the hand area image is a stationary state.
[0109] Step 104 in the embodiment of the present application can also be specifically implemented as follows: for any frame of target hand images among the target hand images, based on the image position coordinates of the first joint point in the any frame of target hand images, obtain a first vector corresponding to the any frame of target hand images; and according to the image position coordinates of the first joint point in the hand area image, obtain a second vector corresponding to the any frame of target hand images; according to the first vector and the second vector, obtain the angle between the first vector and the second vector; if the angle is greater than the specified angle, determine that the motion state of the hand area is a non-stationary state; if the angle is not greater than the specified angle, determine that the motion state of the hand area is a stationary state.
[0110] In the embodiment of the present application, the method for determining the first vector is: subtracting the image position coordinates of the first joint point in any frame of the target hand image from the coordinate origin in the image position coordinate system to obtain the first vector.
[0111] In the embodiment of the present application, the second vector is determined by subtracting the image position coordinates of the first joint point in the hand area image from the coordinate origin in the image position coordinate system to obtain the second vector.
[0112] It should be noted that the specified angle in the embodiment of the present application can be set according to actual conditions, and the embodiment of the present application does not limit the specific value of the specified angle.
[0113] Step 105: Based on the movement state of the hand in the hand area image, the target position coordinates of each hand joint point in any frame of the hand image are obtained.
[0114] In one embodiment, step 105 can be specifically implemented as follows: if the motion state of the hand in the hand region image is a static state, the image position coordinates of each hand joint point in the hand image corresponding to the last time the hand was not in a static state are determined as the target position coordinates of each hand joint point in any one frame of the hand image. Or, if the motion state of the hand in the hand region image is a non-static state, the image position coordinates of each hand joint point in the hand region image are determined as the target position coordinates of each hand joint point in any one frame of the hand image.
[0115] In order to further understand the detection method of hand joint points in the embodiment of the present application, Fig. 9 FIG. 1 is a flow chart of a method for detecting hand joint points, which may specifically include the following steps:
[0116] Step 901: for any frame of hand image, according to the image position coordinates of the target hand joint point in the previous frame of hand image of the any frame of hand image, obtain the position of the palm area in the any frame of hand image;
[0117] Step 902: enlarging the position of the palm region in the arbitrary frame of hand image to obtain the position of the hand region in the arbitrary frame of hand image;
[0118] Step 903: cropping the hand image in any frame according to the position of the hand region in the hand image to obtain a hand region image;
[0119] Step 904: Obtaining a rotation matrix of the hand region image through the image position coordinates of each hand joint point in the previous frame of the hand image and a preset standard vector; wherein the standard vector is used to represent a vector that is perpendicular to the horizontal axis in the image coordinate system and has a direction opposite to that of the vertical axis;
[0120] Step 905: rotating the hand region image using the rotation matrix to obtain a rotated hand region image, and determining the rotated hand region image as the hand region image;
[0121] Step 906: for any one frame of the target hand images, based on the image position coordinates of the first joint point in the any one frame of the target hand image and the image position coordinates of the first joint point in the hand region image, obtain the Euclidean distance between the hand region image and the any one frame of the target hand image;
[0122] Step 907: comparing the average value of the Euclidean distances between the hand region image and each of the target hand images with a preset threshold;
[0123] Step 908: Determine whether the average value is greater than the preset threshold value, if so, execute step 909, if not, execute step 910;
[0124] Step 909: determining that the motion state of the hand in the hand region image is a non-static state, and determining the image position coordinates of each hand joint point in the hand region image as the target position coordinates of each hand joint point in any one frame of the hand image;
[0125] Step 910: Determine that the movement state of the hand in the hand area image is a static state, and determine the image position coordinates of each hand joint point in the hand image corresponding to the last time the hand was not in a static state as the target position coordinates of each hand joint point in any frame of the hand image.
[0126] Based on the same inventive concept, the hand joint point detection method disclosed above can also be implemented by a hand joint point detection device. The effect of the hand joint point detection device is similar to that of the above method, which will not be repeated here.
[0127] Fig.10 The figure is a schematic diagram of the structure of a device for detecting hand joint points according to an embodiment of the present disclosure.
[0128] like Fig.10 As shown, the hand joint point detection device 1000 disclosed in the present invention may include a hand detection module 1010 , an image cropping module 1020 , a first joint point detection module 1030 , a motion state determination module 1040 and a second joint point detection module 1050 .
[0129] The hand detection module 1010 is used to perform hand detection on any frame of hand image according to the image position coordinates of the target hand joint points in the previous frame of hand image of the any frame of hand image, so as to obtain the position of the hand area in the any frame of hand image, wherein the target hand joint points are the joint points in the palm area;
[0130] An image cropping module 1020, configured to crop the hand image of any frame according to the position of the hand region in the hand image of any frame to obtain a hand region image;
[0131] A first joint point detection module 1030 is used to perform joint point detection on the hand region image using a pre-trained hand joint point detection model to obtain image position coordinates of each hand joint point in the hand region image;
[0132] The motion state determination module 1040 is used to determine the motion state of the hand in the hand region image based on the image position coordinates of the first joint point specified in the hand region image and the image position coordinates of the first joint point in each target hand image, wherein each target hand image is a frame of hand image that is a specified number of frames before any frame of hand image, and the motion state is used to indicate whether the hand is stationary;
[0133] The second joint point detection module 1050 is used to obtain the target position coordinates of each hand joint point in any frame of the hand image based on the movement state of the hand in the hand area image.
[0134] In one embodiment, the apparatus further comprises:
[0135] The rotation matrix determination module 1060 is used to obtain the rotation matrix of the hand region image by using the image position coordinates of each hand joint point in the previous frame of the hand image and a preset standard vector before performing joint point detection on the hand region image using the pre-trained hand joint point detection model; wherein the standard vector is used to represent a vector that is perpendicular to the horizontal coordinate axis in the image coordinate system and has a direction opposite to that of the vertical coordinate;
[0136] The image rotation module 1070 is used to rotate the hand region image using the rotation matrix to obtain a rotated hand region image, and determine the rotated hand region image as the hand region image.
[0137] In one embodiment, the rotation matrix determination module 1060 is specifically used to:
[0138] Obtaining a direction vector corresponding to the target hand image according to the image position coordinates of a third joint point specified in each hand joint point in the previous frame of the hand image, wherein the third joint point includes a joint point at which the middle finger and the palm are connected and a root node of the palm;
[0139] Based on the direction vector and the standard vector, a vector angle c between the direction vector and the standard vector is obtained.
[0140] Using the vector angle, obtaining rotation parameters of the target hand image, wherein the rotation parameters include a rotation angle and a rotation direction;
[0141] The rotation matrix of the target hand image is obtained through the rotation parameters of the target hand image.
[0142] In one embodiment, the hand detection module 1010 is specifically used for:
[0143] Obtaining the position of the palm area in any one frame of the hand image according to the image position coordinates of each target hand joint point in the previous frame of the hand image;
[0144] The position of the palm area in the arbitrary frame of hand image is enlarged to obtain the position of the hand area in the arbitrary frame of hand image.
[0145] In one embodiment, the hand detection module 1010 executes the image position coordinates of the target joint points in each hand joint point of the previous frame of hand image to obtain the position of the palm area in any frame of hand image, specifically for:
[0146] Obtaining the horizontal coordinate of the center point of the palm area and the width of the palm area according to the horizontal coordinate with the largest value and the horizontal coordinate with the smallest value in the image position coordinates of each target joint point; and
[0147] Obtaining the ordinate of the center point of the palm area and the height of the palm area according to the ordinate with the largest value and the ordinate with the smallest value in the image position coordinates of each target joint point;
[0148] The position of the palm area is obtained based on the abscissa of the center point, the ordinate of the center point, the width, and the height.
[0149] In one embodiment, the hand detection module 1010 performs the step of enlarging the position of the palm area in the arbitrary frame of hand image to obtain the position of the hand area in the arbitrary frame of hand image, specifically for:
[0150] Determining the image position coordinates of the second joint point specified in the previous frame of the hand image as the position coordinates of the center point of the hand area; and
[0151] The width of the palm area is enlarged by a specified multiple to obtain the width of the hand area; and the height of the palm area is enlarged by the specified multiple to obtain the height of the hand area;
[0152] The position of the hand region in any frame of the hand image is obtained according to the position coordinates of the center point of the hand region, the width of the hand region and the height of the hand region.
[0153] In one embodiment, the motion state determination module 1040 is specifically configured to:
[0154] For any one frame of the target hand images, based on the image position coordinates of the first joint point in the any one frame of the target hand image and the image position coordinates of the first joint point in the hand region image, obtain the Euclidean distance between the hand region image and the any one frame of the target hand image;
[0155] Comparing the average value of the Euclidean distances between the hand region image and each target hand image with a preset threshold;
[0156] If the average value is not greater than the preset threshold, it is determined that the movement state of the hand in the hand area image is a stationary state; or,
[0157] If the average value is greater than the preset threshold, it is determined that the movement state of the hand in the hand area image is a non-stationary state.
[0158] In one embodiment, the second joint point detection module 1050 is specifically used to:
[0159] If the motion state of the hand in the hand area image is a stationary state, the image position coordinates of each hand joint point in the hand image corresponding to the last time the hand was not in a stationary state are determined as the target position coordinates of each hand joint point in any frame of the hand image; or,
[0160] If the movement state of the hand in the hand area image is a non-static state, the image position coordinates of each hand joint point in the hand area image are determined as the target position coordinates of each hand joint point in any frame of the hand image.
[0161] After introducing a method and device for detecting hand joints according to an exemplary embodiment of the present invention, an electronic device according to another exemplary embodiment of the present invention is introduced next.
[0162] It will be appreciated by those skilled in the art that various aspects of the present invention may be implemented as a system, method or program product. Therefore, various aspects of the present invention may be specifically implemented in the following forms, namely: a complete hardware implementation, a complete software implementation (including firmware, microcode, etc.), or a combination of hardware and software, which may be collectively referred to herein as a "circuit", "module" or "system".
[0163] In some possible implementations, the IoT module according to the present invention may include at least one processor and at least one computer storage medium. The computer storage medium stores program code, and when the program code is executed by the processor, the processor executes the steps of the hand joint detection method according to various exemplary embodiments of the present invention described above in this specification. For example, the processor may execute the following steps: Figure 1 Steps 101-105 shown in FIG.
[0164] Refer to the following Fig.11 The electronic device 1100 according to this embodiment of the present invention is described. Fig.11 The electronic device 1100 shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention.
[0165] like Fig.11 As shown, the electronic device 1100 is presented in the form of a general AR device. The components of the electronic device 1100 may include but are not limited to: the at least one processor 1101, the at least one computer storage medium 1102, and a bus 1103 connecting different system components (including the computer storage medium 1102 and the processor 1101).
[0166] Bus 1103 represents one or more of several types of bus structures, including a computer storage media bus or computer storage media controller, a peripheral bus, a processor, or a local bus using any of a variety of bus architectures.
[0167] Computer storage media 1102 may include readable media in the form of volatile computer storage media, such as random access computer storage media (RAM) 1121 and / or cache storage media 1122 , and may further include read-only computer storage media (ROM) 1123 .
[0168] The computer storage medium 1102 may also include a program / utility 1125 having a set (at least one) of program modules 1124, such program modules 1124 including but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment.
[0169] The electronic device 1100 may also communicate with one or more external devices 1104 (e.g., keyboards, pointing devices, etc.), and may also communicate with one or more devices that enable a user to interact with the electronic device 1100, and / or communicate with any device that enables the electronic device 1100 to communicate with one or more other AR devices (e.g., routers, modems, etc.). Such communication may be performed through an input / output (I / O) interface 1105. In addition, the electronic device 1100 may also communicate with one or more networks (e.g., a local area network (LAN), a wide area network (WAN), and / or a public network, such as the Internet) through a network adapter 1106. As shown, the network adapter 1106 communicates with other modules for the electronic device 1100 through a bus 1103. It should be understood that, although not shown in the figure, other hardware and / or software modules may be used in conjunction with the electronic device 1100, including but not limited to: microcode, device drivers, redundant processors, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.
[0170] In some possible embodiments, various aspects of a method for detecting hand joints provided by the present invention may also be implemented in the form of a program product, which includes program code. When the program product runs on a computer device, the program code is used to enable the computer device to execute the steps of the method for detecting hand joints according to various exemplary embodiments of the present invention described above in this specification.
[0171] Obviously, those skilled in the art can make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these modifications and variations.
Claims
1. A method for detecting hand joint points, characterized in that: The method comprises: For any frame of hand image, hand detection is performed on the any frame of hand image according to the image position coordinates of the target hand joint points in the previous frame of hand image of the any frame of hand image, so as to obtain the position of the hand area in the any frame of hand image, wherein the target hand joint points are the joint points in the palm area; Cropping the any frame of hand image according to the position of the hand region in the any frame of hand image to obtain a hand region image; Performing joint point detection on the hand region image using a pre-trained hand joint point detection model to obtain image position coordinates of each hand joint point in the hand region image; Determining the motion state of the hand in the hand area image based on the image position coordinates of the first joint point specified in the hand area image and the image position coordinates of the first joint point in each target hand image, wherein each target hand image is a hand image frame of a specified number of frames before any one frame of the hand image, and the motion state is used to indicate whether the hand is stationary; Based on the movement state of the hand in the hand area image, the target position coordinates of each hand joint point in any frame of the hand image are obtained.
2. The method according to claim 1, characterized in that Before performing joint point detection on the hand region image using the pre-trained hand joint point detection model, the method further includes: Obtaining the rotation matrix of the hand region image through the image position coordinates of each hand joint point in the previous frame of the hand image and a preset standard vector; wherein the standard vector is used to represent a vector that is perpendicular to the horizontal axis in the image coordinate system and has a direction opposite to that of the vertical axis; The hand region image is rotated using the rotation matrix to obtain a rotated hand region image, and the rotated hand region image is determined as the hand region image.
3. The method according to claim 2, characterized in that The step of obtaining the rotation matrix of the hand region image by using the image position coordinates of each hand joint point in the previous frame of the hand image and a preset standard vector includes: Obtaining a direction vector corresponding to the target hand image according to the image position coordinates of a third joint point specified in each hand joint point in the previous frame of the hand image, wherein the third joint point includes a joint point at which the middle finger and the palm are connected and a root node of the palm; Based on the direction vector and the standard vector, obtaining a vector angle between the direction vector and the standard vector; Using the vector angle, obtaining rotation parameters of the target hand image, wherein the rotation parameters include a rotation angle and a rotation direction; The rotation matrix of the target hand image is obtained through the rotation parameters of the target hand image.
4. The method according to claim 1, characterized in that: The step of performing hand detection on the arbitrary frame of hand image according to the image position coordinates of the target hand joint point in the previous frame of hand image of the arbitrary frame of hand image to obtain the position of the hand area in the arbitrary frame of hand image includes: Obtaining the position of the palm area in any one frame of the hand image according to the image position coordinates of each target hand joint point in the previous frame of the hand image; The position of the palm area in the arbitrary frame of hand image is enlarged to obtain the position of the hand area in the arbitrary frame of hand image.
5. The method according to claim 4, characterized in that The step of obtaining the position of the palm area in any one frame of the hand image according to the image position coordinates of the target joint point in each hand joint point of the previous frame of the hand image comprises: Obtaining the horizontal coordinate of the center point of the palm area and the width of the palm area according to the horizontal coordinate with the largest value and the horizontal coordinate with the smallest value in the image position coordinates of each target joint point; and Obtaining the ordinate of the center point of the palm area and the height of the palm area according to the ordinate with the largest value and the ordinate with the smallest value in the image position coordinates of each target joint point; The position of the palm area is obtained based on the abscissa of the center point, the ordinate of the center point, the width, and the height.
6. The method according to claim 4, characterized in that The step of enlarging the position of the palm region in any one frame of the hand image to obtain the position of the hand region in any one frame of the hand image includes: Determining the image position coordinates of the second joint point specified in the previous frame of the hand image as the position coordinates of the center point of the hand area; and The width of the palm area is enlarged by a specified multiple to obtain the width of the hand area; and the height of the palm area is enlarged by the specified multiple to obtain the height of the hand area; The position of the hand region in any frame of the hand image is obtained according to the position coordinates of the center point of the hand region, the width of the hand region and the height of the hand region.
7. The method according to claim 1, characterized in that The determining the motion state of the hand in the hand region image based on the image position coordinates of the first joint point specified in the hand region image and the image position coordinates of the first joint point in each target hand image includes: For any one frame of the target hand images, based on the image position coordinates of the first joint point in the any one frame of the target hand image and the image position coordinates of the first joint point in the hand region image, obtain the Euclidean distance between the hand region image and the any one frame of the target hand image; Comparing the average value of the Euclidean distances between the hand region image and each target hand image with a preset threshold; If the average value is not greater than the preset threshold, it is determined that the movement state of the hand in the hand area image is a stationary state; or, If the average value is greater than the preset threshold, it is determined that the movement state of the hand in the hand area image is a non-stationary state.
8. The method according to claim 1, characterized in that The step of obtaining the target position coordinates of each hand joint point in any frame of the hand image based on the motion state of the hand in the hand region image includes: If the motion state of the hand in the hand area image is a stationary state, the image position coordinates of each hand joint point in the hand image corresponding to the last time the hand was not in a stationary state are determined as the target position coordinates of each hand joint point in any frame of the hand image; or, If the movement state of the hand in the hand area image is a non-static state, the image position coordinates of each hand joint point in the hand area image are determined as the target position coordinates of each hand joint point in any frame of the hand image.
9. An electronic device, characterized in that: comprising a processor and a memory, wherein the processor and the memory are connected via a bus; The memory stores a computer program, and the processor is configured to perform the following operations based on the computer program: For any frame of hand image, hand detection is performed on the any frame of hand image according to the image position coordinates of the target hand joint points in the previous frame of hand image of the any frame of hand image, so as to obtain the position of the hand area in the any frame of hand image, wherein the target hand joint points are the joint points in the palm area; Cropping the any frame of hand image according to the position of the hand region in the any frame of hand image to obtain a hand region image; Performing joint point detection on the hand region image using a pre-trained hand joint point detection model to obtain image position coordinates of each hand joint point in the hand region image; Determining the motion state of the hand in the hand area image based on the image position coordinates of the first joint point specified in the hand area image and the image position coordinates of the first joint point in each target hand image, wherein each target hand image is a hand image frame of a specified number of frames before any one frame of the hand image, and the motion state is used to indicate whether the hand is stationary; Based on the movement state of the hand in the hand area image, the target position coordinates of each hand joint point in any frame of the hand image are obtained.
10. The electronic device according to claim 9, characterized in that: The processor is further configured to: Before performing joint point detection on the hand region image using the pre-trained hand joint point detection model, a rotation matrix of the hand region image is obtained through the image position coordinates of each hand joint point in the previous frame of the hand image and a preset standard vector; wherein the standard vector is used to represent a vector that is perpendicular to the horizontal coordinate axis in the image coordinate system and has a direction opposite to that of the vertical coordinate; The hand region image is rotated using the rotation matrix to obtain a rotated hand region image, and the rotated hand region image is determined as the hand region image.