Methods, apparatus, computer equipment, and readable storage media for detecting finger pointing
By estimating hand pose and detecting key points, finger pose data is calculated to determine the pointing pose of a single finger and its pointing position on the display screen. This solves the problems of large data requirements and insufficient robustness in existing gesture recognition methods, and achieves highly accurate gesture interaction.
Patent Information
- Application Number
- CN202411703184.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-11-26
- Publication Date
- 2026-05-26
AI Technical Summary
Existing gesture recognition methods suffer from problems such as large data requirements, low robustness and accuracy. In particular, when recognizing new gestures, the model needs to be retrained, and the method of using a single judgment index is not robust and accurate enough.
Hand pose estimation is performed by acquiring the image to be detected, calculating the coordinates of key points of the hand, determining whether the hand pose is a single finger pointing pose, and calculating its pointing position on the display screen based on the finger pose data. The spatial orientation and pointing position of the finger are obtained by using a skeletal key point detection model and a depth detection model.
It enables accurate determination of the user's finger position on the display screen without relying on deep learning, improving the accuracy and reliability of user interaction with the display screen and reducing the data requirements for model training.
Smart Images

Figure CN122086227A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of human-computer interaction, and in particular to a method, apparatus, computer device, and readable storage medium for detecting finger pointing. Background Technology
[0002] Currently, hand gesture interaction technology is widely used in the field of human-computer interaction. Single-finger pointing gestures, as a common type of gesture, can achieve an intuitive and natural interactive experience. However, gesture recognition methods in related technologies often have some limitations.
[0003] A common approach is to use deep learning for gesture classification, but this requires a large amount of data for model training, and the model may need to be retrained for recognizing new gestures. Another approach is to define gestures based on two-dimensional angular constraints of hand key points, but this method uses only a single criterion, resulting in lower robustness and accuracy. Summary of the Invention
[0004] In view of this, this application provides a method, apparatus, computer device, and readable storage medium for detecting finger pointing, which enables accurate determination of the user's finger pointing at the display screen and improves the accuracy of user interaction with the display screen.
[0005] In a first aspect, embodiments of this application provide a method for detecting finger pointing, including:
[0006] Acquire the image to be detected, and perform hand pose estimation on the image to obtain the coordinates of key hand points in the image;
[0007] Calculate the finger posture data of the hand based on the coordinates of the key points of the hand, and determine whether the hand posture is a single finger pointing posture based on the finger posture data;
[0008] When the hand is in a single-finger pointing posture, the spatial orientation of the single finger is calculated based on the coordinates of the key points of the hand, and the pointing position of the single finger on the display screen is determined based on the spatial orientation.
[0009] Secondly, embodiments of this application provide a finger pointing detection device, comprising:
[0010] The coordinate determination module is used to acquire the image to be detected and to perform hand pose estimation on the image to be detected to obtain the coordinates of the key hand points in the image to be detected.
[0011] The posture determination module is used to calculate the finger posture data of the hand based on the coordinates of the key points of the hand, and to determine whether the posture of the hand is a single finger pointing posture based on the finger posture data;
[0012] The pointing determination module is used to calculate the spatial orientation of the single finger pointing based on the coordinates of the key points of the hand when the hand is in a single finger pointing posture, and to determine the pointing position of the single finger on the display screen based on the spatial orientation.
[0013] Thirdly, embodiments of this application provide a computer device including a processor and a memory coupled together. The memory stores a program or instructions that can run on the processor, and when the program or instructions are executed by the processor, they implement the steps of the method as described in the first aspect.
[0014] Fourthly, embodiments of this application provide a readable storage medium on which a program or instructions are stored, which, when executed by a processor, implement the steps of the method as described in the first aspect.
[0015] Fifthly, embodiments of this application provide a chip including a processor and a communication interface, the communication interface being coupled to the processor, the processor being used to run programs or instructions to implement the method as described in the first aspect.
[0016] In a sixth aspect, embodiments of this application provide a computer program product stored in a storage medium, which is executed by at least one processor to implement the method as described in the first aspect.
[0017] In this embodiment, hand pose estimation is performed on the image to be detected to obtain the coordinates of key hand points in the image. Then, based on these coordinates, finger pose data, such as the angles and distances between key finger bones, are calculated. Finally, hand pose is determined based on this data to ascertain whether the hand pose is a single-finger pointing pose. Further, when the hand pose is a single-finger pointing pose, the spatial orientation of the single finger and its pointing position on the display screen are calculated based on the hand key point coordinates. This embodiment enables hand pose estimation and finger pointing determination, accurately determining the pointing position of the user's finger on the display screen and improving the accuracy of user interaction with the display screen.
[0018] The above description is only an overview of the technical solution of this application. In order to better understand the technical means of this application and to implement it in accordance with the contents of the specification, and to make the above and other objects, features and advantages of this application more obvious and understandable, specific embodiments of this application are given below. Attached Figure Description
[0019] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:
[0020] Figure 1 A flowchart illustrating a finger pointing detection method according to an embodiment of this application is shown;
[0021] Figure 2 A schematic diagram of the key skeletal points of the hand in an embodiment of this application is shown;
[0022] Figure 3 A schematic diagram illustrating the transformation relationship between the pixel coordinate system and the image coordinate system in an embodiment of this application is shown;
[0023] Figure 4 A schematic diagram illustrating the transformation relationship between the image coordinate system and the camera coordinate system in an embodiment of this application is shown;
[0024] Figure 5 A schematic diagram illustrating the transformation relationship between the camera coordinate system and the world coordinate system in an embodiment of this application is shown;
[0025] Figure 6 A schematic diagram showing the intersection of the extension line of the spatial orientation of an embodiment of this application and the plane where the display screen is located is shown;
[0026] Figure 7 A structural block diagram of a finger pointing detection device according to an embodiment of this application is shown;
[0027] Figure 8 A structural block diagram of a computer device according to an embodiment of this application is shown. Detailed Implementation
[0028] The technical solutions of the embodiments of this application will be clearly described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application are within the scope of protection of this application.
[0029] The terms "first," "second," etc., used in the specification and claims of this application are used to distinguish similar objects and not to describe a specific order or sequence. It should be understood that such use of data can be interchanged where appropriate so that embodiments of this application can be implemented in orders other than those illustrated or described herein, and the objects distinguished by "first," "second," etc., are generally of the same class and the number of objects is not limited; for example, a first object can be one or more. Furthermore, in the specification and claims, "and / or" indicates at least one of the connected objects, and the character " / " generally indicates that the preceding and following objects are in an "or" relationship.
[0030] The following description, in conjunction with the accompanying drawings, details the finger pointing detection method, apparatus, computer device, and readable storage medium provided in this application through specific embodiments and application scenarios.
[0031] This application provides a method for detecting finger pointing, such as... Figure 1 As shown, the method includes:
[0032] Step 101: Obtain the image to be detected and perform hand pose estimation on the image to be detected to obtain the coordinates of the key hand points in the image to be detected;
[0033] Step 102: Calculate the finger posture data of the hand based on the coordinates of the key points of the hand, and determine whether the hand posture is a single finger pointing posture based on the finger posture data;
[0034] Step 103: When the hand posture is a single finger pointing posture, calculate the spatial orientation of the single finger pointing based on the coordinates of the key points of the hand, and determine the pointing position of the single finger on the display screen based on the spatial orientation.
[0035] In this embodiment, hand pose estimation is performed on the image to be detected to obtain the coordinates of key hand points in the image. The skeletal key points of the hand are as follows: Figure 2 As shown, 21 key points of the hand bones, including the wrist, thumb, index finger, middle finger, ring finger, and little finger, are identified. Based on the coordinates of these key points, finger posture data, such as the angles and distances between these key points, are calculated. This data is then used to determine whether the hand posture is a single-finger pointing posture, where a single finger includes any one of the five fingers. Furthermore, when the hand posture is a single-finger pointing posture, the spatial orientation of the single finger and its pointing position on the display screen are calculated based on the coordinates of the key points.
[0036] The embodiments of this application can realize hand posture estimation and finger pointing judgment to accurately determine the pointing position of the user's finger on the display screen, thereby improving the accuracy of user interaction with the display screen.
[0037] Compared to related technologies, this approach eliminates the need for deep learning for gesture classification, avoids the need for extensive model training, and improves the reliability of hand posture judgment by using multiple indicators such as the angles and distances between key points of the finger bones.
[0038] The solutions in this application are applicable to different application scenarios, including human-computer interaction, virtual reality, and augmented reality.
[0039] In one embodiment of this application, hand pose estimation is performed on the image to be detected to obtain the coordinates of the hand key points in the image to be detected, including: hand detection is performed on the image to be detected to obtain a hand image, and skeletal key point detection is performed on the hand image to obtain the coordinates of the hand key points in the image to be detected.
[0040] The process involves detecting skeletal keypoints in a hand image to obtain the coordinates of these keypoints. This includes: inputting the hand image into a skeletal keypoint detection model to obtain the 2D pixel coordinates of each skeletal keypoint in the pixel coordinate system and the 3D coordinates in the camera coordinate system, where the 3D coordinates are relative to the root node of the middle finger; inputting the hand image into a depth detection model to obtain the absolute depth of the root node of the middle finger in the camera coordinate system, and determining the absolute depth of each skeletal keypoint in the camera coordinate system based on the absolute depth of the root node and the 3D coordinates; performing coordinate transformation based on the 2D pixel coordinates and 3D coordinates to obtain the image coordinates of each skeletal keypoint in the image coordinate system; performing coordinate transformation based on the image coordinates and the absolute depth in the camera coordinate system to obtain the camera coordinates of each skeletal keypoint in the camera coordinate system; and performing coordinate transformation based on the camera coordinates to obtain the hand keypoint coordinates in the world coordinate system.
[0041] In this embodiment, hand detection is performed on the image to be detected to identify the hand image. For example, a trained hand detection model, YOLOv7, is used to detect the hand and obtain the hand image. The image to be detected can be a single RGB image. The hand image is then input into a pre-trained skeletal keypoint detection model, which can be a Minimal-Hand model. The model outputs the 2D pixel coordinates (u, v) of each skeletal keypoint of the hand in the pixel coordinate system and the 3D coordinates (dx, dy, dz) in the camera coordinate system.
[0042] Simultaneously, the hand image is input into the pre-trained depth detection model RootNet, which outputs the absolute depth z_r of the middle finger root node in the camera coordinate system. The middle finger root node is... Figure 2 The skeletal key point 9 is shown.
[0043] Using the above model, we can obtain the 2D pixel coordinates (u, v) in the pixel coordinate system and the 3D coordinates (dx, dy, dz) in the camera coordinate system, as well as the absolute depth z_r of the middle finger root node. After obtaining the absolute depth z_r of the middle finger root node, we can calculate the absolute depth Z of other skeletal keypoints. C The absolute depth Z in the z-direction of all skeletal keypoints in the camera coordinate system. C See formula (1):
[0044] Z C =z_r+l ref ×dz (1)
[0045] Among them, l ref The reference bone length is known. u and v are known 2D skeletal keypoints in the pixel plane. By transforming the pixel coordinates to the image coordinates, then to the camera coordinates, and finally to the world coordinates, the coordinates of all 3D hand keypoints in the world coordinates are obtained. Specifically:
[0046] Pixel coordinate system to image coordinate system: (u,v) represents pixel coordinates, (x,y) represents image coordinates. The conversion from pixel coordinates (u,v) to image coordinates (x,y) includes not only numerical values but also unit conversion, i.e., from pixels / px to millimeters / mm. The conversion formula is as shown in formula (2):
[0047]
[0048] Where dx and dy represent the lengths of a single pixel in the x and y directions, respectively, and u0 and v0 represent the pixel coordinates of the principal point, which is the intersection of the camera's optical axis and the imaging plane. First, the pixels are moved to the principal point position, and then the image coordinates are obtained by multiplying the number of pixels by the length of a single pixel. Figure 3 As shown, formula (2) can be transformed into formula (3):
[0049]
[0050] Image coordinate system to camera coordinate system: From Figure 4 As can be seen from this, triangle O C O xy p and triangle O C O P P is similar, f is the focal length, and we can obtain formula (4):
[0051]
[0052] This leads to formula (5):
[0053]
[0054] Extract Z C The transformation formula from image coordinate system to camera coordinate system is obtained as shown in formula (6):
[0055]
[0056] Camera coordinate system to world coordinate system conversion: such as Figure 5 As shown, the 3D hand keypoint coordinates [Xw, Yw, Zw] of all hand skeleton keypoints in the real scene can be obtained. The following is the calculation process for the conversion:
[0057] The transformation from the camera coordinate system to the world coordinate system typically involves the camera's intrinsic and extrinsic parameter matrices. Camera calibration allows us to obtain the specific values of these intrinsic and extrinsic parameters.
[0058] Camera intrinsic parameter matrix: The intrinsic parameter matrix is usually represented as K, which contains the camera's focal length f. x f y and principal point position c x c y The parameters are shown in formula (7):
[0059]
[0060] Camera extrinsic matrix: The extrinsic matrix represents the position and orientation of the camera in the world coordinate system. It is usually represented as [R|t], where R is the rotation matrix and t is the translation vector, as shown in formula (8):
[0061]
[0062] For a point (Xc, Yc, Zc) in the camera coordinate system, its coordinates (Xw, Yw, Zw) in the world coordinate system can be calculated using formula (9). Formula (9) transforms the point in the camera coordinate system to the world coordinate system using the rotation matrix R and the translation vector t:
[0063]
[0064] By using the above method, the coordinates of each skeletal key point in the pixel coordinate system are detected, and then the coordinates in the pixel coordinate system are converted into coordinates in the world coordinate system to obtain each skeletal key point in the world coordinate system, providing an accurate data foundation for subsequent determination of the finger's pointing direction and position.
[0065] In one embodiment of this application, the skeletal keypoint detection model includes a keypoint detection sub-model and an inverse kinematics sub-model;
[0066] The hand image is input into the skeletal keypoint detection model to obtain the 2D pixel coordinates of each skeletal keypoint in the pixel coordinate system and the 3D coordinates in the camera coordinate system. This includes: inputting the hand image into the keypoint detection sub-model, outputting the 2D and 3D skeletal keypoint positions of each skeletal keypoint in the hand image, and inputting the 3D skeletal keypoint positions into the inverse kinematics sub-model to output rotation information; and determining the 2D pixel coordinates of each skeletal keypoint in the pixel coordinate system and the 3D coordinates in the camera coordinate system based on the 2D skeletal keypoint positions, 3D skeletal keypoint positions, and rotation information.
[0067] In this embodiment, the skeletal keypoint detection model includes two modules: a keypoint detection sub-model (DetNet) and an inverse kinematics sub-model (IKNet). DetNet predicts the 2D and 3D positions of skeletal keypoints, while IKNet predicts the rotation of the keypoints, addressing the inverse kinematics problem. The predicted 3D keypoint positions are input, and the rotation information is regressed. After passing through the skeletal keypoint detection model, the 2D pixel coordinates (u, v) in the pixel coordinate system and the 3D coordinates (dx, dy, dz) in the camera coordinate system are regressed. The 3D coordinates are relative to the root node 9 of the middle finger (e.g., ...). Figure 2 The scale-normalized relative coordinates (as shown) are given, with the normalization parameter being the known reference bone length l. ref .
[0068] In this embodiment, based on a single RGB image, a skeletal keypoint detection model is used to regress the 2D pixel coordinates in the pixel coordinate system and the 3D coordinates in the camera coordinate system. This is then compared with depth cameras to obtain depth information, which can greatly reduce application costs.
[0069] In one embodiment of this application, calculating finger posture data of the hand based on the coordinates of key points of the hand, and determining whether the hand posture is a single finger pointing posture based on the finger posture data, includes: calculating finger posture data of the hand based on the coordinates of key points of the hand, wherein the finger posture data includes finger length, distance from fingertip to the center point of the palm, and angle between the fingertip of the index finger and the middle segment of the index finger; determining whether the finger length, distance from fingertip to the center point of the palm, and angle between the fingertip of the index finger and the middle segment of the index finger all meet the corresponding preset conditions; if the finger length, distance from fingertip to the center point of the palm, and angle between the fingertip of the index finger and the middle segment of the index finger all meet the corresponding preset conditions, then determining that the hand posture is a single finger pointing posture.
[0070] In this embodiment, taking the index finger as an example, the finger length, the distance from the fingertip to the center of the palm, and the angle between the fingertip and the middle section of the index finger are calculated based on the coordinates of key points on the hand. Then, by judging whether these parameters meet the preset conditions, it is possible to accurately determine whether the hand posture is a single index finger pointing posture. A single index finger pointing posture refers to a gesture that uses only the index finger to point in any direction.
[0071] In one embodiment of this application, the calculation of finger length includes: calculating the Euclidean distance between the fingertip coordinates and the finger base coordinates as the finger length; the calculation of the distance from the fingertip to the center point of the palm includes: calculating the average of the hand key point coordinates of each skeletal key point of the hand as the coordinates of the center point of the palm, and calculating the Euclidean distance between the coordinates of the center point of the palm and the fingertip coordinates as the distance from the fingertip to the center point of the palm; the calculation of the angle between the fingertip and the middle segment of the index finger includes: calculating a first vector from the fingertip to the root node of the index finger, and calculating a second vector from the center point of the palm to the fingertip; calculating the vector angle between the first vector and the second vector, and obtaining the radian of the vector angle through the inverse cosine function, and converting the radian into an angle as the angle between the fingertip and the middle segment of the index finger;
[0072] The first preset condition corresponding to finger length includes that the index finger is longer than the other fingers; the second preset condition corresponding to the distance from the fingertip to the center of the palm includes that the distance from the fingertip to the center of the palm is longer than the distance from the fingertip to the center of the palm of the other fingers; the third preset condition corresponding to the angle between the fingertip and the middle section of the index finger includes that the angle between the fingertip and the middle section of the index finger is within the preset angle range.
[0073] In this embodiment, by calculating finger posture data such as finger length, the distance from the fingertip to the center of the palm, and the angle between the fingertip and the middle section of the index finger, a comprehensive judgment is made as to whether the hand posture is a gesture pointing with a single index finger. The specific calculation method for finger posture data includes:
[0074] Calculate finger length:
[0075] Euclidean distance is used to measure the straight-line distance between two points. For two points (x1, y1, z1) and (x2, y2, z2) in three-dimensional space, the formula for calculating the Euclidean distance is:
[0076]
[0077] Finger length is defined as the Euclidean distance between the two key skeletal points of the fingertip and the base of the finger. The finger length of each finger can be obtained according to formula (10).
[0078] Calculate the distance from the fingertip to the center of the palm:
[0079] The coordinates of the center point of the palm are the average of the coordinates of the key points of each bone in the hand. After calculating the coordinates of the center point of the palm, the Euclidean distance between the fingertip and the center point of the palm is calculated according to the formula (10), and the distance between the fingertip and the center point of the palm is obtained.
[0080] Calculate the angle between the tip of the index finger and the middle section of the index finger:
[0081] Calculate the vector `finger_vector` from the fingertip (`finger_tip`) to the root node (`finger_tip_5`) of the index finger. The root node of the index finger is as follows: Figure 2 The skeletal keypoint 5 shown is the base of the index finger, and the vector finger_direction from the center point of the palm w_tip to the fingertip of the index finger is calculated using the dot product and norm of the vectors. The formula is as follows:
[0082]
[0083] Next, the arccosine function is used to obtain the radian value of the vector angle cos_angle. Finally, this radian value is converted into an angle, which is used as the angle between the fingertip and the middle of the index finger. Since the radian value obtained by the arccosine function is in the range of 0-π, while it is actually required to be in the range of 0-180 degrees, the angle needs to be adjusted to ensure that it is within the range of 0-180 degrees.
[0084] Determining whether a hand posture is a single-finger pointing posture involves checking if the finger length meets a first preset condition, calculating if the distance from the fingertip to the center of the palm meets a second preset condition, and calculating if the angle between the fingertip and the middle section of the index finger meets a third preset condition. The first preset condition is that the index finger length is greater than the length of the other fingers, ensuring the index finger is extended to a longer distance. The second preset condition is that the distance from the fingertip to the center of the palm is greater than the distance from the fingertip to the center of the palm of the other fingers, ensuring the index finger is the furthest finger. The third preset condition is that the angle between the fingertip and the middle section of the index finger is within a preset angle range, which can be set between 0 degrees and 130 degrees.
[0085] When the length of the index finger is greater than the length of the other fingers, the distance from the tip of the index finger to the center of the palm is greater than the distance from the tips of the other fingers to the center of the palm, and the angle between the tip of the index finger and the middle section of the index finger is within the preset angle range, the hand posture is determined to be a single index finger pointing posture.
[0086] In this embodiment, by combining the angles and distances between key points of the finger bones, it is possible to accurately determine whether a hand gesture is a gesture pointing with a single index finger.
[0087] In one embodiment of this application, calculating the spatial orientation of a single finger based on the coordinates of key points on the hand includes: calculating the angle of a single index finger pointing in the horizontal direction based on the y-axis coordinate of the index fingertip and the y-axis coordinate of the index finger root node, and calculating the angle of a single index finger pointing in the vertical direction based on the x-axis coordinate of the index fingertip and the x-axis coordinate of the index finger root node.
[0088] In this embodiment, the spatial orientation of a single index finger is calculated using the coordinates of key points on the hand, with the z-direction as the reference direction. Specifically:
[0089] Calculate the angle in the horizontal direction (x-direction):
[0090] tan_x=(xyz[8][1]-xyz[5][1]) / (xyz[8][2]-xyz[5][2]) (12)
[0091] angle_x=atan(tan_x)×180 / π (13)
[0092] Where xyz[i][j] represents the j-axis coordinate of the i-th skeletal key point, i = 0, 1, ... 20, j = 0, 1, 2, j = 0 represents the x-axis coordinate, j = 1 represents the y-axis coordinate, j = 2 represents the z-axis coordinate, xyz[8][1] is the y-axis coordinate of the fingertip of the index finger, xyz[5][1] is the y-axis coordinate of the root node of the index finger, xyz[8][2] is the z-axis coordinate of the fingertip of the index finger, xyz[5][2] is the z-axis coordinate of the root node of the index finger. The arctangent function atan(tan_x) is used to calculate the radians, and then the radians are converted into degrees.
[0093] Calculate the angle in the vertical direction (y-direction):
[0094] tan_y=(xyz[8][0]-xyz[5][0]) / (xyz[8][2]-xyz[5][2]) (14)
[0095] angle_y=atan(tan_y)×180 / π (15)
[0096] Where xyz[8][0] is the x-axis coordinate of the fingertip of the index finger, and xyz[5][0] is the x-axis coordinate of the root node of the index finger. The arctangent function atan(tan_y) is used to calculate the radian, and then the radian is converted into the angle.
[0097] The above method enables accurate calculation of the spatial orientation of a single finger.
[0098] In one embodiment of this application, determining the pointing position of a single finger on the display screen based on spatial orientation includes: calculating the intersection point of the extension line of the spatial orientation and the plane where the display screen is located, and determining whether the intersection point is within the screen coordinate range of the display screen in the world coordinate system; if the intersection point is within the screen coordinate range, then the intersection point is taken as the pointing position of a single index finger on the display screen.
[0099] In this embodiment, the intersection of the extended line of the spatial orientation pointed to by a single finger and the display screen is obtained and displayed.
[0100] Set the screen coordinates range of the display screen in the world coordinate system as (x = [-100mm, 100mm], y = [-100mm, 100mm], z = 0).
[0101] Calculate the intersection point of the extended line of the spatial orientation with the plane where the display screen is located. Perform intersection point calculations for lines and planes within the same coordinate system (three-dimensional space). In three-dimensional space, if a line is not parallel to a plane, an intersection point exists; here, the vector method is used to find this intersection point. For example... Figure 6 As shown, we have the following vectors and points: the normal vector V of the plane containing the display screen. P (x,y,z), a point P on the plane P (x, y, z) is a point P on line L. L (x, y, z), the intersection point P of the line L and the plane where the display screen is located. X Vector V L =P X -P L sum vector P P -P L In normal vector V P The projections on the surface are equal, as shown in formula (16), from which k can be calculated, combined with P. X It must be on a straight line, see formula (17), to obtain the intersection point P. X Coordinates. If V L ·V P =0 has no solution, meaning line L is perpendicular to the normal vector V. P That is, the line is parallel to the plane.
[0102] k×V L ·V P =(PP -P L )·V P (16)
[0103] P X =P L +k×V L (17)
[0104] Determine the intersection point P X If the index finger is within the screen coordinate range, then the index finger is pointing at the display screen.
[0105] The above method enables accurate determination of the pointing position of a single finger, thus determining whether the index finger is pointing at the display screen.
[0106] As a specific implementation of the above-mentioned finger pointing detection method, this application provides a finger pointing detection device. For example... Figure 7 As shown, the finger pointing detection device 700 includes: a coordinate determination module 701, a posture judgment module 702, and a pointing determination module 703.
[0107] The coordinate determination module 701 is used to acquire the image to be detected and to perform hand pose estimation on the image to be detected, thereby obtaining the coordinates of the key hand points in the image to be detected.
[0108] The posture judgment module 702 is used to calculate the finger posture data of the hand based on the coordinates of the key points of the hand, and to determine whether the hand posture is a single finger pointing posture based on the finger posture data.
[0109] The pointing determination module 703 is used to calculate the spatial orientation of the single finger pointing based on the coordinates of key points of the hand when the hand is in a single finger pointing posture, and to determine the pointing position of the single finger on the display screen based on the spatial orientation.
[0110] In this embodiment, hand pose estimation is performed on the image to be detected to obtain the coordinates of key hand points in the image. Then, based on these coordinates, finger pose data such as the angles and distances between key finger bones are calculated. Finally, hand pose is determined based on this data to ascertain whether the hand pose is a single-finger pointing pose. Further, when the hand pose is a single-finger pointing pose, the spatial orientation of the single finger and its pointing position on the display screen are calculated based on the hand key point coordinates. This embodiment enables hand pose estimation and finger pointing determination, accurately determining the pointing position of the user's finger on the display screen and improving the accuracy of user interaction with the display screen.
[0111] Furthermore, the coordinate determination module 701 is specifically used for:
[0112] Hand detection is performed on the image to be detected to obtain a hand image;
[0113] The hand image is input into the skeletal keypoint detection model to obtain the 2D pixel coordinates of each skeletal keypoint in the pixel coordinate system and the 3D coordinates in the camera coordinate system. The 3D coordinates are relative to the root node of the middle finger of the hand.
[0114] The hand image is input into the depth detection model to obtain the absolute depth of the middle finger root node in the camera coordinate system. Based on the absolute depth and 3D coordinates of the middle finger root node, the absolute depth of each skeletal key point in the camera coordinate system is determined.
[0115] Coordinate transformation is performed based on 2D pixel coordinates and 3D coordinates to obtain the image coordinates of each skeletal keypoint in the image coordinate system. Coordinate transformation is performed based on the absolute depth in the image coordinate system to obtain the camera coordinates of each skeletal keypoint in the camera coordinate system. Finally, coordinate transformation is performed based on the camera coordinates to obtain the hand keypoint coordinates of each skeletal keypoint in the world coordinate system.
[0116] Furthermore, the skeletal keypoint detection model includes a keypoint detection sub-model and an inverse kinematics sub-model; the coordinate determination module 701 is specifically used for:
[0117] The hand image is input into the keypoint detection sub-model, which outputs the 2D and 3D skeletal keypoint positions of each skeletal keypoint in the hand image. The 3D skeletal keypoint positions are then input into the inverse kinematics sub-model, which outputs rotation information.
[0118] Based on the 2D skeleton keypoint positions, 3D skeleton keypoint positions, and rotation information, the 2D pixel coordinates of each skeleton keypoint in the pixel coordinate system and the 3D coordinates in the camera coordinate system are determined.
[0119] Furthermore, the posture determination module 702 is specifically used for:
[0120] The finger posture data of the hand is calculated based on the coordinates of key points of the hand. The finger posture data includes finger length, distance from fingertip to center of palm, and angle between fingertip and middle of index finger.
[0121] Determine whether the finger length, the distance from the fingertip to the center of the palm, and the angle between the fingertip and the middle of the index finger all meet the corresponding preset conditions.
[0122] If the finger length, the distance from the fingertip to the center of the palm, and the angle between the fingertip and the middle section of the index finger all meet the corresponding preset conditions, then the hand posture is determined to be a single index finger pointing posture.
[0123] Furthermore, the method for calculating finger length includes: calculating the Euclidean distance between the coordinates of the fingertip and the coordinates of the finger base, which is used as the finger length;
[0124] The method for calculating the distance from the fingertip to the center point of the palm includes: calculating the average value of the coordinates of the key points of the hand bones as the coordinates of the center point of the palm, and calculating the Euclidean distance between the coordinates of the center point of the palm and the coordinates of the fingertip as the distance from the fingertip to the center point of the palm.
[0125] The calculation method for the angle between the fingertip and the middle section of the index finger includes: calculating the first vector from the fingertip to the root node of the index finger, and calculating the second vector from the center point of the palm to the fingertip; calculating the vector angle between the first vector and the second vector, and obtaining the radian of the vector angle through the inverse cosine function, and converting the radian into an angle as the angle between the fingertip and the middle section of the index finger.
[0126] The first preset condition for finger length includes that the index finger is longer than the other fingers.
[0127] The second preset condition corresponding to the distance from the fingertip to the center of the palm includes that the distance from the fingertip to the center of the palm is greater than the distance from the fingertip to the center of the palm of the other fingers;
[0128] The third preset condition corresponding to the angle between the fingertip and the middle section of the index finger includes that the angle between the fingertip and the middle section of the index finger is within the preset angle range.
[0129] Furthermore, the pointing determination module 703 is specifically used to: calculate the angle of a single index finger pointing in the horizontal direction based on the y-axis coordinate of the index finger tip and the y-axis coordinate of the index finger root node, and calculate the angle of a single index finger pointing in the vertical direction based on the x-axis coordinate of the index finger tip and the x-axis coordinate of the index finger root node.
[0130] Furthermore, the pointer to the determination module 703 is specifically used for:
[0131] Calculate the intersection point of the extension line of the spatial orientation and the plane where the display screen is located, and determine whether the intersection point is within the screen coordinate range of the display screen in the world coordinate system;
[0132] If the intersection point is within the screen coordinate range, then the intersection point will be used as the pointing position of a single index finger on the display screen.
[0133] The finger-pointing detection device 700 in this embodiment can be a computer device or a component within the computer device, such as an integrated circuit or a chip. The computer device can be a terminal or other devices besides a terminal. For example, the computer device can be a mobile phone, tablet computer, laptop computer, PDA, in-vehicle device, mobile internet device (MID), augmented reality (AR) / virtual reality (VR) device, robot, wearable device, ultra-mobile personal computer (UMPC), netbook, or personal digital assistant (PDA), etc. It can also be a server, network attached storage (NAS), personal computer (PC), television (TV), ATM, or self-service machine, etc. This embodiment does not specifically limit the device.
[0134] The finger pointing detection device 700 provided in this application embodiment can achieve Figure 1 The various processes implemented in the implementation of the finger pointing detection method are not described in detail here to avoid repetition.
[0135] This application also provides a computer device, such as... Figure 8 As shown, the computer device 800 includes a processor 801 and a memory 802. The memory 802 stores a program or instructions that can run on the processor 801. When the program or instructions are executed by the processor 801, they implement the various steps of the above-described finger pointing detection method embodiment and achieve the same technical effect. To avoid repetition, they will not be described again here.
[0136] The memory 802 can be used to store software programs and various data. The memory 802 may primarily include a first storage area for storing programs or instructions and a second storage area for storing data. The first storage area may store the operating system, application programs or instructions required for at least one function (such as sound playback, image playback, etc.). Furthermore, the memory 802 may include volatile memory or non-volatile memory, or both. The non-volatile memory may be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. Volatile memory can be random access memory (RAM), static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM), and direct memory bus RAM (DRRAM). The memory 802 in this embodiment includes, but is not limited to, these and any other suitable types of memory.
[0137] Processor 801 may include one or more processing units; optionally, processor 801 integrates an application processor and a modem processor, wherein the application processor mainly handles operations involving the operating system, user interface, and applications, and the modem processor mainly handles wireless communication signals, such as a baseband processor. It is understood that the aforementioned modem processor may also not be integrated into processor 801.
[0138] This application also provides a readable storage medium storing a program or instructions. When the program or instructions are executed by a processor, they implement the various processes of the above-described finger pointing detection method embodiments and achieve the same technical effect. To avoid repetition, they will not be described again here.
[0139] This application also provides a chip, which includes a processor and a communication interface. The communication interface and the processor are coupled. The processor is used to run programs or instructions to implement the various processes of the above-described finger pointing detection method embodiments and can achieve the same technical effect. To avoid repetition, it will not be described again here.
[0140] It should be understood that the chip mentioned in the embodiments of this application may also be referred to as a system-on-a-chip, system chip, chip system, or system-on-a-chip, etc.
[0141] This application also provides a computer program product, which is stored in a storage medium and executed by at least one processor to implement the various processes of the above-described finger pointing detection method embodiment, and can achieve the same technical effect. To avoid repetition, it will not be described again here.
[0142] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element. Furthermore, it should be noted that the scope of the methods and apparatuses in the embodiments of this application is not limited to performing functions in the order shown or discussed, but may also include performing functions substantially simultaneously or in the reverse order, depending on the functions involved. For example, the described methods may be performed in a different order than described, and various steps may be added, omitted, or combined. Additionally, features described with reference to certain examples may be combined in other examples.
[0143] The embodiments of this application have been described above with reference to the accompanying drawings. However, this application is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of this application without departing from the spirit and scope of the claims, and all of these forms are within the protection scope of this application.
Claims
1. A method for detecting finger pointing, characterized in that, include: Acquire the image to be detected, and perform hand pose estimation on the image to obtain the coordinates of key hand points in the image; Calculate the finger posture data of the hand based on the coordinates of the key points of the hand, and determine whether the hand posture is a single finger pointing posture based on the finger posture data; When the hand is in a single-finger pointing posture, the spatial orientation of the single finger is calculated based on the coordinates of the key points of the hand, and the pointing position of the single finger on the display screen is determined based on the spatial orientation.
2. The method according to claim 1, characterized in that, The step of estimating the hand pose of the image to be detected to obtain the coordinates of key hand points in the image to be detected includes: Hand detection is performed on the image to be detected to obtain a hand image; The hand image is input into the skeletal key point detection model to obtain the 2D pixel coordinates of each skeletal key point of the hand in the pixel coordinate system and the 3D coordinates in the camera coordinate system, wherein the 3D coordinates are relative to the root node of the middle finger of the hand. The hand image is input into the depth detection model to obtain the absolute depth of the middle finger root node in the camera coordinate system, and the absolute depth of each of the skeletal key points in the camera coordinate system is determined based on the absolute depth of the middle finger root node and the 3D coordinates. Coordinate transformation is performed based on the 2D pixel coordinates and the 3D coordinates to obtain the image coordinates of each of the skeletal key points in the image coordinate system. Coordinate transformation is performed based on the image coordinates and the absolute depth in the camera coordinate system to obtain the camera coordinates of each of the skeletal key points in the camera coordinate system. Finally, coordinate transformation is performed based on the camera coordinates to obtain the hand key point coordinates of each of the skeletal key points in the world coordinate system.
3. The method according to claim 2, characterized in that, The skeletal key point detection model includes a key point detection sub-model and an inverse kinematics sub-model. The step of inputting the hand image into the skeletal keypoint detection model to obtain the 2D pixel coordinates of each skeletal keypoint of the hand in the pixel coordinate system and the 3D coordinates in the camera coordinate system includes: The hand image is input into the key point detection sub-model, which outputs the 2D and 3D skeletal key point positions of each skeletal key point in the hand image, and inputs the 3D skeletal key point positions into the inverse kinematics sub-model to output rotation information. Based on the 2D skeleton key point positions, the 3D skeleton key point positions, and the rotation information, the 2D pixel coordinates of each skeleton key point in the pixel coordinate system and the 3D coordinates in the camera coordinate system are determined.
4. The method according to claim 1, characterized in that, The step of calculating the finger posture data of the hand based on the coordinates of the key points of the hand, and determining whether the hand posture is a single finger pointing posture based on the finger posture data, includes: The finger posture data of the hand is calculated based on the coordinates of the key points of the hand, wherein the finger posture data includes finger length, distance from fingertip to the center point of the palm, and the angle between the fingertip of the index finger and the middle section of the index finger; Determine whether the length of the finger, the distance from the fingertip to the center of the palm, and the angle between the fingertip and the middle section of the index finger all meet the corresponding preset conditions; If the length of the finger, the distance from the fingertip to the center of the palm, and the angle between the fingertip and the middle section of the index finger all meet the corresponding preset conditions, then the hand posture is determined to be a single index finger pointing posture.
5. The method according to claim 4, characterized in that, The method for calculating the finger length includes: calculating the Euclidean distance between the coordinates of the fingertip and the coordinates of the finger base, and using this distance as the finger length; The method for calculating the distance from the fingertip to the center point of the palm includes: calculating the average value of the coordinates of the key points of the hand at each skeletal key point of the hand as the coordinates of the center point of the palm, and calculating the Euclidean distance between the coordinates of the center point of the palm and the coordinates of the fingertip as the distance from the fingertip to the center point of the palm. The calculation method for the angle between the fingertip and the middle segment of the index finger includes: calculating a first vector from the fingertip to the root node of the index finger, and calculating a second vector from the center point of the palm to the fingertip; calculating the vector angle between the first vector and the second vector, and obtaining the radian of the vector angle through the inverse cosine function, and converting the radian into an angle as the angle between the fingertip and the middle segment of the index finger; The first preset condition corresponding to the finger length includes that the index finger length is greater than the length of the other fingers; The second preset condition corresponding to the distance from the fingertip to the center point of the palm includes that the distance from the fingertip to the center point of the palm of the index finger is greater than the distance from the fingertip to the center point of the palm of the other fingers; The third preset condition corresponding to the angle between the fingertip and the middle section of the index finger includes that the angle between the fingertip and the middle section of the index finger is within a preset angle range.
6. The method according to any one of claims 1 to 5, characterized in that, The calculation of the spatial orientation of a single finger based on the coordinates of the key points on the hand includes: Calculate the angle of a single index finger pointing horizontally based on the y-axis coordinate of the fingertip and the y-axis coordinate of the root node of the index finger, and calculate the angle of a single index finger pointing vertically based on the x-axis coordinate of the fingertip and the x-axis coordinate of the root node of the index finger.
7. The method according to claim 6, characterized in that, Determining the pointing position of the single finger on the display screen based on the spatial orientation includes: Calculate the intersection point of the extension line of the spatial orientation and the plane where the display screen is located, and determine whether the intersection point is within the screen coordinate range of the display screen in the world coordinate system; If the intersection point is within the screen coordinate range, then the intersection point is used as the pointing position of a single index finger on the display screen.
8. A finger pointing detection device, characterized in that, include: The coordinate determination module is used to acquire the image to be detected and to perform hand pose estimation on the image to be detected to obtain the coordinates of the key hand points in the image to be detected. The posture determination module is used to calculate the finger posture data of the hand based on the coordinates of the key points of the hand, and to determine whether the posture of the hand is a single finger pointing posture based on the finger posture data; The pointing determination module is used to calculate the spatial orientation of the single finger pointing based on the coordinates of the key points of the hand when the hand is in a single finger pointing posture, and to determine the pointing position of the single finger on the display screen based on the spatial orientation.
9. A computer device, characterized in that, The device includes a processor and a memory, the processor and the memory being coupled together, the memory storing a program or instructions that run on the processor, the program or instructions being executed by the processor to implement the steps of the finger pointing orientation detection method as described in any one of claims 1 to 7.
10. A readable storage medium having a program or instructions stored thereon, characterized in that, When the program or instructions are executed by the processor, they implement the steps of the finger pointing direction detection method as described in any one of claims 1 to 7.
