Monocular camera augmented reality gesture interaction method based on key point detection

Abstract

A monocular camera augmented reality gesture interaction method based on key point detection, which is composed of the steps of obtaining an input image, detecting hand key point coordinates, determining string data, transmitting string data S, constructing a virtual hand model, storing string data S, virtual hand movement, triggering augmented reality gesture interaction, and performance evaluation. Since the application adopts the steps of determining string data, storing string data S, virtual hand movement, triggering augmented reality gesture interaction, and using a hand key point detection network, the network is applied to the monocular camera gesture tracking and recognition of augmented reality, effectively filters the features of the current frame hand, and standardizes the tracked hand key point information, so that the data is more standardized and convenient for driving the movement of the virtual world hand, and the real-time performance is high. The application has the advantages of higher real-time performance, stronger immersion, lower device cost, etc., and can perform gesture tracking and recognition in different backgrounds.

Description

Technical Field

[0001] This invention belongs to the field of extended reality interaction technology, specifically relating to a method for gesture tracking and recognition and data transmission. Background Technology

[0002] In recent years, with the gradual popularization of virtual reality devices, virtual reality interaction has become a very active topic and research hotspot. With the rapid development of the economy and society, almost everyone is under more or less pressure and anxiety. In the virtual reality world, people can temporarily detach themselves from their current emotional world and enter a brand new and novel virtual world. For the virtual reality experience, the sense of immersion in the virtual reality world is very important. Immersion largely comes from the immersiveness and naturalness of the interaction. A more realistic interactive experience can bring people a stronger sense of immersion. Therefore, research on more natural interaction has a very profound significance.

[0003] Most interactions are related to the hands. Human hand movements are primarily controlled by muscles that move the fingers, which in turn are driven by nerves that move the muscles and tendons, allowing the bones to move. The human hand bones include the carpal bones, metacarpal bones, and phalanges. The finger joints are mostly the metacarpophalangeal joints and interphalangeal joints, which mainly have flexion, extension, adduction, abduction, and rotation functions. The posture of the hand largely depends on the position of the hand joints. If the positions of the hand joints in reality can be obtained through a monocular camera and transmitted to a virtual hand in the virtual world, various forms of the virtual hand can be controlled, allowing the movement of the virtual hand to be directly controlled through the movement of the real hand.

[0004] There are many types of virtual reality interaction methods. The most traditional one is through game controllers. However, this involves pressing various buttons on the controllers, which is quite different from direct hand interaction in the real world and results in a low level of immersion. Data gloves offer a stronger sense of immersion, but they come with a higher cost of experience, making widespread adoption difficult.

[0005] Interaction based on monocular camera machine vision is a new research direction in the field of gesture interaction. It incorporates machine learning to bring it closer to its original goal—artificial intelligence. Deep learning methods include artificial neural networks, convolutional neural networks, and recurrent neural networks. Deep learning can automatically learn features from large datasets. Currently, deep learning is effectively used for gesture tracking in the field of gesture interaction.

[0006] In the field of extended display interaction technology, one urgent technical problem to be solved is to provide a gesture interaction method with higher accuracy and real-time performance, stronger immersion, and lower device cost. Summary of the Invention

[0007] The technical problem to be solved by the present invention is to overcome the shortcomings of the above-mentioned prior art and provide a monocular camera augmented reality gesture interaction method based on key point detection with higher accuracy, stronger immersion and lower equipment cost.

[0008] The technical solution adopted to solve the above-mentioned technical problems consists of the following steps.

[0009] (1) Obtain the input image

[0010] The real-time image captured by the monocular camera is used as the input image, with a width w of at least 400 pixels and a height h of at least 200 pixels.

[0011] (2) Detect the coordinates of key points on the hand

[0012] The hand keypoint coordinates of the input image are obtained using a hand keypoint detection network as follows:

[0013] 1) Set a confidence threshold θ for the hand, where θ∈(0,1). The hand detection model will start when the confidence of the hand is lower than θ.

[0014] 2) Perform hand detection on the input image, with hands numbered 0 and 1 from left to right, where n is either 0 or 1, and hand number n is denoted as H. n H n This includes left-hand label 'l' or right-hand label 'r'.

[0015] 3) Locate the detected hand and cut out the hand area.

[0016] 4) Input each hand region into the hand keypoint detection network for hand keypoint detection, and output H. n The coordinates of the 21 key points of the hand are as follows:

[0017] Key point 0 is the wrist, denoted as Key points 1 through 4 are the four joints of the thumb from the base to the tip, denoted as... Key points 5 through 8 are the four joints of the index finger from the base to the tip, denoted as... Key points 9 to 12 are the four joints of the middle finger from the base to the tip, denoted as... Key points 13 to 16 are the four joints of the ring finger from the base to the tip, denoted as... Key points 17 to 20 are the four joints of the little finger from the base to the tip, denoted as...

[0018] H n The key point j of the hand is denoted as Its x, y, and z coordinates are respectively in The x and y coordinates are their relative coordinates on the input image, and the z-axis coordinate of keypoint 0 is... It is an extremely small value, denoted as H of hand number n. n The origin of the z-axis coordinate. If the number is negative, the base of the wrist is farther away from the camera; conversely, if the number is positive, the wrist is closer to the camera.

[0019] (3) Determine the string data

[0020] H is adjusted according to the following formula n The coordinates of key hand points were transformed, and the resulting x, y, and z axis coordinates are as follows:

[0021]

[0022]

[0023]

[0024] Calculate the number of hands H using the following formula n The distance between key point 5 and key point 17 is taken as the palm width L. n :

[0025]

[0026] The string data S is obtained by the following formula:

[0027]

[0028] Where k∈{0,1,...,20}.

[0029] (4) Transmit string data S

[0030] The string data S is transmitted to the Unity engine via the User Datagram Protocol.

[0031] (5) Constructing a virtual hand model

[0032] Use the Unity engine to draw the bone positions, and add bone rotation angles and relative displacements at key points.

[0033] (6) Store string data S

[0034] 1) Hand coordinate data is passed into the Unity engine.

[0035] 2) Obtain the standardized string data S′ using the following formula:

[0036]

[0037] 3) Store standardized string data S′

[0038] Store the standardized string data S′ into the left-hand string h according to the following formula. l With the right-hand string h r middle:

[0039]

[0040]

[0041] Where N is a null value.

[0042] (7) Virtual hand movements

[0043] 1) Use commas to separate the left-hand string h l With the right-hand string h r Divide into left-hand string arrays F, each containing 64 substrings. l ′ and the right-hand string array F r ', use the float.Parse function to convert the left-hand string array F l ′ and the right-hand string array F r Convert each of the above into a left-handed floating-point array F. l With right-hand floating-point array F r :

[0044]

[0045]

[0046] Among them, the left-hand string h l When the value is N, it is not assigned to the left-hand floating-point array F. l ; right-hand string h r When the value is N, it is not assigned to the right-hand floating-point array F. r .

[0047] 2) The virtual hand as a whole moves with the camera as a sub-object of the camera.

[0048] 3) Determine the relative coordinates of key points

[0049] The relative coordinates of left-hand keypoint i with respect to left-hand keypoint 0 are determined by the following formula. Left hand width d lH :

[0050]

[0051]

[0052]

[0053]

[0054] in, For left-handed floating-point array F l element i, For left-handed floating-point array F l The 3ith element receives the x-axis coordinate of the left-hand keypoint i. For left-handed floating-point array F l Element 0 receives the x-axis coordinate of the left-hand keypoint 0. For left-handed floating-point array F l The 3i+1 element receives the y-axis coordinate of the left-hand key point i. For left-handed floating-point array F l Element 1 receives the y-coordinate of the left-hand keypoint 0. For left-handed floating-point array F l The 3i+2 element receives the z-axis coordinate of the left-hand key point i. For left-handed floating-point array F l The second element receives the z-axis coordinate of the left-hand key point 0.

[0055] The relative coordinates of right-hand keypoint i with respect to right-hand keypoint 0 are determined by the following formula. Right hand width d rH :

[0056]

[0057]

[0058]

[0059]

[0060] in, F is a right-handed floating-point array r element i, F is a right-handed floating-point array r The 3ith element receives the x-axis coordinate of the right-hand keypoint i. F is a right-handed floating-point array r Element 0 receives the x-coordinate of the right-hand keypoint 0. F is a right-handed floating-point array r The 3i+1 element receives the y-axis coordinate of the right-hand key point i. F is a right-handed floating-point array r Element 1 receives the y-coordinate of the right-hand key point 0. F is a right-handed floating-point array rThe element 3i+2 receives the z-axis coordinate of the right-hand key point i. F is a right-handed floating-point array r The second element receives the z-axis coordinate of the right-hand key point 0.

[0061] 4) Determine the scaling ratio of the virtual hand

[0062] Determine the distance d between left-hand key point 0 and left-hand key point 1 using the following formula. lR :

[0063]

[0064] Determine the distance d between right-hand key point 0 and right-hand key point 1 using the following formula. rR :

[0065]

[0066] Determine the virtual left hand scaling ratio M using the following formula. l :

[0067]

[0068] Determine the virtual right hand scaling ratio M using the following formula. r :

[0069]

[0070] Where, d lM d represents the distance between keypoints 0 and 1 of the left hand in the virtual hand model. rM This is the distance between keypoint 0 and keypoint 1 of the right hand in the virtual hand model.

[0071] 5) Determine the relative position of hand movements

[0072] The coordinates C relative to the camera are determined by the following formula when the virtual left hand moves. lx C ly C lz :

[0073]

[0074]

[0075]

[0076] Among them, D lx D ly D lz The initial position D of the virtual left hand l Coordinates relative to the camera.

[0077] The coordinates C relative to the camera during the movement of the virtual right hand are determined by the following formula. rx C ry C rz :

[0078]

[0079]

[0080]

[0081] Among them, D rx D ry D rz The initial position D of the virtual right hand r Coordinates relative to the camera.

[0082] (8) Triggering augmented reality gesture interactions

[0083] 1) When the distance between the current position of the augmented reality virtual hand and the virtual object is ≤0.3 and the distance between the virtual hand key points 8, 12, 16, 20 and the virtual hand key point 0 is ≤0.05, the gesture interaction of picking up the virtual object is triggered.

[0084] 2) When the distance between augmented reality virtual hand keypoints 8, 12, 16, 20 and virtual hand keypoint 0 is all greater than 0.05, the gesture interaction of putting down the virtual object is triggered.

[0085] (9) Performance Evaluation

[0086] The frame rate (FPS) of images processed by the keypoint detection-based monocular camera augmented reality gesture interaction method is evaluated using the following formula:

[0087]

[0088] Among them, t e It is the time when a frame is processed. s This refers to the time when processing begins for a single frame. When the number of frames processed per second (FPS) is greater than 30 frames per second, this method is considered to have high real-time performance.

[0089] In the step of detecting the coordinates of key points of the hand in (2) of the present invention, the confidence threshold for hand detection is preferably 0.68.

[0090] In the step of constructing a virtual hand model in (5) of the present invention, the rotation angle of the bones added to the key points is as follows: the rotation angle of the wrist key point 0 is 0° to 180°, the rotation angle of the other key points is 0° to 90°, and the fingers are within a radius of 12d centered on the camera. lM can move within a range, where dlM The distance between keypoint 0 and keypoint 1 of the left hand in the virtual hand model.

[0091] This invention employs a hand confidence threshold in the hand keypoint detection network step. The hand detection model restarts only when the hand confidence falls below this threshold. In the first frame of the image, hand detection is performed before keypoint detection. Because the video is continuous, the hand region can be predicted based on the coordinates of the hand keypoints in the previous frame. This hand region is then fed into the keypoint detection model for the next frame, eliminating the need for repeatedly using the hand detection model. Each frame only needs to infer the hand region from the keypoints of the previous frame before feeding it into the keypoint detection model for the next frame.

[0092] Compared with the prior art, the present invention has the following advantages:

[0093] Because this invention employs a hand keypoint detection network and applies it to augmented reality monocular camera gesture tracking and recognition, it effectively filters hand features in the current frame. Simultaneously, it standardizes the tracked hand keypoint information, making the data more standardized and easier to drive virtual hand movement, resulting in high real-time performance. This invention offers advantages such as higher real-time performance, enhanced immersion, and lower equipment cost, and can perform gesture tracking and recognition in various backgrounds. Attached Figure Description

[0094] Figure 1 This is a flowchart of Embodiment 1 of the present invention.

[0095] Figure 2 This is a diagram showing the key numbering of the hand.

[0096] Figure 3 This is the frame rate curve of hand key point tracking and detection using the method in Example 1. Detailed Implementation

[0097] The present invention will be further described in detail below with reference to the accompanying drawings and embodiments, but the present invention is not limited to the following embodiments.

[0098] Example 1

[0099] The monocular camera augmented reality gesture interaction method based on key point detection in this embodiment consists of the following steps (see...). Figure 1 ):

[0100] (1) Obtain the input image

[0101] The real-time image captured by the monocular camera is used as the input image, with a width w of at least 400 pixels and a height h of at least 200 pixels.

[0102] (2) Detect the coordinates of key points on the hand

[0103] The hand keypoint coordinates of the input image are obtained using a hand keypoint detection network as follows:

[0104] 1) Set the hand confidence threshold θ, θ∈(0,1). In this embodiment, the value of θ is 0.68. When the hand confidence is lower than θ, the hand detection model is started.

[0105] 2) Perform hand detection on the input image, with hands numbered 0 and 1 from left to right, where n is either 0 or 1, and hand number n is denoted as H. n H n This includes left-hand label 'l' or right-hand label 'r'.

[0106] 3) Locate the detected hand and cut out the hand area.

[0107] 4) Input each hand region into the hand keypoint detection network for hand keypoint detection, and output H. n The coordinates of the 21 key points of the hand are as follows:

[0108] exist Figure 2 In the middle, the key point 0 is the wrist, denoted as... Key points 1 through 4 are the four joints of the thumb from the base to the tip, denoted as... Key points 5 through 8 are the four joints of the index finger from the base to the tip, denoted as... Key points 9 to 12 are the four joints of the middle finger from the base to the tip, denoted as... Key points 13 to 16 are the four joints of the ring finger from the base to the tip, denoted as... Key points 17 to 20 are the four joints of the little finger from the base to the tip, denoted as...

[0109] H n The key point j of the hand is denoted as Its x, y, and z coordinates are respectively in The x and y coordinates are their relative coordinates on the input image, and the z-axis coordinate of keypoint 0 is... It is an extremely small value, denoted as H of hand number n. n The origin of the z-axis coordinate. If the number is negative, the base of the wrist is farther away from the camera; conversely, if the number is positive, the wrist is closer to the camera.

[0110] (3) Determine the string data

[0111] H is adjusted according to the following formula n The coordinates of key hand points were transformed, and the resulting x, y, and z axis coordinates are as follows:

[0112]

[0113]

[0114]

[0115] Calculate the number of hands H using the following formula n The distance between key point 5 and key point 17 is taken as the palm width L. n :

[0116]

[0117] The string data S is obtained by the following formula:

[0118]

[0119] Where k∈{0,1,...,20}.

[0120] Because this invention employs a step of determining string data, it is more suitable for transmission and use of User Datagram Protocol (UDP), thereby improving data transmission speed and processing efficiency.

[0121] (4) Transmit string data S

[0122] The string data S is transmitted to the Unity engine via the User Datagram Protocol.

[0123] (5) Constructing a virtual hand model

[0124] The skeleton positions are drawn using the Unity engine. Skeleton rotation angles and relative displacements are added to keypoints. The rotation angle of wrist keypoint 0 is 0°–180°, while the rotation angles of other keypoints are 0°–90°. In this embodiment, the rotation angle of wrist keypoint 0 is 90°, and the rotation angles of the other keypoints are 45°. The fingers are positioned with a radius of 12d centered on the camera. lM can move within a range, where d lM The distance between keypoint 0 and keypoint 1 of the left hand in the virtual hand model.

[0125] (6) Store string data S

[0126] 1) Hand coordinate data is passed into the Unity engine.

[0127] 2) Obtain the standardized string data S′ using the following formula:

[0128]

[0129] 3) Store standardized string data S′

[0130] Store the standardized string data S′ into the left-hand string h according to the following formula. l With the right-hand string h r middle:

[0131]

[0132]

[0133] Where N is a null value.

[0134] Because this invention employs the step of storing string data, meaningful information obtained from the User Datagram Protocol (UDP) transmission is filtered out, laying the foundation for driving the movement of the virtual hand.

[0135] (7) Virtual hand movements

[0136] 1) Use commas to separate the left-hand string h l With the right-hand string h r Divide into left-hand string arrays F, each containing 64 substrings. l ′ and the right-hand string array F r ', use the float.Parse function to convert the left-hand string array F l ′ and the right-hand string array F r Convert each of the above into a left-handed floating-point array F. l With right-hand floating-point array F r :

[0137]

[0138]

[0139] Among them, the left-hand string h l When the value is N, it is not assigned to the left-hand floating-point array F. l ; right-hand string h r When the value is N, it is not assigned to the right-hand floating-point array F. r .

[0140] 2) The virtual hand as a whole moves with the camera as a sub-object of the camera.

[0141] 3) Determine the relative coordinates of key points

[0142] The relative coordinates of left-hand keypoint i with respect to left-hand keypoint 0 are determined by the following formula. Left hand width d lH :

[0143]

[0144]

[0145]

[0146]

[0147] in, For left-handed floating-point array F l element i, For left-handed floating-point array F l The 3ith element receives the x-axis coordinate of the left-hand keypoint i. For left-handed floating-point array F l Element 0 receives the x-axis coordinate of the left-hand keypoint 0. For left-handed floating-point array F l The 3i+1 element receives the y-axis coordinate of the left-hand key point i. For left-handed floating-point array F l Element 1 receives the y-coordinate of the left-hand keypoint 0. For left-handed floating-point array F l The 3i+2 element receives the z-axis coordinate of the left-hand key point i. For left-handed floating-point array F l The second element receives the z-axis coordinate of the left-hand key point 0.

[0148] The relative coordinates of right-hand keypoint i with respect to right-hand keypoint 0 are determined by the following formula. Right hand width d rH :

[0149]

[0150]

[0151]

[0152]

[0153] in, F is a right-handed floating-point array r element i, F is a right-handed floating-point array r The 3ith element receives the x-axis coordinate of the right-hand keypoint i. F is a right-handed floating-point array r Element 0 receives the x-coordinate of the right-hand keypoint 0. F is a right-handed floating-point array r The 3i+1 element receives the y-axis coordinate of the right-hand key point i. F is a right-handed floating-point array rElement 1 receives the y-coordinate of the right-hand key point 0. F is a right-handed floating-point array r The element 3i+2 receives the z-axis coordinate of the right-hand key point i. F is a right-handed floating-point array r The second element receives the z-axis coordinate of the right-hand key point 0.

[0154] 4) Determine the scaling ratio of the virtual hand

[0155] Determine the distance d between left-hand key point 0 and left-hand key point 1 using the following formula. lR :

[0156]

[0157] Determine the distance d between right-hand key point 0 and right-hand key point 1 using the following formula. rR :

[0158]

[0159] Determine the virtual left hand scaling ratio M using the following formula. l :

[0160]

[0161] Determine the virtual right hand scaling ratio M using the following formula. r :

[0162]

[0163] Where, d lM d represents the distance between keypoints 0 and 1 of the left hand in the virtual hand model. rM This is the distance between keypoint 0 and keypoint 1 of the right hand in the virtual hand model.

[0164] 5) Determine the relative position of hand movements

[0165] The coordinates C relative to the camera are determined by the following formula when the virtual left hand moves. lx C ly C lz :

[0166]

[0167]

[0168]

[0169] Among them, D lx D ly D lzThe initial position D of the virtual left hand l Coordinates relative to the camera.

[0170] The coordinates C relative to the camera during the movement of the virtual right hand are determined by the following formula. rx C ry C rz :

[0171]

[0172]

[0173]

[0174] Among them, D rx D ry D rz The initial position D of the virtual right hand r Coordinates relative to the camera.

[0175] Because this invention uses data-driven virtual hand movements, the virtual hand movements can be more precise and natural, and more consistent with real-world hand movements, thereby enhancing the user's immersion in the augmented reality environment.

[0176] (8) Triggering augmented reality gesture interactions

[0177] 1) When the distance between the current position of the augmented reality virtual hand and the virtual object is ≤0.3 and the distance between the virtual hand key points 8, 12, 16, 20 and the virtual hand key point 0 is ≤0.05, the gesture interaction of picking up the virtual object is triggered.

[0178] 2) When the distance between augmented reality virtual hand keypoints 8, 12, 16, 20 and virtual hand keypoint 0 is all greater than 0.05, the gesture interaction of putting down the virtual object is triggered.

[0179] Because this invention uses a key point detection network to capture hand data for interaction, it has lower device costs and greater versatility compared to traditional controllers and data gloves that are expensive and require specialized hardware.

[0180] (9) Performance Evaluation

[0181] The frame rate (FPS) of images processed by the keypoint detection-based monocular camera augmented reality gesture interaction method is evaluated using the following formula:

[0182]

[0183] Among them, t e It is the time when a frame is processed. sThis refers to the time when processing begins for a single frame. When the number of frames processed per second (FPS) is greater than 30 frames per second, this method is considered to have high real-time performance.

[0184] Complete a monocular camera-based augmented reality gesture interaction method based on key point detection.

[0185] Example 2

[0186] The monocular camera augmented reality gesture interaction method based on key point detection in this embodiment consists of the following steps:

[0187] (1) Obtain the input image

[0188] The steps are the same as in Example 1.

[0189] (2) Detect the coordinates of key points on the hand

[0190] The hand keypoint coordinates of the input image are obtained using a hand keypoint detection network as follows:

[0191] 1) Set the hand confidence threshold θ, θ∈(0,1). In this embodiment, θ is 0.01. When the hand confidence is lower than θ, the hand detection model is started.

[0192] The other steps in this procedure are the same as in Example 1.

[0193] (3) Determine the string data

[0194] The steps are the same as in Example 1.

[0195] (4) Transmit string data S

[0196] The steps are the same as in Example 1.

[0197] (5) Constructing a virtual hand model

[0198] The skeleton positions are drawn using the Unity engine. Skeleton rotation angles and relative displacements are added to keypoints. The rotation angle of wrist keypoint 0 is 0°–180°, while the rotation angles of other keypoints are 0°–90°. In this embodiment, the rotation angle of wrist keypoint 0 is 0°, and the rotation angles of all other keypoints are 0°. The fingers are positioned with a radius of 12d centered on the camera. lM can move within a range, where d lM The distance between keypoint 0 and keypoint 1 of the left hand in the virtual hand model.

[0199] The other steps are the same as in Example 1. This completes the monocular camera-based augmented reality gesture interaction method based on key point detection.

[0200] Example 3

[0201] The monocular camera augmented reality gesture interaction method based on key point detection in this embodiment consists of the following steps:

[0202] (1) Obtain the input image

[0203] The steps are the same as in Example 1.

[0204] (2) Detect the coordinates of key points on the hand

[0205] The hand keypoint coordinates of the input image are obtained using a hand keypoint detection network as follows:

[0206] 1) Set the hand confidence threshold θ, θ∈(0,1). In this embodiment, the value of θ is 0.98. When the hand confidence is lower than θ, the hand detection model is started.

[0207] The other steps in this procedure are the same as in Example 1.

[0208] (3) Determine the string data

[0209] The steps are the same as in Example 1.

[0210] (4) Transmit string data S

[0211] The steps are the same as in Example 1.

[0212] (5) Constructing a virtual hand model

[0213] The skeleton positions are drawn using the Unity engine. Skeleton rotation angles and relative displacements are added to keypoints. The rotation angle of wrist keypoint 0 is 0°–180°, while the rotation angles of other keypoints are 0°–90°. In this embodiment, the rotation angle of wrist keypoint 0 is 180°, and the rotation angles of the other keypoints are 90°. The fingers are positioned with a radius of 12d centered on the camera. lM can move within a range, where d lM The distance between keypoint 0 and keypoint 1 of the left hand in the virtual hand model.

[0214] The other steps are the same as in Example 1. This completes the monocular camera-based augmented reality gesture interaction method based on key point detection.

[0215] To verify the beneficial effects of the present invention, the inventors conducted a simulation experiment using the method of Embodiment 1 of the present invention. The experimental results are as follows:

[0216] 1. Simulation conditions

[0217] Software environment: PyCharm 2019.3.1 x64.

[0218] The hardware requirements are: 1 personal computer, 1 Nvidia 3060Ti graphics card, 1 1080P webcam, and 1 personal mobile phone.

[0219] Computer configuration:

[0220] 1) Processor: Intel(R) Core(TM) i7-10700 CPU @ 2.90GHz 2.90GHz.

[0221] 2) Memory: 32.0GB.

[0222] The software platform is Python 3.8.

[0223] Other third-party libraries: opencv-python4.6.0, mediapipe0.9.1, socket.

[0224] 2. Simulation Content and Results

[0225] The experiment was conducted under the above simulation conditions, and the results are shown below. Figure 3 .

[0226] exist Figure 3 In the diagram, the horizontal axis represents the running time of the invention, and the vertical axis represents the number of frames that the invention can process per second, i.e., FPS; Figure 3 As can be seen, the number of video images processed by this invention fluctuates around 30 per second, indicating that this invention processes video images quickly and has real-time performance.

Claims

1. A monocular camera-based augmented reality gesture interaction method based on key point detection, characterized in that... It consists of the following steps: (1) Obtain the input image The real-time image captured by the monocular camera is used as the input image, and the width w of the input image is at least 400 pixels and the height h is at least 200 pixels; (2) Detect the coordinates of key points on the hand The hand keypoint coordinates of the input image are obtained using a hand keypoint detection network as follows: 1) Set a confidence threshold θ for the hand, where θ∈(0,1). The hand detection model will start when the confidence level for the hand is lower than θ. 2) Perform hand detection on the input image, with hands numbered 0 and 1 from left to right, where n is either 0 or 1, and hand number n is denoted as H. n H n This includes left-hand label 'l' or right-hand label 'r'; 3) Locate the detected hand and cut out the hand area; 4) Input each hand region into the hand keypoint detection network for hand keypoint detection, and output H. n The coordinates of the 21 key points of the hand are as follows: Key point 0 is the wrist, denoted as Key points 1 through 4 are the four joints of the thumb from the base to the tip, denoted as... Key points 5 through 8 are the four joints of the index finger from the base to the tip, denoted as... Key points 9 to 12 are the four joints of the middle finger from the base to the tip, denoted as... Key points 13 to 16 are the four joints of the ring finger from the base to the tip, denoted as... Key points 17 to 20 are the four joints of the little finger from the base to the tip, denoted as... H n The key point j of the hand is denoted as Its x, y, and z coordinates are respectively in The x and y coordinates are their relative coordinates on the input image, and the z-axis coordinate of keypoint 0 is... It is an extremely small value, denoted as H of hand number n. n The origin of the z-axis coordinate system. If the number is negative, the base of the wrist should be further away from the camera; conversely, if the number is positive, the wrist should be closer to the camera. (3) Determine the string data H is calculated using the following formula n The coordinates of key hand points were transformed, and the resulting x, y, and z axis coordinates are as follows: Calculate the number of hands H using the following formula n The distance between key point 5 and key point 17 is taken as the palm width L. n : The string data S is obtained by the following formula: Where k∈{0,1,...,20}; (4) Transmit string data S Transmit string data S to the Unity engine via User Datagram Protocol; (5) Constructing a virtual hand model Use the Unity engine to draw the bone positions, and add bone rotation angles and relative displacements at key points; (6) Store string data S 1) Hand coordinate data is passed into the Unity engine; 2) Obtain the standardized string data S′ using the following formula: 3) Store standardized string data S′ Store the standardized string data S′ into the left-hand string h according to the following formula. l With the right-hand string h r middle: Where N is a null value; (7) Virtual hand movements 1) Use commas to separate the left-hand string h l With the right-hand string h r Divide into left-hand string arrays F, each containing 64 substrings. l ′ and the right-hand string array F r ', use the float.Parse function to convert the left-hand string array F l ′ and the right-hand string array F r Convert each of the above into a left-handed floating-point array F. l With right-hand floating-point array F r : Among them, the left-hand string h l When the value is N, it is not assigned to the left-hand floating-point array F. l ; right-hand string h r When the value is N, it is not assigned to the right-hand floating-point array F. r ; 2) The virtual hand as a whole moves with the camera as a sub-object of the camera; 3) Determine the relative coordinates of key points The relative coordinates of left-hand keypoint i with respect to left-hand keypoint 0 are determined by the following formula. Left hand width d lH : in, For left-handed floating-point array F l element i, For left-handed floating-point array F l The 3ith element receives the x-axis coordinate of the left-hand keypoint i. For left-handed floating-point array F l Element 0 receives the x-axis coordinate of the left-hand keypoint 0. For left-handed floating-point array F l The 3i+1 element receives the y-axis coordinate of the left-hand key point i. For left-handed floating-point array F l Element 1 receives the y-coordinate of the left-hand keypoint 0. For left-handed floating-point array F l The 3i+2 element receives the z-axis coordinate of the left-hand key point i. For left-handed floating-point array F l Element number 2 receives the z-axis coordinate of the left-hand key point 0; The relative coordinates of right-hand keypoint i with respect to right-hand keypoint 0 are determined by the following formula. Right hand width d rH : in, F is a right-handed floating-point array r element i, F is a right-handed floating-point array r The 3ith element receives the x-axis coordinate of the right-hand keypoint i. F is a right-handed floating-point array r Element 0 receives the x-coordinate of the right-hand keypoint 0. F is a right-handed floating-point array r The 3i+1 element receives the y-axis coordinate of the right-hand key point i. F is a right-handed floating-point array r Element 1 receives the y-coordinate of the right-hand key point 0. F is a right-handed floating-point array r The element 3i+2 receives the z-axis coordinate of the right-hand key point i. F is a right-handed floating-point array r Element number 2 receives the z-axis coordinate of the right-hand key point 0; 4) Determine the scaling ratio of the virtual hand Determine the distance d between left-hand key point 0 and left-hand key point 1 using the following formula. lR : Determine the distance d between right-hand key point 0 and right-hand key point 1 using the following formula. rR : Determine the virtual left hand scaling ratio M using the following formula. l : Determine the virtual right hand scaling ratio M using the following formula. r : Where, d lM d represents the distance between keypoints 0 and 1 of the left hand in the virtual hand model. rM The distance between keypoint 0 and keypoint 1 of the right hand in the virtual hand model; 5) Determine the relative position of hand movements The coordinates C relative to the camera are determined by the following formula when the virtual left hand moves. lx C ly C lz : Among them, D lx D ly D lz The initial position D of the virtual left hand l Coordinates relative to the camera; The coordinates C relative to the camera during the movement of the virtual right hand are determined by the following formula. rx C ry C rz : Among them, D rx D ry D rz The initial position D of the virtual right hand r Coordinates relative to the camera; (8) Triggering augmented reality gesture interactions 1) When the distance between the current position of the augmented reality virtual hand and the virtual object is ≤0.3 and the distance between virtual hand key points 8, 12, 16, 20 and virtual hand key point 0 is ≤0.05, the gesture interaction of picking up the virtual object is triggered. 2) When the distance between augmented reality virtual hand keypoints 8, 12, 16, 20 and virtual hand keypoint 0 is all greater than 0.05, the gesture interaction of putting down the virtual object is triggered. (9) Performance Evaluation The frame rate (FPS) of images processed by the keypoint detection-based monocular camera augmented reality gesture interaction method is evaluated using the following formula: Among them, t e It is the time when a frame is processed. s This refers to the time when processing begins for a single frame. When the number of frames processed per second (FPS) is greater than 30 frames per second, this method is considered to have high real-time performance.

2. The monocular camera augmented reality gesture interaction method based on key point detection according to claim 1, characterized in that: In step (2) of detecting the coordinates of key points on the hand, the confidence threshold for hand detection is 0.

68.

3. The monocular camera augmented reality gesture interaction method based on key point detection according to claim 1, characterized in that: In step (5) of constructing the virtual hand model, the rotation angle of the bones added to the key points is as follows: the rotation angle of the wrist key point 0 is 0° to 180°, the rotation angle of the other key points is 0° to 90°, and the fingers are within a radius of 12d centered on the camera. lM can move within a range, where d lM The distance between keypoint 0 and keypoint 1 of the left hand in the virtual hand model.

Images