Interaction method, apparatus and device of near-eye display device and storage medium

Abstract

The application relates to the technical field of gesture interaction, and provides an interaction method and device of a near-eye display device, equipment and a storage medium, which comprises the following steps: acquiring a first hand image, the first hand image comprising at least two first key points of a user's hand, an object, and the size of the object in a corresponding direction being a preset size; determining a target length of a part corresponding to at least two second key points and a fourth key point in the user's hand according to the first positions of the at least two first key points and the at least two second key points in the first hand image, the preset size and the target length; acquiring a second hand image; determining a target position of the first key point according to the second positions of the at least two second key points in the second hand image, the third positions of the at least two second key points, the third positions of the at least two first key points and the target length; and performing an interaction operation on a virtual keyboard according to the target position of the first key point, so that the interaction convenience of the near-eye display device for the virtual keyboard is improved.

Description

Technical Field

[0001] This application relates to the field of gesture interaction technology, and in particular to an interaction method, apparatus, device and storage medium for near-eye display devices. Background Technology

[0002] Currently, in near-eye display devices, when a virtual keyboard is displayed, users can input information by simulating pressing the virtual keyboard with their fingers. From the user's eye, the virtual keyboard displayed on the near-eye display device appears to float in mid-air. Therefore, when a user interacts with the virtual keyboard using their fingers, the near-eye display device needs to obtain the depth information of the user's fingers to determine the interaction operation.

[0003] In related technologies, depth estimation of the user's hand is typically performed to determine the depth information of the user's fingers. However, the relative error is smaller when estimating the depth of distant objects, but larger when estimating the depth of nearby objects, and the distance between the user's hand and the near-eye display device is generally short. If the near-eye display device uses depth estimation to identify the depth information of the user's hand for interaction with the virtual keyboard, the accuracy of the near-eye display device in recognizing the user's hand depth is likely to be poor, negatively impacting the ease of interaction with the virtual keyboard. Summary of the Invention

[0004] The main objective of this application is to provide an interaction method, apparatus, device, and storage medium for a near-eye display device, aiming to solve the technical problem that the accuracy of near-eye display devices in recognizing the depth information of the user's hand is poor due to the use of depth estimation to identify the depth information of the user's hand for interaction with the virtual keyboard, resulting in poor interaction convenience of the near-eye display device with the virtual keyboard.

[0005] In a first aspect, this application provides an interaction method for a near-eye display device, the interaction method comprising:

[0006] A first hand image captured by a near-eye display device is acquired. The first hand image includes at least two first key points of the user's hand and an object between the at least two first key points. The size of the object in the direction corresponding to the at least two first key points is a preset size.

[0007] Based on the first position of each of the at least two first key points and the at least two second key points of the user's hand in the first hand image and the preset size, determine the target length of the part of the user's hand corresponding to the at least two second key points;

[0008] Acquire a second hand image of the user's hand captured by the near-eye display device;

[0009] Based on the second and third positions of the at least two second key points in the second hand image, the third position of the at least two first key points in the second hand image, and the target length, the target position of the first key point is determined; the second position includes 2D coordinates, and the third position includes 2.5D coordinates.

[0010] Based on the target position of the first key point, perform corresponding interactive operations on the virtual keyboard displayed by the near-eye display device.

[0011] Secondly, this application provides an interactive device for a near-eye display device, the interactive device comprising:

[0012] The first image acquisition module is used to acquire a first hand image captured by a near-eye display device. The first hand image includes at least two first key points of the user's hand and an object between the at least two first key points. The size of the object in the direction corresponding to the at least two first key points is a preset size.

[0013] The distance determination module is used to determine the target length of the part of the user's hand corresponding to the at least two second key points based on the first position of each of the at least two first key points and the at least two second key points of the user's hand in the first hand image and the preset size;

[0014] The second image acquisition module is used to acquire a second hand image obtained by the near-eye display device from the user's hand;

[0015] The target position determination module is used to determine the target position of the first key point based on the second and third positions of the at least two second key points in the second hand image, the third position of the at least two first key points in the second hand image, and the target length; the second position includes 2D coordinates, and the third position includes 2.5D coordinates;

[0016] The interaction module is used to perform corresponding interactive operations on the virtual keyboard displayed by the near-eye display device according to the target position of the first key point.

[0017] Thirdly, this application provides a near-eye display device, which includes a memory and a processor;

[0018] The memory is used to store computer programs;

[0019] The processor is configured to execute the computer program and, in executing the computer program, implement the steps of the interaction method for the near-eye display device as described above.

[0020] Fourthly, this application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the interaction method for a near-eye display device as described above.

[0021] This application provides an interaction method, apparatus, device, and storage medium for a near-eye display device. The interaction method includes: acquiring a first hand image captured by the near-eye display device, the first hand image including at least two first key points of a user's hand and an object between the at least two first key points, the size of the object in the direction corresponding to the at least two first key points being a preset size; determining a target length of a part of the user's hand corresponding to the at least two second key points based on the first position of each of the at least two first key points and at least two second key points in the first hand image and the preset size; acquiring a second hand image captured by the near-eye display device; determining a target position of a first key point based on the second position and third position of each of the at least two second key points in the second hand image, the third position of each of the at least two first key points in the second hand image, and the target length; the second position including 2D coordinates and the third position including 2.5D coordinates; and performing a corresponding interaction operation on a virtual keyboard displayed on the near-eye display device based on the target position of the first key point.

[0022] Since the preset size of the object is its actual size in the user's space, it can be used by the near-eye display device to determine the first distance between at least two first key points. Correspondingly, the near-eye display device can combine the first hand image and the preset size to determine the target length of the corresponding parts of at least two second key points. Because the target length of this part does not change with the user's gestures, when the near-eye display device subsequently captures a second image of the user's hand, it can combine the target length to determine the target position of the corresponding key point on the user's hand. For example, the near-eye display device can combine the target length and the second hand image to determine the target position of the first key point. Since the target position can include the target depth, this improves the accuracy of the near-eye display device's depth recognition of the user's hand. The target position of the first key point allows the near-eye display device to perform corresponding interactive operations on the virtual keyboard, thus improving the ease of interaction with the virtual keyboard. Attached Figure Description

[0023] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0024] Figure 1 This is a flowchart illustrating an interaction method for a near-eye display device provided in an embodiment of this application;

[0025] Figure 2 This is a schematic diagram of preset key points of a user's hand according to an embodiment of this application;

[0026] Figure 3 This is a schematic diagram of a first hand image according to an embodiment of this application;

[0027] Figure 4 This is a schematic diagram of a user's hand with preset key points at a first position in a first hand image, according to an embodiment of this application.

[0028] Figure 5 This is a schematic diagram of the display range corresponding to a near-eye display device according to an embodiment of this application;

[0029] Figure 6 This is a schematic diagram of the display range corresponding to a near-eye display device according to another embodiment of this application;

[0030] Figure 7 This is a schematic diagram of the display range corresponding to a near-eye display device according to another embodiment of this application;

[0031] Figure 8 This is a schematic block diagram of an interactive device for a near-eye display provided in an embodiment of this application;

[0032] Figure 9 This is a schematic block diagram of a near-eye display device provided in an embodiment of this application. Detailed Implementation

[0033] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0034] The flowchart shown in the attached diagram is for illustrative purposes only and does not necessarily include all content and operations / steps, nor does it necessarily have to be performed in the order described. For example, some operations / steps can be broken down, combined, or partially merged, so the actual execution order may change depending on the actual situation.

[0035] This application provides a gesture recognition method, apparatus, device, and storage medium. The gesture recognition method can be applied to near-eye display devices. Near-eye display devices may include augmented reality (AR) glasses, virtual reality (VR) glasses, AR headsets, VR headsets, etc., and are not limited thereto.

[0036] The following detailed description of some embodiments of this application is provided in conjunction with the accompanying drawings. Unless otherwise specified, the following embodiments and features can be combined with each other.

[0037] Please see Figure 1 , Figure 1 This is a flowchart illustrating an interaction method for a near-eye display device according to an embodiment of this application. It should be noted that the interaction method for a near-eye display device provided in this embodiment can be used in near-eye display devices.

[0038] like Figure 1 As shown, the interaction method of the near-eye display device includes steps S101 to S105.

[0039] S101. Acquire a first hand image captured by a near-eye display device. The first hand image includes at least two first key points of the user's hand and an object between the at least two first key points. The size of the object in the direction corresponding to the at least two first key points is a preset size.

[0040] For example, when a user wearing a near-eye display device interacts with the device, they can perform corresponding interactive operations using gestures. Based on this, the near-eye display device can acquire an image of the user's hand to recognize the gestures made by the user during interaction with the device.

[0041] In some implementations, the near-eye display device has a camera function. Based on this camera function, the near-eye display device can capture images of the user's hand. For example, when the near-eye display device detects that the user's hand is within a preset camera range, it can capture an image of the user's hand.

[0042] For example, a near-eye display device may be equipped with a camera module to capture images of the user's hand. The camera module may include an RGB camera, a grayscale fisheye camera, etc., and is not limited thereto. The number of camera modules on the near-eye display device may be one or more, and is not limited thereto. When a near-eye display device is equipped with multiple camera modules, the device can capture an image of the user's hand using one of the multiple camera modules.

[0043] In some implementations, the user's hand may include multiple preset key points. For example, such as Figure 2 As shown, the user's hand can include multiple key points. The near-eye display device can set each key point of the user's hand as a corresponding preset key point. For example, the user's hand can include key points 0 to 20, a total of 21 preset key points.

[0044] Because the distance between some joints in a user's hand does not change with the user's hand gestures. For example, when the back of the hand cannot bend, the distance between the wrist points (such as...) remains constant. Figure 2 The key point 0) and the base of the ring finger (such as Figure 2 The length of the back of the palm corresponding to key point 13) does not change with the user's gesture. Correspondingly, the length of the wrist point (e.g., ...) in the user's hand... Figure 2 The key point 0 in the middle finger and the base of the middle finger (such as...) Figure 2 The key point 9) corresponds to the length of the back of the palm, and the wrist point in the user's hand (such as...). Figure 2 The key point 0 in the middle and the base of the index finger (such as...) Figure 2 The length of the back of the hand corresponding to key point 5) will not change with the user's gesture.

[0045] Based on this, the near-eye display device can identify the target position of each preset key point when the user makes different gestures, based on the invariance of the distance between the at least two key points, so that the near-eye display device can perform interactive operations on the virtual keyboard.

[0046] In some implementations, the near-eye display device can acquire a first image of the user's hand obtained by photographing the user's hand. The first hand image can be used to determine the target length of the corresponding portion of at least two joints in the user's hand that have distance invariance.

[0047] For example, at least two first keypoints can be determined from multiple preset keypoints on the user's hand. If the first hand image includes at least two first keypoints on the user's hand and an object between the at least two first keypoints, and the size of the object in the direction corresponding to the at least two first keypoints is a preset size, the near-eye display device can determine that the user is pinching the object through the at least two first keypoints on the user's hand, and the size of the object pinched by the user includes the preset size. The preset size may include the object's length, width, diameter, and other object-related dimensions, and is not limited here. Figure 3 As shown, the first key point may include the user's thumb tip and index finger tip. The user pinches an object using the tips of their thumb and index finger. The near-eye display device can obtain the size of the object being pinched. For example, the object may include a companion object for the near-eye display device. Companion objects may include a container for the near-eye display device, replacement accessories for the near-eye display device, etc., without limitation. The near-eye display device may pre-store the sizes of the companion objects. Another example is that the object may include everyday items. Everyday items may include milk cartons, tissue boxes, toothpick holders, water cup lids, etc., without limitation. The near-eye display device can obtain the sizes of everyday items via a network. In an exemplary embodiment, the near-eye display device can search for size diagrams of everyday items from the network based on images, trademarks, and other identifying information to determine the size of the everyday object. However, this is not limited to this; the near-eye display device can output size prompts to prompt the user to input the object's size, allowing the user to respond to the prompts, obtain the object's size, and input it into the near-eye display device, without limitation.

[0048] When a near-eye display device determines, based on a first hand image, that a user is pinching an object through at least two first key points on the user's hand, and the near-eye display device has acquired the size of the object, the near-eye display device can determine a first distance between the at least two first key points based on the object's size. For example, since the distance between the at least two first key points in the first hand image can change in response to variations in the image's capture parameters, by introducing the object between the at least two key points as a reference, when the size of the object in the directions corresponding to the at least two first key points is determined to be a preset size, the first distance between the at least two first key points can also be determined as a preset size. For instance, when the object pinched by the user through at least two first key points is rectangular, and the user is pinching the object's long side, the near-eye display device can determine that the size of the object in the directions corresponding to the at least two first key points is the object's length. The near-eye display device can then determine the object's length as a preset size. Furthermore, the near-eye display device's determination that the distance between the at least two first key points of the user's hand when pinching the object is a preset size is equivalent to the object's size in the directions corresponding to the at least two first key points being a preset size. Based on this, the near-eye display device can then combine the first hand image with the preset size of the object to determine the target length of the user's hand corresponding to at least two joints with distance invariance.

[0049] In some implementations, when acquiring the first hand image, the near-eye display device can authenticate the user to determine if the user is a new user. When the near-eye display device determines that the user is a new user, it can output a first prompt message, which prompts the user to pinch an object using at least two first key points on the user's hand. When the near-eye display device detects that the user is pinching an object using at least two first key points on the user's hand, it can capture a first hand image. For example, the near-eye display device can acquire the user's voiceprint, fingerprint, iris, account password, or other identity identifiers that can be used to identify the user for authentication, without limitation. For example, the near-eye display device can output the first prompt message in text, voice, animated GIF, or other forms, without limitation. When the near-eye display device determines that the user is not a new user, it can acquire the pre-stored target length of the part of the user's hand corresponding to at least two second key points based on the user's identity identifier, for subsequent interaction with the virtual keyboard, without limitation.

[0050] Thus, when a near-eye display device acquires a first hand image, it can determine the distance between at least two first key points on the user's hand based on the dimensions of the object in the first hand image in the directions corresponding to at least two first key points on the user's hand as preset dimensions. These preset dimensions allow the near-eye display device to subsequently determine the target length of the portion of the user's hand corresponding to at least two key points with invariant distance, thereby improving the ease of determining the target length.

[0051] S102. Based on the first position of each of the at least two first key points and the at least two second key points of the user's hand in the first image of the first hand and the preset size, determine the target length of the part of the user's hand corresponding to the at least two second key points.

[0052] For example, when a near-eye display device acquires a first image of a hand, it can perform key point detection on that image. Figure 4 As shown, the near-eye display device can determine at least two first key points and at least two second key points of the user's hand, each at a first position in a first hand image, based on a preset key point detection algorithm or a deep learning model. Specifically, at least two second key points can be determined from multiple preset key points of the user's hand based on distance invariance. The second key points are different from the first key points.

[0053] In some implementations, the first position may include 2.5D coordinates. The near-eye display device can perform 2.5D keypoint detection on a first hand image to obtain the 2.5D coordinates of multiple preset keypoints of the user's hand in the first hand image. The multiple preset keypoints include at least two first keypoints and at least two second keypoints. The near-eye display device can determine the 2.5D coordinates of the at least two first keypoints and the at least two second keypoints in the first hand image as the first position of each keypoint in the first hand image.

[0054] For example, since a user can pinch an object using at least two first key points on their hand, these at least two first key points do not possess distance invariance. For instance, if the same object can have different sizes, such as the length of the object's long side differing from the length of its short side, the distance between these at least two first key points can change as the user pinches the object's side length. Similarly, if different objects can have different sizes, and the user can pinch different objects using these at least two first key points, the distance between these at least two first key points will change as the object being pinched changes.

[0055] Based on the user's ability to make different gestures, the near-eye display device can select at least two second key points with distance invariance from multiple preset key points on the user's hand. These second key points are used to identify the target position of the first key point when the user makes different gestures, thereby performing corresponding interactive operations on the virtual keyboard displayed by the near-eye display device. Since the target length of the corresponding part of the at least two second key points does not change with the user's gestures, the near-eye display device can subsequently identify the target position of the first key point in the user's hand based on the target length captured by the near-eye display device when the user makes any gesture. This allows the near-eye display device to perform corresponding interactive operations on the virtual keyboard. For example, the second key points may include wrist points (such as...). Figure 2 Key points 0), the base of the index finger (such as...) Figure 2 Key point 5), the base of the middle finger (such as...) Figure 2 Key point 9), the base of the ring finger (such as...) Figure 2 Key point 13), the base of the little finger (such as...) Figure 2 Key points 17), etc., are not limited here. Since the palm portions corresponding to at least two second key points cannot bend, the target length of the palm portion will not change with different gestures made by the user. Accordingly, when the user makes different gestures, the target length of the same palm portion is the same, so the near-eye display device can combine this target length to determine the target position of the first key point in the user's hand when the user makes different gestures later.

[0056] For example, a near-eye display device can determine the target length of the part of the user's hand corresponding to the at least two second key points by combining the first positions of at least two first key points, the at least two second key points, and a preset size.

[0057] In some implementations, a first proportional relationship between a preset size and a target length is determined based on the ratio between a first distance between at least two first key points and a second distance between at least two second key points; wherein the first distance is determined based on the first position of each of the at least two first key points, and the second distance is determined based on the first position of each of the at least two second key points; the target length is obtained by substituting the first positions of the at least two first key points, the at least two second key points, and the preset size into the first proportional relationship.

[0058] The first key point includes the tip of the index finger (e.g.) Figure 2 Key point 8), thumb tip point (e.g.) Figure 2 Key point 4), the second key point includes the wrist point (such as... Figure 2 Key point 0), the base of the middle finger (such as...) Figure 2Taking key point 9 as an example. The first positions of key points 0, 4, 8, and 9 in the first hand image determined by the near-eye display device can be represented as (X10, Y10, Z10), (X14, Y14, Z14), (X18, Y18, Z18), and (X19, Y19, Z19), respectively. Based on the first positions of key points 8 and 4, the first distance between key points 8 and 4 can be determined. Since the size of the object in the corresponding direction of key points 8 and 4 is a preset size, the first distance between key points 8 and 4 can be determined as a preset size. Correspondingly, based on the first positions of key points 0 and 9, the second distance between key points 0 and 9 can be determined. Based on the invariance of distance between key points 0 and 9, the distance between key points 0 and 9 is the target length. Based on the ratio between the first distance and the second distance, a first proportional relationship between the preset size and the target length can be established. The first proportional relationship can be expressed as:

[0059]

[0060] Where L indicates the target length and M indicates the preset size.

[0061] When the near-eye display device obtains the preset size, it can substitute the first positions of key points 0, 4, 8, and 9 into the first proportional relationship to obtain the target length.

[0062] In this way, the near-eye display device can determine the target length of the part of the user's hand corresponding to at least two second key points based on the first positions of at least two first key points, the first dimensions of at least two second key points, and a preset size. This target length can be used by the near-eye display device to determine the target position of the first key points on the user's hand when making different gestures. The target position may include target depth, which helps improve the accuracy of the near-eye display device in recognizing the depth of the user's hand. Furthermore, the near-eye display device can subsequently perform corresponding interactive operations on the virtual keyboard displayed on the near-eye display device based on the target position of the first key points on the user's hand, thus improving the ease of interaction with the virtual keyboard.

[0063] For example, after determining the target length, the target length can be used by the near-eye display device to determine the target display depth for the virtual keyboard.

[0064] In some implementations, the target display depth of the virtual keyboard on the near-eye display device is determined based on a multiple of the target length; the near-eye display device is then controlled to display the virtual keyboard at the target display depth.

[0065] For example, the multiple of the target length can include 1x, 1.5x, 2x, 2.5x, 3x, 4x, etc., without limitation. When displaying a virtual keyboard, the near-eye display device can determine the target display depth of the virtual keyboard based on the multiple of the target length. Since the user's arm length is longer than the user's palm length, when the near-eye display device determines the target display depth corresponding to the virtual keyboard based on the multiple of the target length, the user can choose at least one of the states of straightening or bending their arm to interact with the near-eye display device, which helps to improve the interaction flexibility of the near-eye display device with the virtual keyboard. Correspondingly, when different users have different palm lengths, the near-eye display device can set the target length according to the different palm lengths of different users, and then the near-eye display device can determine the target display depth of the virtual keyboard based on different multiples of the target length. The longer the target length, the greater the target display depth can be, to adapt to the different gesture interaction needs of different users with the near-eye display device, which helps to improve the flexibility of adjusting the target display depth, thereby improving the user's gesture interaction experience with the near-eye display device.

[0066] S103. Acquire a second hand image obtained by capturing the user's hand using a near-eye display device.

[0067] For example, a user can make any gesture within a preset shooting range of the near-eye display device to interact with the virtual keyboard displayed on the device. Correspondingly, the near-eye display device can capture a second image of the user's hand. This second hand image can then be used by the near-eye display device to determine the user's interaction needs with the virtual keyboard.

[0068] S104. Based on the second and third positions of at least two second key points in the second hand image, the third position of at least two first key points in the second hand image, and the target length, determine the target position of the first key point; the second position includes 2D coordinates, and the third position includes 2.5D coordinates.

[0069] For example, when a near-eye display device acquires a second hand image, it can perform keypoint detection on the second hand image. The near-eye display device can determine the second and third positions of multiple preset keypoints in the user's hand in the second hand image based on a preset keypoint detection algorithm or a deep learning model. For instance, since both the first and second keypoints are selected from multiple preset keypoints, and the first and second keypoints are different, the near-eye display device can determine the second and third positions of at least two second keypoints in the second hand image, or it can determine the third position of at least two first keypoints in the second hand image.

[0070] In some embodiments, the near-eye display device can perform 2D keypoint detection on the second hand image to obtain the 2D coordinates of multiple preset keypoints of the user's hand in the second hand image. If the multiple preset keypoints include at least two second keypoints, the near-eye display device can determine the 2D coordinates of each of the at least two second keypoints as the second position of the corresponding keypoint. Correspondingly, the near-eye display device can perform 2.5D keypoint detection on the second hand image to obtain the 2.5D coordinates of multiple preset keypoints of the user's hand in the second hand image. If the multiple preset keypoints include at least two first keypoints and at least two second keypoints, the near-eye display device can determine the 2.5D coordinates of each of the at least two first keypoints and at least two second keypoints as the third position of the corresponding keypoint.

[0071] Given that at least two second key points are respectively located at a second position and a third position in the second hand image, at least two first key points are respectively located at a third position in the second hand image, and a target length, the near-eye display device can combine the second position, the third position, and the target length to determine the target position of the user's hand corresponding to the second hand image in the spatial coordinate system corresponding to the near-eye display device.

[0072] In some implementations, the projection relationship between the target position and the second position of the second key point is determined based on the camera intrinsic parameters of the near-eye display device and the second position of the second key point; the target distance between the target positions of at least two second key points is determined based on the target length; a second proportional relationship between a first difference and a second difference is determined based on the target length and the third positions of at least two second key points; wherein the first difference is determined based on the difference between the target positions of at least two second key points, and the second difference is determined based on the difference between the third positions of at least two second key points; the target positions of at least two second key points are determined based on the projection relationship, the target distance, and the second proportional relationship; and the target position of the first key point is determined based on the target position of the second key point, the third position of the first key point, and the second proportional relationship.

[0073] The first key point includes the tip of the index finger (e.g.) Figure 2 Key point 8), thumb tip point (e.g.) Figure 2 Key point 4), the second key point includes the wrist point (such as... Figure 2 Key point 0), the base of the middle finger (such as...) Figure 2Taking key point 9 as an example. The second positions of key points 0 and 9 can be represented as (U0, V0) and (U9, V9) respectively. The third positions of key points 0, 4, 8, and 9 can be represented as (X0, Y0, Z0), (X4, Y4, Z4), (X8, Y8, Z8), and (X9, Y9, Z9) respectively. W0 Y W0 Z W0 ), (X W4 Y W4 Z W4 ), (X W8 Y W8 Z W8 ) and (X W9 Y W9 Z W9 ).

[0074] Taking the camera coordinate system corresponding to the near-eye display device, such as the camera coordinate system corresponding to the second hand image, as the world coordinate system corresponding to the near-eye display device, as an example.

[0075] The camera intrinsic parameters of a near-eye display device can be expressed as:

[0076]

[0077] Among them, f x Used to indicate the focal length along the x-axis for near-eye display devices, f y u0 and v0 are used to indicate the focal length in the y-axis direction corresponding to the near-eye display device, and are used to indicate the actual position of the principal point corresponding to the near-eye display device.

[0078] Using camera intrinsic parameters, the second keypoint can be projected from the world coordinate system corresponding to the near-eye display device to the pixel coordinate system corresponding to the second hand image, thus obtaining the second position of the second keypoint. Based on this, the near-eye display device can determine the projection relationship between the target position and the second position of the second keypoint according to the camera intrinsic parameters and the second position of the second keypoint. For example, the near-eye display device can determine the target position (X) of keypoint 0 according to the camera intrinsic parameters and the second position (U0, V0) of keypoint 0. W0 Y W0 Z W0 The projection relationship between key point 9 and the second position (U0, V0) of key point 0. Accordingly, the near-eye display device can determine the target position (X) of key point 9 based on the camera intrinsic parameters and the second position (U9, V9) of key point 9. W9 Y W9 Z W9The projection relationship between the second position (U9, V9) of key point 9 and key point 9.

[0079] The projection relationship can be expressed as:

[0080]

[0081] Where (U, V) is used to uniformly represent the second position of each of the multiple preset key points, (X) W YW, Z W () is used to uniformly represent the target positions of multiple preset key points. In (U, V) and (X) W Y W Z W When a numerical index is set, it can correspond to different preset key points. For example, the numerical index corresponding to key point 0 is 0, the numerical index corresponding to key point 9 is 9, and so on.

[0082] Near-eye display devices can substitute the second and third positions of key points 0 and 9 into the projection relationship, and then reorganize the projection relationship accordingly to obtain formulas (1) to (4), as shown below:

[0083] Z W0 ·U0=f x ·X W0 +u0·Z W0 (1)

[0084] Z W0 ·V0=f y ·Yx0+u0·Z W0 (2)

[0085] Z W9 ·U9=f x ·X W9 +u0·Z W9 (3)

[0086] Z W9 ·V9=f y ·X W9 +u0·Z W9 (4)

[0087] Near-eye display devices can determine the target distance between the target positions of at least two second key points based on the target length. For example, based on the target length L of the corresponding parts of key points 0 and 9 on the user's hand, the target position (X) of key point 0 can be determined. W0 Y W0 Z W0 ) and the target location (X) of key point 9 W9 Y W9 Z W9The target distance between them is shown in formula (5).

[0088] (X W0 -X W9 ) 2 +(Y W0 -Y W9 ) 2 +(Z W0 -Z W9 ) 2 =L 2 (5)

[0089] A near-eye display device can determine a second proportional relationship between a first difference and a second difference based on the target length and the third positions of at least two second key points. For example, the near-eye display device can determine the first difference based on the target positions of key point 0 and key point 9, which may include (X... W0 -X W9 ), (Y W0 -Y W9 ) and (Z W0 -Z W9 Accordingly, the near-eye display device can determine the second difference, which may include (X0-X9), (Y0-Y9), and (Z0-Z9), based on the third positions of key points 0 and 9 respectively. Since the second proportional relationship between the first and second differences is related to the target distance L, the second proportional relationship can be determined by combining the target distance L. Furthermore, since the coordinate axes of the coordinate systems corresponding to the third positions of key points 0 and 9 are in the same direction as the coordinate axes of the camera coordinate system corresponding to the near-eye display device, the second proportional relationship can be determined to be greater than 0, as shown in formula (6).

[0090]

[0091] R is used to indicate the second proportional relationship, and the second proportional relationship R is related to the target distance L between the target positions of key point 0 and key point 9.

[0092] Given formulas (1) to (6), the near-eye display device can combine formulas (1) to (6) to determine the target positions of key points 0 and 9, which is equivalent to combining the projection relationship, target distance, and second proportional relationship to determine the target position (X) of key point 0. W0 Y W0 Z W0 ) and the target location of key point 9 (X) W9 Y W9 Z W9 ).

[0093] For example, formulas (1) to (6) can be combined to apply X. W0 Y W0 Z W0 X W9 Y W9 and Z W9 Solve the problem.

[0094] For example, we can obtain from formula (6):

[0095]

[0096] Substituting formulas (7) and (8) into formula (5) yields (Z). W0 -Z W9 ) 2 Z can be determined according to formula (6). W0 -Z W9 The sign of Z is then used to obtain Z. W0 -Z W9 X W0 -X W9 and Y W0 -Y W9 .

[0097] Assumption:

[0098] Z W0 -Z W9 =a (9)

[0099] X W0 -X W9 =b (10)

[0100] Subtracting formula (3) from formula (1) yields:

[0101] Z W0 ·U0-Z W9 ·U9=f x ·(X W0 -X W9 )+u0·(Z W0 -Z W9 (11)

[0102] Substituting formulas (9) and (10) into formula (11) yields Z. W0 ·U0-Z W9 ·U9.

[0103] Assumption:

[0104] Z W0 ·U0-Z W9 ·U9=c (12)

[0105] Z can be obtained from formulas (9) and (12). W0 and Z W9 Therefore, X can be obtained. W0 Y W0 X W9 and Y W9 .

[0106] Determine the target location (X) of key point 0. W0 Y W0 Z W0 ) and the target location of key point 9 (X) W9 Y W9 Z W9 In the case of ), the second proportional relationship R can be determined by combining the third position (X0, Y0, Z0) of key point 0 and the third position (X9, Y9, Z9) of key point 9.

[0107] Accordingly, the target position of the first key point can be determined based on the target position of the second key point, the third position of the first key point, and the second proportional relationship. For example, the target position of key point 8 can be determined based on the target position of key point 0, the target position of key point 9, the third position of key point 8, and the second proportional relationship R. Similarly, the target position of key point 4 can be determined based on the target position of key point 0, the target position of key point 9, the third position of key point 4, and the second proportional relationship R.

[0108] Taking the determination of the target position of key point 8 as an example, the third position (X8, Y8, Z8) of key point 8 can be substituted into formula (6), combined with the target position (X8, Y8, Z8) of key point 0. W0 Y W0 Z W0 Determine the target location (X) of key point 8. W8 Y W8 Z W8 The relationship between ) and the third position (X8, Y8, Z8) of key point 8 is shown in formulas (13) to (15):

[0109] X W8 =R·(X8-X0)+X W0 (13)

[0110] Y W8 =R·(Y8-Y0)+Y W0 (14)

[0111] Z W8 =R·(Z8-Z0)+Z W0 (15)

[0112] Since the third positions (X0, Y0, Z0) of key point 0 and the third positions (X8, Y8, Z8) of key point 8 can be determined by the second hand image acquired by the near-eye display device, the near-eye display device can substitute the acquired second proportional relationship R, the third positions (X0, Y0, Z0) of key point 0 and the third positions (X8, Y8, Z8) of key point 8 into formulas (13) to (15) to obtain the target position (X) of key point 8. W8 Y W8 Z W8 ).

[0113] Similarly, near-eye display devices can also determine the target position (X) of key point 4 on the user's hand. W4 Y W4 Z W4 Of course, it is not limited to this. Near-eye display devices can determine the target position of all preset key points on the user's hand, and there are no restrictions on this.

[0114] Thus, when the target position of the first key point is determined, the target position of the first key point is equivalent to the position of the first key point on the user's hand in the world coordinate system corresponding to the near-eye display device. The target position can include the target depth, which helps improve the accuracy of the near-eye display device in recognizing the depth of the user's hand. Furthermore, the near-eye display device can subsequently use the target position of the first key point to perform corresponding interactive operations on the displayed virtual keyboard, which helps improve the convenience of subsequent interaction with the virtual keyboard on the near-eye display device.

[0115] For example, the target location includes the target depth. After determining the target location of the first keypoint, the target location of the first keypoint can be used by a near-eye display device to display the corresponding virtual effect.

[0116] In some implementations, a keypoint identifier of the first keypoint is displayed at the target location of the first keypoint; the display size of the keypoint identifier is inversely proportional to the target depth of the first keypoint.

[0117] For example, when a user interacts with a virtual keyboard displayed on a near-eye display device, the user typically interacts with the virtual keyboard through a first keypoint. Based on this, the near-eye display device can add corresponding virtual effects at the target position of the first keypoint to indicate to the user the current position of the first keypoint within the display range of the near-eye display device. The first keypoint can be preset or set by the user, and the number of first keypoints can be one or more, without limitation.

[0118] Taking the first key point, which includes key point 8 and key point 4, as an example, the near-eye display device can display the key point marker of key point 8 at the target location of key point 8. The near-eye display device can also display the key point marker of key point 4 at the target location of key point 4. The key point markers of key point 8 and key point 4 can be represented in the form of virtual effects. For example, the key point markers can be accompanied by visual effects such as marker color changing or marker diffusion; this is not limited here.

[0119] Accordingly, to display the target depth of the first key point, the near-eye display device can limit the display size of the key point marker of the first key point based on the target depth. The display size may include the length, width, radius, and diameter of the key point marker, etc., which are not limited here. For example, when the target depth of one of the two first key points is less than the target depth of the other, the display size of the key point marker of the one first key point displayed by the near-eye display device is larger than the display size of the key point marker of the other first key point. When the target depth of one of the two first key points is greater than the target depth of the other, the display size of the key point marker of the one first key point displayed by the near-eye display device is smaller than the display size of the key point marker of the other first key point. Accordingly, the larger the target depth, the smaller the display size of the key point marker; therefore, the display size of the first key point is inversely proportional to the target depth of the first key point.

[0120] In this way, the near-eye display device displays a keypoint marker based on the target location of the first keypoint. Users can then determine the current position of the first keypoint within the display area of ​​the near-eye display device based on this marker. When users need to interact with the virtual keyboard displayed on the near-eye display device, they can clearly identify the current interaction position based on the keypoint marker's display location, thus improving the ease of interaction with the virtual keyboard. Correspondingly, since the display size of the keypoint marker is inversely proportional to the target depth of the first keypoint, it enhances the user's intuitive experience of target depth, thereby improving the user's visual experience with the near-eye display device.

[0121] S105. Based on the target position of the first key point, perform the corresponding interactive operation on the virtual keyboard displayed on the near-eye display device.

[0122] For example, the first key point can include the tip of the thumb, the tip of the index finger, the tip of the middle finger, the tip of the ring finger, the tip of the little finger, and so on, without limitation.

[0123] In some implementations, when a user interacts with the virtual keyboard displayed on the near-eye display device, the user can extend corresponding fingers to press the keys on the virtual keyboard. Correspondingly, the user can also use corresponding fingers to make pinch or release gestures to change the display parameters of the virtual keyboard. When the user interacts with the virtual keyboard, the fingertips of the corresponding fingers on the user's hand will change position according to the user's gesture. Based on this, the near-eye display device can determine the user's interaction with the virtual keyboard based on the target position of the first key point, and then execute the corresponding interaction operation on the virtual keyboard, thereby improving the ease of interaction with the virtual keyboard on the near-eye display device.

[0124] For example, the interactive operations performed by a near-eye display device on a virtual keyboard may include adjusting the display parameters of the virtual keyboard. The display parameters of the virtual keyboard may include at least one of the following: display position and display size.

[0125] In some implementations, when the target positions of at least two first key points are both within the range of the first position corresponding to the virtual keyboard, a third distance between the at least two first key points is determined based on the target positions of the at least two first key points respectively; based on the third distance, at least one display parameter among the display position and display size of the virtual keyboard is adjusted.

[0126] For example, the first position range corresponding to the virtual keyboard can be used to indicate the position range of the virtual keyboard within the display range corresponding to the near-eye display device.

[0127] The first key point includes the tip of the index finger (e.g.) Figure 2 Key point 8), thumb tip point (e.g.) Figure 2 Taking key point 4 as an example. When the target positions of key point 8 and key point 4 are both within the first position range corresponding to the virtual keyboard, the near-eye display device can determine that the user needs to perform corresponding interactive operations on the virtual keyboard. Based on this, the near-eye display device can determine the third distance between key point 8 and key point 4 according to the target positions of key point 8 and key point 4. The near-eye display device can adjust at least one display parameter, such as the display position and display size of the virtual keyboard, according to the third distance. Of course, the first key point is not limited to the fingertip of the index finger, and the second key point is not limited to the fingertip of the thumb; no restrictions are imposed here.

[0128] For example, when the third distance is less than or equal to a preset distance threshold, the display position of the virtual keyboard is adjusted in response to the movement of the user's hand.

[0129] The first key point includes the tip of the index finger (e.g.) Figure 2 Key point 8), thumb tip point (e.g.)Figure 5 Take point 4) as an example. Figure 2 As shown, when the third distance between key point 8 and key point 4 is less than or equal to a preset distance threshold, the near-eye display device can determine that the user has made a pinch gesture. When the user makes a pinch gesture, the user can maintain the pinch gesture and move their hand to drag the virtual keyboard. Based on this, when the user makes a pinch gesture, the near-eye display device can determine that the user needs to drag the virtual keyboard to adjust its display position. Accordingly, the near-eye display device can adjust the display position of the virtual keyboard in response to the user's hand movement.

[0130] For example, when the third distance is greater than a preset distance threshold, the display size of the virtual keyboard is adjusted in response to the change in the third distance; wherein, the display size of the virtual keyboard is directly proportional to the change in the third distance.

[0131] The display size of the virtual keyboard may include at least one of the following: display area, display length, and display width, and is not limited herein. The first key point includes the fingertip of the index finger (e.g., Figure 2 Key point 8), thumb tip point (e.g.) Figure 6 Taking key point 4 as an example. When the third distance between key point 8 and key point 4 is greater than a preset distance threshold, the near-eye display device can determine that the user has made a release gesture. Figure 7 As shown, the user can increase the third distance between keypoint 8 and keypoint 4 by making a release gesture. When the near-eye display device detects that the third distance between keypoint 8 and keypoint 4 has increased, the near-eye display device can increase the display size of the virtual keyboard. Figure 2 As shown, the user can also reduce the third distance between keypoint 8 and keypoint 4 by making a release gesture. When the near-eye display device detects that the third distance between keypoint 8 and keypoint 4 has decreased, the near-eye display device can reduce the display size of the virtual keyboard. Based on this, the near-eye display device can adjust the display size of the virtual keyboard in response to changes in the third distance; wherein, the display size of the virtual keyboard and the change in the third distance are directly proportional.

[0132] Thus, when the target positions of at least two first key points are within the range of the first position corresponding to the virtual keyboard, the near-eye display device can adjust at least one display parameter among the display position and display size of the virtual keyboard according to the third distance between the at least two first key points, which helps to improve the ease of adjustment of the virtual keyboard by the near-eye display device.

[0133] For example, the interactive operations performed by a near-eye display device on a virtual keyboard may include pressing preset keys on the virtual keyboard. The near-eye display device can then display the input content corresponding to the key within its display area by pressing the key.

[0134] In some implementations, the target position includes a target depth. When the target position of the first key point is within the second position range corresponding to a preset key in the virtual keyboard, if the depth difference between the target depth of the first key point and the preset depth of the preset key first decreases to less than or equal to a preset depth difference threshold, and then increases to greater than the preset depth difference threshold, then a pressing operation is performed on the preset key to display the input information corresponding to the preset key.

[0135] For example, the second position range corresponding to the preset button can be used to indicate the position range of the preset button within the display range corresponding to the near-eye display device.

[0136] The first key point includes the tip of the index finger (e.g.) Figure 2 Taking key point 8 and preset keys, including key J, as an example, when key point 8 is within the second position range corresponding to key J in the virtual keyboard, the near-eye display device can determine that the user has pressed key J by passing through key point 8, based on the depth difference between the target depth of key point 8 and the preset depth of key J. This difference first decreases to less than or equal to a preset depth difference threshold, and then increases to greater than the preset depth difference threshold. For example, if the near-eye display device detects that key point 8 gradually approaches key J and then gradually moves away from key J, the near-eye display device can determine that the user has pressed key J by passing through key point 8. Similarly, if the near-eye display device detects that key point 8 gradually approaches the preset depth of key J and then gradually moves away from key J, the near-eye display device can determine that the user has pressed key J by passing through key point 8. When the near-eye display device determines that the user has pressed key J by passing through key 8, the near-eye display device can perform a press operation on key J to display the input information corresponding to key J. For example, displaying the character "J" within the display area corresponding to a near-eye display device, displaying an input box within the display area corresponding to a near-eye display device and displaying the character "J" within the input box, etc., are not limited here. The first key point includes the thumb tip (e.g., Figure 8 Taking key point 4 as an example, with the space bar as a preset key, when the target position of key point 4 is within the second position range corresponding to the space bar on the virtual keyboard, the near-eye display device can determine that the user has pressed the space bar through key point 4, based on the depth difference between the target depth of key point 4 and the preset depth of the space bar. This difference is first reduced to less than or equal to a preset depth difference threshold, and then increased to greater than the preset depth difference threshold. The space bar information corresponding to the space bar is then displayed within the display range corresponding to the near-eye display device.

[0137] Similarly, when a near-eye display device detects a user pressing a preset button, it can execute the pressing operation on the preset button, thereby displaying the input information of the preset button within the corresponding display area of ​​the near-eye display device.

[0138] In this way, when a user's first key point on their hand is within the second position range corresponding to a preset button, the near-eye display device can detect whether the change in the depth difference between the target depth of the first key point and the preset depth of the preset button conforms to the pattern of first decreasing to less than or equal to a preset depth difference threshold, and then increasing to greater than the preset depth difference threshold, to determine whether the user has pressed the preset button. When the near-eye display device detects that the user has pressed the preset button, it can perform a press operation on the preset button to display the input information corresponding to the preset button. The user can then view the input information corresponding to the preset button displayed on the near-eye display device, which improves both the interactivity and intuitiveness of the near-eye display device's display of input information.

[0139] The interaction method for a near-eye display device provided in the above embodiments includes: acquiring a first hand image captured by the near-eye display device, the first hand image including at least two first key points of the user's hand and an object between the at least two first key points, the size of the object in the direction corresponding to the at least two first key points being a preset size; determining a target length of the part of the user's hand corresponding to the at least two second key points based on the first position of the at least two first key points and the at least two second key points in the first hand image and the preset size; acquiring a second hand image captured by the near-eye display device; determining a target position of the first key point based on the second position and third position of the at least two second key points in the second hand image, the third position of the at least two first key points in the second hand image and the target length; the second position including 2D coordinates and the third position including 2.5D coordinates; and performing a corresponding interaction operation on a virtual keyboard displayed by the near-eye display device based on the target position of the first key point.

[0140] Since the preset size of the object is its actual size in the user's space, it can be used by the near-eye display device to determine the first distance between at least two first key points. Correspondingly, the near-eye display device can combine the first hand image and the preset size to determine the target length of the corresponding parts of at least two second key points. Because the target length of this part does not change with the user's gestures, when the near-eye display device subsequently captures a second image of the user's hand, it can combine the target length to determine the target position of the corresponding key point on the user's hand. For example, the near-eye display device can combine the target length and the second hand image to determine the target position of the first key point. Since the target position can include the target depth, this improves the accuracy of the near-eye display device's depth recognition of the user's hand. The target position of the first key point allows the near-eye display device to perform corresponding interactive operations on the virtual keyboard, thus improving the ease of interaction with the virtual keyboard.

[0141] Please see Figure 8 , Figure 8 This is a schematic block diagram of an interaction device for a near-eye display device according to an embodiment of this application. The interaction device is used to execute the aforementioned interaction method for the near-eye display device. Near-eye display devices may include AR glasses, VR glasses, AR helmets, VR helmets, etc., and are not limited thereto.

[0142] like Figure 9 As shown, the interactive device includes a first image acquisition module 110, a distance determination module 120, a second image acquisition module 130, a target location determination module 140, and an interaction module 150.

[0143] The first image acquisition module 110 is used to acquire a first hand image captured by a near-eye display device. The first hand image includes at least two first key points of the user's hand and an object between the at least two first key points. The size of the object in the direction corresponding to the at least two first key points is a preset size.

[0144] The distance determination module 120 is used to determine the target length of the part of the user's hand corresponding to the at least two second key points based on the first position of the at least two first key points and the at least two second key points of the user's hand in the first hand image and the preset size.

[0145] The second image acquisition module 130 is used to acquire a second hand image obtained by the near-eye display device from the user's hand.

[0146] The target position determination module 140 is used to determine the target position of the first key point based on the second position and third position of each of the at least two second key points in the second hand image, the third position of each of the at least two first key points in the second hand image, and the target length; the second position includes 2D coordinates, and the third position includes 2.5D coordinates.

[0147] The interaction module 150 is used to perform corresponding interactive operations on the virtual keyboard displayed by the near-eye display device according to the target position of the first key point.

[0148] For example, the distance determination module 120 includes a first ratio establishment submodule and a first calculation submodule.

[0149] The first ratio establishment submodule is used to determine a first proportional relationship between the preset size and the target length based on the ratio between the first distance between the at least two first key points and the second distance between the at least two second key points; wherein the first distance is determined based on the first position of each of the at least two first key points, and the second distance is determined based on the first position of each of the at least two second key points.

[0150] The first calculation submodule is used to substitute the first positions of the at least two first key points and the at least two second key points, as well as the preset size, into the first proportional relationship to obtain the target length.

[0151] For example, the target location determination module 140 includes a projection relationship determination submodule, a target distance determination submodule, a second scale establishment submodule, a first target location determination submodule, and a second target location determination submodule.

[0152] The projection relationship determination submodule is used to determine the projection relationship between the target position of the second key point and the second position of the second key point based on the camera intrinsic parameters of the near-eye display device and the second position of the second key point.

[0153] The target distance determination submodule is used to determine the target distance between the target positions of the at least two second key points based on the target length.

[0154] The second ratio establishment submodule is used to determine a second ratio relationship between a first difference and a second difference based on the target length and the third positions of the at least two second key points respectively; wherein the first difference is determined based on the difference between the target positions of the at least two second key points respectively, and the second difference is determined based on the difference between the third positions of the at least two second key points respectively.

[0155] The first target location determination submodule is used to determine the target location of each of the at least two second key points based on the projection relationship, the target distance, and the second proportional relationship.

[0156] The second target location determination submodule is used to determine the target location of the first key point based on the target location of the second key point, the third location of the first key point, and the second proportional relationship.

[0157] For example, the interaction module 150 includes a distance determination submodule and an adjustment submodule.

[0158] The distance determination submodule is used to determine a third distance between the at least two first key points based on their respective target positions when the target positions of the at least two first key points are both within the first position range corresponding to the virtual keyboard.

[0159] The adjustment submodule is used to adjust at least one of the display parameters, namely the display position and display size of the virtual keyboard, according to the third distance.

[0160] For example, the adjustment submodule includes a position adjustment submodule.

[0161] The position adjustment submodule is used to adjust the display position of the virtual keyboard in response to the movement of the user's hand when the third distance is less than or equal to a preset distance threshold.

[0162] For example, the resizing submodule includes a resizing submodule.

[0163] The size adjustment submodule is used to adjust the display size of the virtual keyboard in response to the change of the third distance when the third distance is greater than a preset distance threshold; wherein the display size of the virtual keyboard is directly proportional to the change of the third distance.

[0164] For example, the target location includes the target depth. The interaction module 150 includes an information input submodule.

[0165] The information input submodule is used to perform a pressing operation on the preset key when the target position of the first key point is within the second position range corresponding to the preset key in the virtual keyboard. If the depth difference between the target depth of the first key point and the preset depth of the preset key first decreases to less than or equal to the preset depth difference threshold, and then increases to greater than the preset depth difference threshold, the input information corresponding to the preset key is displayed.

[0166] For example, the interactive device includes a display depth determination submodule and a display module.

[0167] The display depth determination submodule is used to determine the target display depth of the near-eye display device for the virtual keyboard based on a multiple of the target length.

[0168] The display submodule is used to control the near-eye display device to display the virtual keyboard at the target display depth.

[0169] For example, the interactive device includes an identification display submodule.

[0170] The identification display submodule is used to display the key point identifier of the first key point at the target location of the first key point; the display size of the key point identifier is inversely proportional to the target depth of the first key point.

[0171] It should be noted that those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the above-described apparatus and its modules and units can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0172] The method of this application can be used in a wide variety of general-purpose or special-purpose computer system environments or configurations. Examples include: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics devices, network PCs, minicomputers, mainframe computers, and distributed computing environments including any of the above systems or devices. This application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform specific tasks or implement specific abstract data types. This application can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0173] For example, the above-described method and apparatus can be implemented as a computer program that runs on a near-eye display device to perform corresponding interactive operations on a virtual keyboard via the near-eye display device. For example, the near-eye display device may include VR glasses, AR glasses, VR headsets, AR headsets, etc., and is not limited thereto.

[0174] Please see Figure 9 , Figure 9 This is a schematic block diagram of the structure of a near-eye display device provided in an embodiment of this application.

[0175] like Figure 9As shown, the near-eye display device includes a memory and a processor. The memory and processor can be connected via a system bus, and the memory may include a storage medium and internal memory.

[0176] The storage medium can store the operating system and computer programs. When the computer program is executed, it enables the processor to perform any interactive method of the near-eye display device.

[0177] The processor provides computing and control capabilities to support the operation of the entire near-eye display device.

[0178] Internal memory provides an environment for the execution of computer programs stored in storage media. When these computer programs are executed by a processor, the processor can perform any interactive method of a near-eye display device.

[0179] Those skilled in the art will understand that ​ The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the near-eye display device to which the present application is applied. A specific near-eye display device may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0180] It should be understood that a processor can be a Central Processing Unit (CPU), but it can also be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other convertible logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Among these, a general-purpose processor can be a microprocessor or any conventional processor.

[0181] In one embodiment, the processor is configured to execute a computer program and, when executing the computer program, perform the following steps:

[0182] A first hand image captured by a near-eye display device is acquired. The first hand image includes at least two first key points of the user's hand and an object between the at least two first key points. The size of the object in the direction corresponding to the at least two first key points is a preset size.

[0183] Based on the first position of each of the at least two first key points and the at least two second key points of the user's hand in the first hand image and the preset size, determine the target length of the part of the user's hand corresponding to the at least two second key points;

[0184] Acquire a second hand image of the user's hand captured by the near-eye display device;

[0185] Based on the second and third positions of the at least two second key points in the second hand image, the third position of the at least two first key points in the second hand image, and the target length, the target position of the first key point is determined; the second position includes 2D coordinates, and the third position includes 2.5D coordinates.

[0186] Based on the target position of the first key point, perform corresponding interactive operations on the virtual keyboard displayed by the near-eye display device.

[0187] It should be noted that those skilled in the art will understand that, for the sake of convenience and brevity, the specific working process of the interaction of the near-eye display device described above can be referred to the corresponding process in the aforementioned embodiments of the interaction method of the near-eye display device, and will not be repeated here.

[0188] This application also provides a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, the method implemented can be referred to various embodiments of the interaction method of the near-eye display device of this application.

[0189] The computer-readable storage medium can be an internal storage unit of the near-eye display device described in the foregoing embodiments, such as the hard disk or memory of the near-eye display device. Alternatively, the computer-readable storage medium can be an external storage device of the near-eye display device, such as a plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, etc., provided on the near-eye display device.

[0190] It should be understood that the terminology used in this specification is for the purpose of describing particular embodiments only and is not intended to limit the scope of the application. As used in this specification and the appended claims, the singular forms “a,” “an,” and “the” are intended to include the plural forms unless the context clearly indicates otherwise.

[0191] It should also be understood that the term "and / or" as used in this specification and the appended claims refers to any combination and all possible combinations of one or more of the associated listed items, and includes such combinations. It should be noted that, herein, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or system that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or system. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or system that includes that element.

[0192] The sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments. The above descriptions are merely specific implementations of this application, but the scope of protection of this application is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in this application, and these modifications or substitutions should all be covered within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A method of interaction for a near-eye display device, the method comprising: The interaction method comprises: obtaining a first hand image captured by a near-eye display device, the first hand image comprising at least two first key points of a user's hand and an object between the at least two first key points, the object having a preset size in a direction corresponding to the at least two first key points; determining a target length of a part of the user's hand corresponding to at least two second key points according to the at least two first key points, the at least two second key points, first positions of the at least two first key points and the at least two second key points in the first hand image, and the preset size; obtaining a second hand image of the user's hand captured by the near-eye display device; determining a target position of the first key points according to second positions of the at least two second key points in the second hand image, third positions of the at least two second key points in the second hand image, third positions of the at least two first key points in the second hand image, and the target length; the second positions comprising 2D coordinates, the third positions of the first key points comprising 2.5D coordinates of the first key points, and the third positions of the second key points comprising 2.5D coordinates of the second key points; performing a corresponding interaction operation on a virtual keyboard displayed by the near-eye display device according to the target position of the first key points.

2. The interaction method of claim 1, wherein, The determining of the target length of the part of the user's hand corresponding to the at least two second key points according to the at least two first key points, the at least two second key points, the first positions of the at least two first key points and the at least two second key points in the first hand image, and the preset size comprises: determining a first proportional relationship between the preset size and the target length according to a ratio between a first distance between the at least two first key points and a second distance between the at least two second key points; the first distance being determined according to the first positions of the at least two first key points, and the second distance being determined according to the first positions of the at least two second key points; substituting the first positions of the at least two first key points, the first positions of the at least two second key points, and the preset size into the first proportional relationship to obtain the target length.

3. The interaction method of claim 1, wherein, The determining of the target position of the first key points according to the second positions of the at least two second key points in the second hand image, the third positions of the at least two second key points in the second hand image, the third positions of the at least two first key points in the second hand image, and the target length comprises: determining a projection relationship between the target positions of the at least two second key points and the second positions of the corresponding second key points according to camera intrinsic parameters of the near-eye display device and the second positions of the at least two second key points; determining a target distance between the target positions of the at least two second key points according to the target length. determine a second proportional relationship between a first difference value and a second difference value according to the target length and the third positions of the at least two second key points, wherein the first difference value is determined according to a difference between the target positions of the at least two second key points, and the second difference value is determined according to a difference between the third positions of the at least two second key points; determine the target positions of the at least two second key points according to the projection relationship, the target distance, and the second proportional relationship; determine the target position of the first key point according to one of the target positions of the at least two second key points, the third position of the first key point, and the second proportional relationship.

4. The interaction method according to any one of claims 1 to 3, characterized in that, the corresponding interaction operation performed on the virtual keyboard displayed by the near-eye display device according to the target position of the first key point, comprises: when the target positions of the at least two first key points are all within the first position range corresponding to the virtual keyboard, determine a third distance between the at least two first key points according to the target positions of the at least two first key points; adjust at least one display parameter of the virtual keyboard, such as a display position or a display size, according to the third distance.

5. The interaction method of claim 4, wherein, the adjustment of the display position of the virtual keyboard according to the third distance, comprises: when the third distance is less than or equal to a preset distance threshold, adjust the display position of the virtual keyboard in response to the movement of the user's hand.

6. The interaction method of claim 4, wherein, the adjustment of the display size of the virtual keyboard according to the third distance, comprises: when the third distance is greater than a preset distance threshold, adjust the display size of the virtual keyboard in response to the change of the third distance, wherein the display size of the virtual keyboard and the change of the third distance are in a positive proportional relationship.

7. The interaction method according to any one of claims 1 to 3, characterized in that, the target position comprises a target depth; the corresponding interaction operation performed on the virtual keyboard displayed by the near-eye display device according to the target position of the first key point, comprises: when the target position of the first key point is within a second position range corresponding to a preset key in the virtual keyboard, if a depth difference between the target depth of the first key point and a preset depth of the preset key first decreases to be less than or equal to a preset depth difference threshold and then increases to be greater than the preset depth difference threshold, perform a pressing operation on the preset key to display input information corresponding to the preset key.

8. The interaction method of any one of claims 1 to 3, characterized in that, after the target length of the part corresponding to the at least two second key points in the user's hand is determined according to the first positions of the at least two first key points and the at least two second key points in the first hand image and the preset size, the interaction method further comprises: determine a target display depth of the virtual keyboard displayed by the near-eye display device according to a multiple of the target length; control the near-eye display device to display the virtual keyboard at the target display depth.

9. The interaction method according to any one of claims 1 to 3, characterized in that, the target position comprises a target depth; After the target position of the first key point is determined according to the second positions of the at least two second key points in the second hand image, the third positions of the at least two second key points in the second hand image, the third positions of the at least two first key points in the second hand image, and the target length, the interaction method further includes: A key point identifier of the first key point is displayed at the target position of the first key point, and a display size of the key point identifier is inversely proportional to the target depth of the first key point.

10. An interaction device for a near-eye display device, the interaction device comprising: The interaction device includes: A first image acquisition module configured to acquire a first hand image captured by a near-eye display device, the first hand image including at least two first key points of a user's hand and an object between the at least two first key points, the object having a preset size in a direction corresponding to the at least two first key points; A distance determination module configured to determine a target length of a part of the user's hand corresponding to at least two second key points according to the at least two first key points of the user's hand, first positions of the at least two second key points in the first hand image, and the preset size; A second image acquisition module configured to acquire a second hand image of the user's hand captured by the near-eye display device; A target position determination module configured to determine a target position of a first key point according to second positions of the at least two second key points in the second hand image, third positions of the at least two second key points in the second hand image, third positions of the at least two first key points in the second hand image, and the target length; the second positions including 2D coordinates, the third positions of the first key points including 2.5D coordinates of the first key points, and the third positions of the second key points including 2.5D coordinates of the second key points; An interaction module configured to perform a corresponding interaction operation on a virtual keyboard displayed by the near-eye display device according to the target position of the first key point.

11. A near-eye display device, comprising: The near-eye display device includes a memory and a processor; The memory is configured to store a computer program; The processor is configured to execute the computer program and implement steps of the interaction method of the near-eye display device according to any one of claims 1 to 9 when the computer program is executed.

12. A computer-readable storage medium having stored thereon a computer program, characterized in that The computer program is executed by the processor to implement steps of the interaction method of the near-eye display device according to any one of claims 1 to 9.

Images