Head-mounted display, control method, and non-transitory computer readable storage medium thereof

Abstract

A head-mounted display, a control method, and a non-transitory computer-readable recording medium thereof are disclosed. The head-mounted display calculates a hand gesture based on a real-time image, wherein the gesture includes a reference point located on the hand. In response to initiating an input operation based on the reference point and a relative position of a virtual object displayed on the display, the head-mounted display selects a first gesture corresponding to the reference point from a plurality of gestures based on a plurality of inertial measurement parameters received from the wearable device and the reference point. The head-mounted display generates an input event corresponding to the virtual object based on the first gesture and an input target corresponding to the input operation.

Description

[Technical Field]

[0001] This disclosure relates to a head-mounted display, a control method, and a non-transitory computer-readable storage medium thereof, and particularly to a head-mounted display, a control method, and a non-transitory computer-readable storage medium thereof that combines image recognition and inertial measurement parameters to generate control signals. [Previous Technology]

[0002] In recent years, various technologies related to virtual reality have developed rapidly, and various head-mounted display technologies and applications have been proposed one after another.

[0003] In the prior art, when a user wears a head-mounted display, the head-mounted display obtains the data that the user wants to input (e.g., input text) by detecting the user's typing gestures on a virtual keyboard, or the user's wearable device receives physical key press signals to obtain input data.

[0004] However, natural hand tremors or image recognition delays can cause jittering or recognition errors, resulting in a poor user experience.

[0005] In addition, with the popularization of smartphones, they have gradually become the most frequently used electronic products, and the posture of typing on a mobile phone has also become a familiar human-computer interaction mode for the public.

[0006] In view of this, how to make head-mounted displays provide a more stable input method and reduce the difficulty of operating head-mounted displays according to the user's habits is an urgent goal for the industry. [Summary of the Invention]

[0007] To address the above problems, this disclosure provides a head-mounted display comprising a display, a communication interface, a camera, and a processor. The communication interface is communicatively connected to a wearable device. The camera is used to capture a real-time image of a hand worn by a user. The processor is coupled to the display, the communication interface, and the camera to perform the following operations: calculating a posture corresponding to the hand based on the real-time image, wherein the posture includes a reference point located on the hand; in response to determining and initiating an input operation based on the reference point and a relative position of a virtual object displayed on the display; selecting a first gesture corresponding to the reference point from a plurality of gestures based on a plurality of inertial measurement parameters received from the wearable device and the reference point; and generating an input event corresponding to the virtual object based on the first gesture and an input target corresponding to the input operation.

[0008] This disclosure also provides a control method applicable to an electronic device, wherein the electronic device is communicatively connected to a wearable device, the steps of which include: capturing a real-time image of a hand of a user wearing the wearable device; calculating a posture corresponding to the hand based on the real-time image, wherein the posture includes a reference point located on the hand; determining to initiate an input operation in response to a relative position of the reference point and a virtual object displayed on a display; selecting a first gesture corresponding to the reference point from a plurality of gestures based on a plurality of inertial measurement parameters received from the wearable device and the reference point; and generating an input event corresponding to the virtual object based on the first gesture and an input target corresponding to the input operation.

[0009] This disclosure also provides a non-transitory computer-readable storage medium having at least one instruction stored thereon, which, when executed by a processor, performs a control method comprising the steps of: capturing a real-time image of a wearable device worn on a user's hand; calculating a posture corresponding to the hand based on the real-time image, wherein the posture includes a reference point located on the hand; determining to initiate an input operation in response to a relative position of the reference point and a virtual object displayed on a display; selecting a first gesture corresponding to the reference point from a plurality of gestures based on a plurality of inertial measurement parameters received from the wearable device and the reference point; and generating an input event corresponding to the virtual object based on the first gesture and an input target corresponding to the input operation.

[0010] It should be understood that the foregoing general description and the following specific description are merely exemplary and explanatory, and are intended to provide further explanation of the claimed disclosure.

Implementation Method

[0012] To make the description of this disclosure more detailed and complete, reference can be made to the accompanying drawings and the various embodiments described below, in which the same numbers represent the same or similar elements.

[0013] First, let me explain the applicable scenario of this embodiment. The schematic diagram is shown in Figure 1. As shown in Figure 1, in the application environment of the present invention, a user U can use a head-mounted display 2 and wear a wearable device 3 on a body part (for example, the user U wears a smart ring on the index finger of the left hand) to perform control operations (for example, applications) corresponding to the display screen of the head-mounted display 2.

[0014] In a first embodiment of the present invention, the control signal generation system 1 includes a head-mounted display 2 and a wearable device 3, the head-mounted display 2 being communicatively connected to the wearable device 3. The control signal generation system 1 is used to generate input control signals based on an image of the user's hand and inertial measurement parameters of the hand.

[0015] In this embodiment, the architecture of the head-mounted display 2 is shown in Figure 2. The head-mounted display 2 includes a processor 22, a camera 24, a communication interface 26 and a display 28, wherein the processor 22 is coupled to the camera 24, the communication interface 26 and the display 28.

[0016] In some embodiments, camera 24 may include a plurality of image capturing units (e.g., a plurality of depth camera lenses) for capturing real-time images of a wearable device 3 worn on the hand of a user U.

[0017] In some embodiments, the camera 24 is used to capture a real-time image corresponding to a field of view (FOV), wherein the hand of the user U wearing the wearable device 3 is included in the field of view.

[0018] The communication interface 26 is used to communicate with the wearable device 3 and receive multiple inertial measurement parameters corresponding to the user's U hand from the wearable device 3.

[0019] The display 28 is used to display images and provide an interface for user U to interact. In some embodiments, the display 28 displays virtual objects in space to allow user U to view and interact with the virtual objects.

[0020] In some embodiments, the wearable device 3 includes an inertial measurement unit for detecting inertial measurement parameters of the hand of the user U wearing the wearable device 3. Specifically, the inertial measurement unit continuously generates a plurality of inertial measurement parameters (e.g., an inertial measurement parameter stream generated at a frequency of 10 times per second), and each of these inertial measurement parameters may include an acceleration, a rotational magnitude, and an angular acceleration. During operation, the head-mounted display 2 may periodically receive these inertial measurement parameters from the wearable device 3.

[0021] It should be noted that the inertial measurement parameters generated by the wearable device 3 can be mapped to the hand of the user U. For example, the user U can wear the wearable device 3 on any finger to collect data. For ease of explanation, in this embodiment, the user U wearing the wearable device 3 on the index finger will be used for illustration.

[0022] In some embodiments, the communication interface 26 is an interface that can receive and transmit data or other interfaces that can receive and transmit data that are known to those skilled in the art to which this invention pertains. The communication interface 26 can receive data from sources such as external devices, external web pages, external applications, etc.

[0023] In some embodiments, the processor 22 may include a central processing unit (CPU), a graphics processing unit (GPU), a multiprocessor, a distributed processing system, an application-specific integrated circuit (ASIC), and / or a suitable arithmetic unit.

[0024] It should be noted that Figure 1 is only for illustrative purposes, and the present invention does not limit the content of the control signal generation system 1. For example, the present invention does not limit the number of wearable devices 3 connected to the head-mounted display 2. The head-mounted display 2 can be connected to multiple wearable devices simultaneously via a network, depending on the scale of the control signal generation system 1 and actual needs.

[0025] For details on how the head-mounted display 2 generates control signals, please refer to Figure 3.

[0026] After the camera 24 acquires the real-time image, the processor 22 executes operation OP1 to calculate the posture of the user U's hand based on the real-time image. Specifically, the processor 22 calculates a posture corresponding to the hand based on the real-time image, wherein the posture includes a reference point located on the hand.

[0027] In some embodiments, the operation of the processor 22 to calculate the pose further includes: the processor 22 calculating a plurality of key points of the hand in the live image; and the processor 22 generating the pose of the hand in a three-dimensional space based on the key points and a depth information in the live image.

[0028] In some embodiments, the operation of the processor 22 in calculating the pose further includes: the processor 22 selecting one of the key points as the reference point.

[0029] For details regarding the calculation of hand pose, please refer to Figure 4. As shown in the figure, after obtaining the real-time image RI captured by the camera 24, the processor 22 performs image processing on the real-time image RI to generate key points KP and depth information DI of the user U's hand.

[0030] For example, the processor 22 uses an image recognition model to identify multiple joint positions of the hand as key points in the real-time image, such as the positions of the palm, fingers, knuckles, and base of the fingers. On the other hand, when the camera 24 includes a depth camera lens, the real-time image RI it captures contains depth information DI corresponding to each pixel. Furthermore, the processor 22 combines the calculated multiple key points with the depth information DI in the real-time image RI to confirm the position of the key points in three-dimensional space. Accordingly, the processor 22 can obtain the hand pose HP of the user U in three-dimensional space.

[0031] It should be noted that the reference point is selected by the head-mounted display 2 from multiple key points KP and is used to confirm the reference point of the user's operation object. Its function is similar to the cursor of a personal computer or the click position of the user on the touch screen.

[0032] Referring to Figure 5A, in some embodiments, the head-mounted display 2 selects the index finger of the user's hand H wearing the wearable device 3 as the reference point RP. As shown, the user U wears the wearable device 3 on his thumb, and can operate the head-mounted display 2 by loosely holding the device with his palm and tapping the side of his index finger with his thumb. This hand posture is similar to the way a smartphone is operated, where the user holds the phone with his palm and four fingers and taps the screen with his thumb. Accordingly, the head-mounted display 2 can provide the user with a familiar way to input information. In addition, since the user U's thumb is usually facing the user U's head and not obstructed by other fingers when making the gesture shown in Figure 5A, the head-mounted display 2 can more easily capture the real-time image of the reference point.

[0033] In some embodiments, the virtual object is a virtual keyboard that includes a plurality of keys, and the input target corresponds to one of these keys.

[0034] Please return to Figure 3. After calculating the user U's hand posture, the head-mounted display 2 executes operation OP2 to determine whether the virtual keyboard has been activated. If the virtual keyboard has been activated, it means that the head-mounted display 2 is in input mode, and further determines the input information based on the user U's hand posture and gestures; otherwise, if the virtual keyboard has not been activated, the head-mounted display 2 is not in input mode, and returns to operation OP1 to continue determining the user U's hand posture.

[0035] It should be noted that the operation OP2 in this embodiment is used as an example to determine whether the virtual keyboard has been started. In other embodiments, the head-mounted display 2 may skip this operation or determine whether it is necessary to further determine the information input by the user U through other operations, such as determining whether the menu is opened.

[0036] If it is determined that the virtual keyboard has been activated, the head-mounted display 2 determines the function or information that the user U wants to operate based on the position of the user U's hand. Specifically, the processor 22 determines whether to initiate an input operation based on the reference point and the relative position of a virtual object displayed on the display 28.

[0037] In the embodiment illustrated in Figure 3, the head-mounted display 2 first performs operation OP3 to determine whether the reference point RP is located in the vertical extension area of ​​the keys on the virtual keyboard.

[0038] Specifically, the processor 22 determines whether the relative position of the reference point is located in a vertically extending region of the virtual object, wherein the vertically extending region is generated by a distance vertically extending from a plurality of sub-objects in the virtual object; and in response to the relative position of the reference point being located in the vertically extending region of the virtual object, the processor 22 decides to initiate the input operation.

[0039] Please refer to Figure 5B, which is a schematic diagram of the vertical extension region VS corresponding to key K in some embodiments of this disclosure. As shown in the figure, the range within the vertical distance d of key K on the virtual keyboard constitutes the vertical extension region VS. Accordingly, processor 22 determines whether the reference point RP on the user U's hand is located within the multiple vertical extension regions formed by multiple keys on the virtual keyboard. If so, processor 22 determines that user U may interact with the virtual keyboard next, and therefore initiates input operation.

[0040] In addition, the head-mounted display 2 selects the input target of the user U based on the position of the reference point. For example, when the reference point RP is located in the vertical extension area VS of the button K, the processor 22 determines that the user U may interact with the button K next, and therefore uses the button K as the input target.

[0041] Specifically, when the first gesture is a click gesture, the input target is generated based on the following operations: the processor 22 calculates a projection point of the reference point on a virtual plane corresponding to the virtual object; and the processor 22 selects a first sub-object from a plurality of sub-objects corresponding to the virtual object as the input target based on the projection point.

[0042] For example, when the virtual object is a virtual keyboard, the processor 22 calculates the position of the projection point of the user's thumb (i.e., the reference point RP) on the plane formed by the virtual keyboard, and selects the corresponding key according to the position of the projection point on the virtual keyboard.

[0043] It should be noted that, for ease of explanation, this disclosure uses a virtual keyboard as an example of a virtual object. However, in other embodiments, the head-mounted display 2 may also use other objects as virtual objects (e.g., menus, dashboards).

[0044] When the reference point RP is located within the vertical extension area VS, it means that its projection point falls on the button K, and the reference point RP is located within the vertical extension area VS at a distance d from the button K. Based on this, the head-mounted display 2 determines that the user U may select button K. Conversely, if the reference point RP is not located within the vertical extension area VS, the head-mounted display 2 determines that the user U may select other buttons or has not interacted with the virtual keyboard.

[0045] It should be noted that the head-mounted display 2 can be applied to multiple keys in the virtual keyboard in the same way to determine the keys that the user U may interact with next.

[0046] Please return to Figure 3. If the reference point RP is not located within the vertical extension area VS, the head-mounted display 2 returns to operation OP1 to continue judging the user U's hand posture.

[0047] On the other hand, if the reference point RP is located within the vertically extending region VS, the head-mounted display 2 performs operation OP4, marking the button corresponding to the reference point RP. Specifically, in response to deciding to initiate the input operation, the processor 22 marks one of the sub-objects closest to the reference point.

[0048] In the embodiment illustrated in Figure 5B, the processor 22 can mark the button K by means of illumination, different colors, etc., to provide the user U with confirmation of the button corresponding to the current hand posture. It should be noted that in some embodiments, operation OP4 can be selectively omitted.

[0049] After the input operation is initiated, the head-mounted display 2 performs operation OP5, which identifies the user U's gesture through the inertial measurement parameters transmitted by the wearable device 3. Specifically, in response to the decision to initiate the input operation, the processor 22 selects a first gesture corresponding to the reference point from the plurality of gestures based on the plurality of inertial measurement parameters received from the wearable device and the reference point.

[0050] In some embodiments, the processor 22 inputs the inertial measurement parameters into a classification model to select the first gesture from the gestures. Specifically, the classification model is a trained machine learning model used to classify the user U's hand gesture into one of a plurality of known gestures based on the inertial measurement parameters. Further, the input events include a click event, a double click event, and a finger flick event.

[0051] In some embodiments, the head-mounted display 2 acquires inertial measurement parameters generated within a certain period of time (e.g., within 50 milliseconds) after the input operation is initiated to determine the user U's gesture. Specifically, in response to deciding to initiate the input operation, the processor 22 receives the inertial measurement parameters acquired within a time interval after the input operation is initiated from the wearable device 3.

[0052] After recognizing the gesture of the user U, the head-mounted display 2 triggers the corresponding event based on the gesture and the button interaction of the user U. As shown in Figure 3, the processor 22 may recognize n kinds of gestures such as click gesture, double click gesture and finger flick gesture (corresponding to operation OP61 to OP6n respectively), where n is a positive integer.

[0053] Further, the head-mounted display 2 triggers an event corresponding to the user's U-gesture and generates a corresponding control signal. Specifically, the processor 22 generates an input event corresponding to the virtual object based on the first gesture and an input target corresponding to the input operation.

[0054] Through the above operation, user U can trigger different functions through different hand gestures. Taking Figure 5A as an example, user U can trigger a click event by tapping the side of the index finger once with the thumb (i.e., a tap gesture), and trigger a double click event by tapping the side of the index finger twice with the thumb (i.e., a double tap gesture). Taking Figure 6 as an example, user U can also trigger a finger-flick event by pinching the tips of the thumb and index finger together and then flicking the index finger in the direction F (i.e., a finger-flick gesture).

[0055] In one example, when the user U makes a click gesture (operation OP61), the head-mounted display 2 performs operation OP71, triggering a click event.

[0056] The processor 22 first confirms the child object (e.g., key K) that the user U interacts with through the aforementioned operation of selecting the input target, and then further executes operation OP8 to generate a control signal for clicking key K (e.g., typing input).

[0057] In another example, at least one boundary position of the virtual object includes a virtual label, and when the first gesture is a double-click gesture, generating the input event corresponding to the virtual object further includes the following operations: the processor 22 determines whether the reference point is located at one of the spatial positions of the virtual label; and in response to the reference point being located at the spatial position of the virtual label, the processor 22 generates the input event corresponding to the virtual object based on one of the displacement distances of the double-click gesture.

[0058] In some embodiments, the input event includes a zoom-in / zoom-out operation or a virtual object drag operation.

[0059] For example, referring to Figure 7, when the user U makes a double-click gesture (operation OP62), and it is determined that the user U is interacting with the virtual label VT of the virtual keyboard VK (i.e., the reference point RP is located in the vertical extension area of ​​the virtual label VT), the head-mounted display 2 executes operation OP72, triggering the double-click event of the virtual label VT, and further executes operation OP8 to generate a control signal for the double-click of the virtual label VT. Accordingly, the head-mounted display 2 adjusts the size and / or position of the virtual keyboard VK according to the movement of the user U's hand.

[0060] In another example, when the first gesture is a snap gesture, the input event generated for the virtual object further includes the following operations: the processor 22 moves the virtual object to an initial position; and the processor 22 adjusts the size of one of the virtual objects.

[0061] For example, when the user U makes a snap gesture (operation OP6n), the head-mounted display 2 executes operation OP7n to trigger the snap gesture event, and further executes operation OP8 to generate a control signal for the snap gesture (e.g., resetting the virtual keyboard VK). Accordingly, the head-mounted display 2 resets the virtual keyboard VK to a preset position and / or preset size.

[0062] It should be noted that the types, number, and corresponding functions of gestures in this embodiment are only for illustrative purposes. In actual applications, the head-mounted display 2 can be set with one or more gestures and correspond to one or more functions. In addition, the same gesture can correspond to different interactive blocks (e.g., different keys on a virtual keyboard), and the head-mounted display 2 will also trigger different events and generate different control signals (e.g., inputting different characters when typing).

[0063] In summary, the control signal generation system 1 disclosed herein uses real-time images captured by the head-mounted display 2 to determine the user's hand posture, and then uses inertial measurement parameters obtained by the wearable device 3 to determine the user's hand movements (i.e., gestures). The control signal generation system 1 can solve two problems: the low accuracy of image recognition in recognizing subtle hand movements, and the difficulty in locating the hand position in three-dimensional space using only inertial measurement parameters. By combining the two technologies, the control signal generation system 1 can more accurately determine the user's input information, while providing a more user-friendly and easier-to-use human-computer interaction experience.

[0064] Please refer to Figure 8, which is a flowchart of the control method 400 in the second embodiment of this disclosure. The control method 400 includes steps S401 to S405. The control method 400 is used to generate an input control signal based on an image of the user's hand and inertial measurement parameters of the hand. The control method 400 can be executed by an electronic device (e.g., the head-mounted display 2 in the first embodiment), wherein the electronic device is communicatively connected to a wearable device.

[0065] First, in step S401, the electronic device captures a real-time image of the wearable device being worn on the hand of a user.

[0066] Next, in step S402, the electronic device calculates a posture corresponding to the hand based on the real-time image, wherein the posture includes a reference point located on the hand.

[0067] Next, in step S403, in response to a decision to initiate an input operation based on the reference point and a relative position of a virtual object displayed on a display, the electronic device selects a first gesture corresponding to the reference point from a plurality of gestures based on a plurality of inertial measurement parameters received from the wearable device and the reference point.

[0068] Finally, in step S404, the electronic device generates an input event corresponding to the virtual object based on the first gesture and the input target corresponding to the input operation.

[0069] In some embodiments, step S402 further includes the electronic device calculating a plurality of key points of the hand in the live image; and the electronic device generating the pose of the hand in a three-dimensional space based on the key points and depth information in the live image.

[0070] In some embodiments, step S402 further includes the electronic device selecting one of the key points as the reference point.

[0071] In some embodiments, the control method 400 further includes the electronic device determining whether the relative position of the reference point is located in a vertically extending region of the virtual object, wherein the vertically extending region is generated by a distance vertically extending from a plurality of sub-objects in the virtual object; and in response to the relative position of the reference point being located in the vertically extending region of the virtual object, the electronic device deciding to initiate the input operation.

[0072] In some embodiments, when the first gesture is a click gesture, the input target is generated based on the following steps: calculating a projection point of the reference point on a virtual plane corresponding to the virtual object; and selecting a first sub-object from a plurality of sub-objects corresponding to the virtual object as the input target based on the projection point.

[0073] In some embodiments, at least one boundary position of the virtual object includes a virtual label, and when the first gesture is a double-click gesture, step S405 further includes the electronic device determining whether the reference point is located at one of the spatial positions of the virtual label; and in response to the reference point being located at the spatial position of the virtual label, the electronic device generating the input event corresponding to the virtual object based on one of the displacement distances of the double-click gesture.

[0074] In some embodiments, the input event includes a zoom-in / zoom-out operation or a virtual object drag operation.

[0075] In some embodiments, when the first gesture is a snap gesture, step S405 further includes the electronic device moving the virtual object to an initial position; and the electronic device adjusting the size of one of the virtual objects.

[0076] In some embodiments, the virtual object includes a plurality of sub-objects, and the control method 400 further includes, in response to deciding to initiate the input operation, the electronic device marking one of the sub-objects closest to the reference point.

[0077] In some embodiments, the control method 400 further includes, in response to a decision to initiate the input operation, the electronic device receiving the inertial measurement parameters acquired within a time interval after the input operation is initiated from the wearable device.

[0078] In some embodiments, step S405 further includes the electronic device inputting the inertial measurement parameters into a classification model to select the first gesture from the gestures.

[0079] In some embodiments, the input events include a click event, a double click event, and a finger snap event.

[0080] In summary, the control method 400 disclosed herein utilizes real-time images captured by an electronic device to confirm the user's hand posture, and then uses inertial measurement parameters obtained from a wearable device to determine the user's hand movements (i.e., gestures). Control method 400 solves two problems: the low accuracy of image recognition in recognizing subtle hand movements, and the difficulty in determining the hand's position in three-dimensional space using inertial measurement parameters. By combining these two technologies, control method 400 can more accurately determine the user's input information, while providing a more user-friendly and easier-to-use human-computer interaction experience.

[0081] Although several embodiments have been described in detail above as examples, the electronic devices and methods disclosed herein can also be implemented in other systems, hardware, software, storage media, or combinations thereof. Therefore, the scope of protection of this disclosure should not be limited to the specific implementations described in the embodiments disclosed herein, but should be determined by the claims defined in the appended patent application.

[0082] It will be apparent to those skilled in the art to which this disclosure pertains that various modifications and variations can be made to the structure of this disclosure without departing from its scope or spirit. In view of the foregoing, the scope of protection of this disclosure also covers modifications and variations made within the scope of the appended patent applications. [Simplified Explanation of the Diagram]

[0011] To make the above and other objects, features, advantages and embodiments of this disclosure more apparent and understandable, the accompanying drawings are described as follows: Figure 1 is a schematic diagram of a user operating a head-mounted display in the first embodiment of this disclosure; Figure 2 is a schematic diagram of a head-mounted display in the first embodiment of this disclosure; Figure 3 is a flowchart of the operation of a head-mounted display generating control signals based on gesture input in some embodiments of this disclosure; Figure 4 is a flowchart of the operation of generating hand gestures based on real-time images in some embodiments of this disclosure; Figure 5A is a schematic diagram of a user wearing a wearable device in some embodiments of this disclosure; Figure 5B is a schematic diagram of the vertical extension area corresponding to the key in some embodiments of this disclosure; Figure 6 is a schematic diagram of a finger-flicking gesture in some embodiments of this disclosure; and Figure 7 is a schematic diagram of a user operating a virtual keyboard in some embodiments of this disclosure; and Figure 8 is a flowchart of a control method in the second embodiment of this disclosure. [Biomaterial Storage]

[0084] Domestic storage information (please note in order of storage institution, date, and number) None Foreign storage information (please note in order of storage country, institution, date, and number) None

Claims

1. A head-mounted display comprising: a display; a communication interface communicatively connected to a wearable device; a camera for capturing a real-time image including a hand of a user wearing the wearable device; and a processor coupled to the display, the communication interface, and the camera for performing the following operations: calculating a pose corresponding to the hand based on the real-time image, wherein the pose includes a reference point located on the hand; and determining, in response to a relative position of the reference point and a virtual object displayed on the display, whether the relative position of the reference point is located within a vertically extending region of the virtual object, wherein the vertically extending region is generated by a distance vertically extending from a plurality of sub-objects in the virtual object; In response to the relative position of the reference point being located in the vertical extension area of ​​the virtual object, an input operation is initiated, and based on a plurality of inertial measurement parameters received from the wearable device and the reference point, a first gesture corresponding to the reference point is selected from a plurality of gestures; and based on the first gesture and an input target corresponding to the input operation, an input event corresponding to the virtual object is generated.

2. The head-mounted display as claimed in claim 1, wherein the operation of calculating the pose comprises: calculating a plurality of key points of the hand in the live image; and generating the pose of the hand in a three-dimensional space based on the key points and depth information in the live image.

3. The head-mounted display as described in claim 2, wherein the operation of calculating the posture further includes: selecting one of the key points as the reference point.

4. The head-mounted display as claimed in claim 1, wherein when the first gesture is a click gesture, the input target is generated based on the following operations: calculating a projection point of the reference point on a virtual plane corresponding to the virtual object; and selecting a first sub-object from a plurality of sub-objects corresponding to the virtual object as the input target based on the projection point.

5. The head-mounted display as claimed in claim 1, wherein at least one boundary location of the virtual object includes a virtual label, and the operation of generating the input event corresponding to the virtual object when the first gesture is a double-click gesture further includes: determining whether the reference point is located at one of the spatial locations of the virtual label; and in response to the reference point being located at the spatial location of the virtual label, generating the input event corresponding to the virtual object based on one of the displacement distances of the double-click gesture.

6. The head-mounted display as described in claim 1, wherein when the first gesture is a snap gesture, generating the input event corresponding to the virtual object further includes the following operations: moving the virtual object to an initial position; and adjusting one of the sizes of the virtual object.

7. The head-mounted display as described in claim 1, wherein the virtual object comprises a plurality of sub-objects, and the processor further performs the following operation: in response to deciding to initiate the input operation, marking one of the sub-objects closest to the reference point.

8. The head-mounted display as claimed in claim 1, wherein the processor further performs the following operations: in response to deciding to initiate the input operation, receiving the inertial measurement parameters acquired within a time interval after the initiation of the input operation from the wearable device.

9. A control method applicable to an electronic device, wherein the electronic device is communicatively connected to a wearable device, the control method comprising the steps of: capturing a real-time image of a hand of a user wearing the wearable device; calculating a posture corresponding to the hand based on the real-time image, wherein the posture includes a reference point located on the hand; determining, in response to a relative position of the reference point and a virtual object displayed on a display, whether the relative position of the reference point is located within a vertically extending region of the virtual object, wherein the vertically extending region is generated by a distance vertically extending from a plurality of sub-objects in the virtual object; determining to initiate an input operation in response to the relative position of the reference point being located within the vertically extending region of the virtual object, selecting a first gesture corresponding to the reference point from a plurality of gestures based on a plurality of inertial measurement parameters received from the wearable device and the reference point; and generating an input event corresponding to the virtual object based on the first gesture and an input target corresponding to the input operation.

10. A non-transitory computer-readable storage medium having at least one instruction stored thereon, wherein when a processor executes the at least one instruction, the at least one instruction executes a control method, the control method comprising the steps of: capturing a real-time image including a wearable device worn on a user's hand; calculating a posture corresponding to the hand based on the real-time image, wherein the posture includes a reference point located on the hand; determining, in response to a relative position of the reference point and a virtual object displayed on a display, whether the relative position of the reference point is located within a vertically extending region of the virtual object, wherein the vertically extending region is generated by a distance vertically extending from a plurality of sub-objects in the virtual object; determining to initiate an input operation in response to the relative position of the reference point being located within the vertically extending region of the virtual object, selecting a first gesture corresponding to the reference point from a plurality of gestures based on a plurality of inertial measurement parameters received from the wearable device and the reference point; and generating an input event corresponding to the virtual object based on the first gesture and an input target corresponding to the input operation.

Images