Method for providing input interface, host and computer readable storage medium

Abstract

The embodiment of the invention provides a method for providing an input interface, a host and a computer readable storage medium. The method includes: tracking a gesture of a hand; and in response to determining that the gesture of the hand indicates that the hand has presented the target gesture, displaying a virtual input interface in visual content associated with the real service, where the virtual input interface provides functionality of a simulation controller, and the virtual input interface moves with the hand.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to a mechanism for providing a reality service, and more particularly, to a method, a host, and a computer-readable storage medium for providing an input interface. BACKGROUND

[0002] Most virtual reality (VR) software still relies on users to input instructions through operating hardware / physical input elements on a traditional physical handheld controller interface. However, the advent of gesture tracking technology is gradually changing this phenomenon. This transition is causing software developers to start rethinking and adjusting their designs to take full advantage of the potential of gesture tracking technology. At the same time, users also need some time to get used to and master the new gesture-based interaction, which can require some teaching and guidance. SUMMARY

[0003] Therefore, the present disclosure provides a method, a host, and a computer-readable storage medium for providing an input interface, which can be used to solve the above technical problems.

[0004] The present disclosure provides a method for providing an input interface, applied to a host. The method comprises: tracking a hand gesture of a hand by the host; and in response to determining that the hand gesture of the hand indicates that the hand has assumed a target hand gesture, displaying, by the host, a virtual input interface in visual content associated with a reality service, wherein the virtual input interface provides a simulated controller function, and the virtual input interface moves with the hand.

[0005] The present disclosure provides a host, comprising a storage circuit and a processor. The storage circuit stores program code. The processor is coupled to the storage circuit and accesses the program code to perform: tracking a hand gesture of a hand; and in response to determining that the hand gesture of the hand indicates that the hand has assumed a target hand gesture, displaying a virtual input interface in visual content associated with a reality service, wherein the virtual input interface provides a simulated controller function, and the virtual input interface moves with the hand.

[0006] The present disclosure provides a computer-readable storage medium, which records an executable computer program. When the executable computer program is loaded by a host, the following steps are performed: tracking a hand gesture of a hand; and in response to determining that the hand gesture of the hand indicates that the hand has assumed a target hand gesture, displaying a virtual input interface in visual content associated with a reality service, wherein the virtual input interface provides a simulated controller function, and the virtual input interface moves with the hand. BRIEF DESCRIPTION OF DRAWINGS

[0007] Figure 1 is a schematic diagram of a host according to an embodiment of the present disclosure.

[0008] Figure 2 FIG. 1 is a flowchart illustrating a method for providing an input interface according to an embodiment of the present disclosure.

[0009] Figure 3A and Figure 3B FIG. 2 is a diagram illustrating different variations of a target gesture according to an embodiment of the present disclosure.

[0010] Figure 4 FIG. 3 is a diagram illustrating a virtual input interface according to an embodiment of the present disclosure.

[0011] Figure 5 FIG. 4 is an application scenario according to a third embodiment of the present disclosure.

[0012] Figure 6 FIG. 5 is a flowchart illustrating a method for configuring a virtual input interface according to an embodiment of the present disclosure.

[0013] Figure 7 FIG. 6 is a diagram illustrating virtual input interfaces with different sizes according to a fifth embodiment of the present disclosure.

[0014] Figure 8 FIG. 7 is a diagram illustrating configuration of a virtual input interface according to a sixth embodiment of the present disclosure. DETAILED DESCRIPTION

[0015] Reference will now be made to Figure 1 , Figure 1 FIG. 8 is a diagram illustrating a host according to an embodiment of the present disclosure. In various embodiments, the host 100 can be any device capable of performing a tracking function (e.g., inward-outward tracking and / or outward-inward tracking) on one or more objects to be tracked (e.g., a user's hand) located within a tracking range of the host 100.

[0016] In embodiments of the present disclosure, the host 100 can be configured with a tracking camera having an image capturing range corresponding to the tracking range. When the objects to be tracked (e.g., a hand) are located within the tracking range, the camera on the host 100 can capture images of the objects to be tracked, and the host 100 can track the poses of each of the objects to be tracked. For better understanding, this tracking mechanism can be referred to as a computer vision (CV) approach, but the present disclosure is not limited thereto.

[0017] In some embodiments, the host 100 can track a gesture of a hand located within the tracking range and render a corresponding hand object in the provided visual content in accordance with the gesture.

[0018] In some embodiments, one or more wearable devices can be worn on an object to be tracked (e.g., a user's hand) to facilitate the host 100 to track the object to be tracked. For example, two wearable devices (e.g., smart rings or gloves) can be worn on the thumb and index finger of a user's hand, respectively, for tracking gestures of the user's hand.

[0019] In one embodiment, a wearable device can be provided with a motion detection circuit (e.g., an inertia measurement unit (IMU)) for providing motion data (e.g., IMU data) characterizing a motion / gesture / posture of a user's hand / finger. For better understanding, such a tracking mechanism can be referred to as a motion detection approach, but the present disclosure is not limited thereto.

[0020] Additionally or alternatively, a wearable device can be provided with an electromyography (EMG) detection circuit for providing EMG data characterizing a motion / gesture / posture of a user's hand / finger, but the present disclosure is not limited thereto. For better understanding, such a tracking mechanism can be referred to as an EMG detection approach, but the present disclosure is not limited thereto.

[0021] In various embodiments, the host 100 can be any smart device and / or computer device capable of providing visual content of a reality service such as a virtual reality (VR) service, an augmented reality (AR) service, and / or a mixed reality (MR) service. In some embodiments, the host 100 can be a head-mounted display (HMD) capable of displaying / providing visual content (e.g., AR / VR / MR content) for a wearer / user to view. For better understanding of the concepts of the present disclosure, it will be assumed in the following that the host 100 is an HMD for providing a VR content (e.g., a VR world) to a user, but the present disclosure is not limited thereto.

[0022] In Figure 1 The host 100 includes a storage circuit 102 and a processor 104. The storage circuit 102 is one or a combination of a fixed or mobile random access memory (RAM), a read-only memory (ROM), a flash memory, a hard disk, or any other similar device, which records a plurality of modules / program codes and the modules / program codes can be executed by the processor 104.

[0023] The processor 104 can be coupled to the storage circuit 102 and can be, for example, a general-purpose processor, a special-purpose processor, a conventional processor, a digital signal processor (DSP), one or more microprocessors related to a DSP core, a controller, a microcontroller, an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) circuit, any other type of integrated circuit (IC), a state machine, and the like.

[0024] In embodiments of the present application, the processor 104 can access modules / program codes stored in the storage circuit 102 to implement the method of providing an input interface provided by the present application, which will be further described below.

[0025] Please refer to Figure 2 , Figure 2 is a flowchart of a method for providing an input interface according to embodiments of the present application. The method of the present embodiment can be executed by the host 100, which will be further described below with reference to the elements shown in Figure 1 to illustrate the details of each step. Figure 1 Figure 2 In step S210, the processor 104 tracks the hand gesture. In embodiments of the present application, the hand can be the hand of the user of the HMD, but the present application is not limited thereto.

[0026] In various embodiments, the processor 104 can track the hand gesture by using one or a combination of the above-mentioned tracking mechanisms, such as the CV method, the motion detection method, and / or the EMG method.

[0027] For example, the processor 104 can receive first motion data (e.g., IMU data) from a first wearable device (e.g., a first smart ring) worn on a first finger (e.g., the thumb) of the hand, receive second motion data from a second wearable device (e.g., a second smart ring) worn on a second finger (e.g., the index finger) of the hand, and determine the hand gesture based on the first motion data and the second motion data.

[0028] For another example, the processor 104 can receive first EMG data from a first wearable device worn on a first finger of the hand, receive second EMG data from a second wearable device worn on a second finger of the hand, and determine the hand gesture based on the first EMG data and the second EMG data.

[0029] For yet another example, the processor 104 can control a tracking camera to capture an image of the hand and determine the hand gesture therefrom.

[0030] In embodiments of the present application, the processor 104 can determine whether the hand gesture indicates that the hand has assumed a target hand gesture.

[0031] In embodiments of the present application, the processor 104 can determine whether the hand gesture indicates that the hand has assumed a target hand gesture.

[0032] ​Referring to Figure 3A and Figure 3B wherein Figure 3A and Figure 3B are different variations of the target gesture as illustrated in accordance with embodiments of the present application.

[0033] In the present embodiment, the gesture of the hand 30 can be the gesture tracked by the processor 104 in step S210, wherein the two wearable devices 31 and 32 (e.g. smart rings) can be worn on the thumb and index finger respectively, although the present application is not limited thereto.

[0034] In embodiments of the present application, the target gesture can be a first fist gesture, wherein the thumb of the hand 30 is located on the side of the index finger of the hand 30, and the tip of the thumb substantially contacts the knuckle or the tip of the index finger of the hand 30, as shown in the target gesture P1 of Figure 3A .

[0035] In other embodiments of the present application, the target gesture can be a second fist gesture, wherein the thumb of the hand 30 at least partially covers the fist eye 33 of the second fist gesture, as shown in the target gestures P1 and P2 of Figure 3A and Figure 3B . For the sake of clarity, the term "fist eye" generally refers to a central point or area within a clenched fist, which can also include a central indentation or opening formed by the arrangement of the fingers other than the thumb when the fist is clenched, although the present application is not limited thereto.

[0036] In step S220, in response to determining that the gesture of the hand indicates that the hand has assumed the target gesture, the processor 104 displays a virtual input interface in the visual content associated with the reality service.

[0037] Referring to Figure 4 , Figure 4 are schematic diagrams of the virtual input interface as illustrated in accordance with embodiments of the present application.

[0038] In Figure 4 , when the processor 104 determines that the gesture of the hand 30 has assumed the target gesture, the processor 104 can display a virtual input interface 41 in the visual content (e.g. VR content).

[0039] In the present embodiment, the virtual input interface 41 provides a simulated controller function or controller-like function. In one embodiment, the virtual input interface 41 corresponds to a physical handheld controller 49 having physical input elements (e.g. a physical joystick 491, a first physical button 492, a second physical button 493 and a third physical button 494), wherein the physical handheld controller 49 can be, for example, a VR handheld controller used for interacting with the VR content, although the present application is not limited thereto.

[0040] In an embodiment of the present application, the virtual input interface 41 can include one or more virtual input elements. In Figure 4 In an embodiment of the present application, the virtual input elements in the virtual input interface 41 can include a virtual joystick 411, a first virtual button 412, and a second virtual button 413, which can respectively correspond to the physical joystick 491, the first physical button 492 (e.g., the A button), and the second physical button 493 (e.g., the B button) on the physical handheld controller 49, but the present application is not limited thereto.

[0041] In other embodiments, the virtual input interface 41 can be designed to have other appearances, layouts, and / or configurations, but the present application is not limited thereto.

[0042] In some embodiments, the layout of the virtual input interface 41 (e.g., the sizes, shapes, and / or virtual input elements therein) can be self-adjusted / adjusted by the user, which will be discussed later in the present application.

[0043] In an embodiment of the present application, the virtual input interface 41 moves with the movement of the hand. For example, when the user is experiencing or interacting with visual content (e.g., VR content) provided by the host 100 (e.g., the HMD) using the target hand gesture, the processor 104 can display the virtual input interface 41 in the visual content after detecting the target hand gesture. Thereafter, the user can interact with the visual content while substantially / roughly maintaining the target hand gesture, and the processor 104 can move the virtual input interface 41 with the movement of the hand 30 during this period.

[0044] In an embodiment, the processor 104 can track a feature point on the hand 30 and maintain alignment of one virtual input element on the virtual input interface 41 with the feature point during the movement of the virtual input interface 41 with the hand.

[0045] In Figure 4 In an embodiment, the feature point considered for tracking by the processor 104 can be the feature point I3, which can be one of the knuckles of the index finger of the hand 30. In addition, the virtual input element maintained in alignment with the considered feature point can be the virtual joystick 411 of the virtual input interface 41. In this case, the virtual input interface 41 can move with the movement of the hand gesture during the period when the user substantially / roughly maintains the target hand gesture to experience the visual content.

[0046] In the present embodiment, the processor 104 can also track other feature points on the hand 30, such as the feature points I1, I2, T1, which respectively correspond to the tip of the index finger, another knuckle of the index finger, and the tip of the thumb.

[0047] In other embodiments, the processor 104 can maintain other virtual input elements on the virtual input interface 41 in alignment with other feature points during the movement of the virtual input interface 41 by the hand. For example, the processor 104 can maintain the first virtual button 412 in alignment with the feature point Ii during the movement of the virtual input interface 41 by the hand. As another example, the processor 104 can maintain the second virtual button 413 in alignment with the feature point I2during the movement of the virtual input interface 41 by the hand, although the application is not limited thereto.

[0048] In one embodiment, the processor 104 can also track the eye gaze point of the user on the visual content. In this case, in response to determining that the hand gesture of the hand 30 indicates that the hand 30 has assumed the target hand gesture (e.g., the first / second fist hand gesture), the processor 104 can further determine whether the eye gaze point is on the hand 30. In one embodiment, in response to determining that the eye gaze point is on the hand 30, it indicates that the user intends to interact with the visual content using the virtual input interface 41, and thus the processor 104 can display the virtual input interface 41 in the visual content associated with the reality service. On the other hand, in response to determining that the eye gaze point is not on the hand 30, it indicates that the user can not intend to interact with the visual content using the virtual input interface 41, and thus the processor 104 can not display the virtual input interface 41.

[0049] As mentioned previously, once the virtual input interface 41 is displayed, the user can interact with the visual content through the virtual input interface 41. In particular, the user can perform one or more input operations through the virtual input interface 41 in a manner similar to operating the physical handheld controller 49, and the processor 104 can determine the input operations from the interaction between the hand 30 and the virtual input interface 41.

[0050] In the first embodiment, the processor 104 can determine whether the interaction between the hand 30 and the virtual input interface 41 indicates that the hand 30 has triggered a virtual input element on the virtual input interface 41, where the virtual input element can correspond to the first physical input element. If so, the processor 104 can generate a first input signal corresponding to the first physical input element.

[0051] For example, assume that the triggered virtual input element is a first virtual button 412 corresponding to, for example, an A button on the physical handheld controller 49 (e.g., the first physical input element under consideration). The processor 104 can generate a first input signal corresponding to the A button on the physical handheld controller 49. That is, when the interaction between the hand 30 and the virtual input interface 41 indicates that the first virtual button 412 has been triggered, the processor 104 can determine that the user intends to perform an input operation corresponding to triggering the A button. In this case, the processor 104 can generate the first input signal corresponding to the A button and adjust the visual content accordingly, although the disclosure is not limited in this respect.

[0052] In the first embodiment, in response to determining that the thumb of the hand 30 triggers the virtual input element, the processor 104 can determine that the hand 30 triggers the virtual input element.

[0053] In a first variation of the first embodiment, the virtual input element under consideration can be a virtual button (e.g., the first virtual button 412). In this case, in response to determining that the fingertip of the thumb of the hand 30 is located within the display range of the virtual button, the processor 104 can determine that the thumb of the hand 30 triggers the virtual input element. For example, when the processor 104 determines that the feature point T1 (corresponding to the fingertip of the thumb) is located within the display range of the first virtual button 412, the processor 104 can determine that the thumb of the hand 30 triggers the first virtual button 412. That is, the user can trigger the first virtual button 412 by using the fingertip of the thumb to (virtually) touch the first virtual button 412, although the disclosure is not limited in this respect.

[0054] In a second variation of the first embodiment, the virtual input element under consideration can be a virtual button (e.g., the first virtual button 412 corresponding to the feature point I1). In this case, in response to determining that the distance between the fingertip of the thumb and the feature point on the hand 30 corresponding to the virtual button is less than a distance threshold, the processor 104 can determine that the thumb of the hand 30 triggers the virtual input element. For example, when the processor 104 determines that the distance between the feature point T1 (corresponding to the fingertip of the thumb) and the feature point I1 (corresponding to the first virtual button 412) is less than a distance threshold, the processor 104 can determine that the thumb of the hand 30 triggers the first virtual button 412. That is, the user can trigger the first virtual button 412 by moving the fingertip of the thumb to a position close to (or even in actual contact with) the fingertip of the index finger, although the disclosure is not limited in this respect.

[0055] In the second embodiment, the processor 104 can determine whether the interaction between the hand 30 and the virtual input interface 41 indicates that the hand 30 triggers a virtual input element on the virtual input interface 41, where the virtual input element can correspond to the first physical input element. If so, the processor 104 can generate a first input signal corresponding to the first physical input element.

[0056] For example, assume that the triggered virtual input element is, for example, a virtual joystick 411 corresponding to a physical joystick 491 on the physical handheld controller 49 (e.g., the first physical input element under consideration). The processor 104 can generate a first input signal corresponding to the physical joystick 491 on the physical handheld controller 49. That is, when the interaction between the hand 30 and the virtual input interface 41 indicates that the virtual joystick 411 has been triggered, the processor 104 can determine that the user intends to perform an input operation corresponding to triggering the physical joystick 491. In this case, the processor 104 can generate a first input signal corresponding to the physical joystick 491 and adjust the visual content accordingly, although the present application is not limited thereto.

[0057] In the second embodiment, in response to determining that the thumb of the hand 30 triggers the virtual input element, the processor 104 can determine that the hand 30 triggers the virtual input element.

[0058] In a first variation of the second embodiment, the virtual input element under consideration can be a virtual joystick 411 having a center 411a. In this case, in response to determining that the fingertip of the thumb of the hand 30 is located within the display range of the virtual input element, the processor 104 can determine that the thumb of the hand 30 triggers the virtual input element and generate a first input signal accordingly. For example, when the processor 104 determines that the feature point T1 (corresponding to the fingertip of the thumb) is located within the display range of the virtual joystick 411, the processor 104 can determine that the thumb of the hand 30 triggers the virtual joystick. That is, the user can trigger the virtual joystick 411 by (virtually) touching the virtual joystick 411 using the fingertip of the thumb, although the present application is not limited thereto.

[0059] In the first variation of the second embodiment, the first input signal can be generated based on a first relative position between the fingertip of the thumb and the center 411a of the virtual joystick 411. For example, when the thumb of the thumb (feature point T1) is located on a first side of the center 411a, the processor 104 can generate a first input signal corresponding to the physical joystick 491 being pushed toward the first side accordingly.

[0060] In the second variation of the second embodiment, the virtual input element under consideration can be a virtual joystick 411 having a center 411a. In this case, in response to determining that the distance between the fingertip of the thumb and the feature point on the hand 30 corresponding to the center 411a of the virtual joystick 411 is within the distance range, the processor 104 can determine that the thumb of the hand 30 triggers the virtual input element. For example, when the processor 104 determines that the distance between the feature point T1 (corresponding to the fingertip of the thumb) and the feature point I3 (which can correspond to the center 411a) is within the distance range, the processor 104 can determine that the thumb of the hand 30 triggers the virtual joystick 411, where the distance range can be dependent on the radius of the display range of the virtual joystick 411, although the present application is not limited thereto.

[0061] In the second variation of the second embodiment, the first input signal can be generated based on the second relative position between the fingertip of the thumb and the previously considered feature point (e.g., the feature point I3). For example, if the second relative position indicates that the fingertip of the thumb (characterized by the feature point T1) is on the first side of the feature point I3, the processor 104 can generate the first input signal corresponding to the physical joystick 491 being pushed to the first side.

[0062] In embodiments of the present application, if the first wearable device and the second wearable device track the gesture of the hand 30, the processor 104 can determine the interaction between the hand 30 and the virtual input interface 41 by considering the relative readings between the first and second motion data.

[0063] For example, if the readings of the first and second motion data each indicate that the first wearable device and the second wearable device have moved, but the relative readings between the first and second motion data indicate that the relative position between the first wearable device and the second wearable device has not moved, this represents that the user is only moving the entire hand and not moving the fingers to interact with the virtual input interface 41.

[0064] On the other hand, if the relative readings between the first and second motion data indicate that the relative position between the first wearable device and the second wearable device has changed, this represents that the user is moving the fingers to interact with the virtual input interface 41, and thus the processor 104 can determine whether the interaction between the hand 30 and the virtual input interface 41 indicates that the hand 30 triggers a virtual input element on the virtual input interface 41, although the present application is not limited thereto.

[0065] In addition, since the physical hand-held controller 49 is provided with the third physical button 494 (which allows the user to perform an action similar to pulling the trigger of a gun), the embodiments of the present application also provide a solution that allows the user to perform an input operation corresponding to the third physical button 494 after displaying the virtual input interface 41. The relevant details will be discussed in the third embodiment below. For better understanding, the third embodiment will be described with reference to the first embodiment. Figure 5 As an example, but the present application is not limited thereto.

[0066] Please refer to Figure 5 , Figure 5 is the application scenario illustrated according to the third embodiment of the present application.

[0067] In the third embodiment, the processor 104 can determine whether the specific finger 51 (e.g., the index finger) of the hand 30 has performed a specific action to become the first predetermined finger gesture 510.

[0068] In the third embodiment, the processor 104 may, for example, determine whether the specific finger 51 of the hand 30 has performed a specific motion based on the second motion data and / or the relative readings between the first motion data and the second motion data mentioned earlier.

[0069] In the third embodiment, in response to determining that the specific finger 51 has been straightened and not fully bent, the processor 104 can determine that the specific finger 51 of the hand 30 has performed a specific action to become the first predetermined finger gesture 510. From another point of view, if the processor 104 determines that the specific finger 51 that was originally part of the target gesture is first straightened and then becomes a semi-bent state, the processor 104 can determine that the specific finger 51 of the hand 30 has performed a specific action to become the first predetermined finger gesture 510, but the present application is not limited thereto.

[0070] In the third embodiment, in response to determining that the specific finger 51 of the hand 30 has performed a specific action to become the first predetermined finger gesture 510, the processor 104 can then determine whether the specific finger 51 has changed from the first predetermined finger gesture 510 to the second predetermined finger gesture 520.

[0071] In the third embodiment, in response to determining that the specific finger 51 has changed from not fully bent to fully bent, the processor 104 can determine that the specific finger 51 has changed from the first predetermined finger gesture 510 to the second predetermined finger gesture 520. From another point of view, if the processor 104 determines that the specific finger 51 has changed from a semi-bent state to a fully bent state, the processor 104 can determine that the specific finger 51 has changed from the first predetermined finger gesture 510 to the second predetermined finger gesture 520, but the present application is not limited thereto.

[0072] In the third embodiment, in response to determining that the particular finger 51 has changed from the first predetermined finger gesture 510 to the second predetermined finger gesture 520, the processor 104 can generate a second input signal corresponding to a second physical input element.

[0073] In one embodiment, the second physical input element can be a third physical button 494 on the physical handheld controller 49. In this case, when the processor 104 determines that the particular finger 51 has changed from the first predetermined finger gesture 510 to the second predetermined finger gesture 520, this can indicate that the user intends to perform an input operation corresponding to triggering the third physical button 494 (which can be understood as a trigger button). Accordingly, the processor 104 can generate a second input signal corresponding to the third physical button 494 and adjust the visual content accordingly, although the present disclosure is not limited thereto.

[0074] In the third embodiment, in response to determining that the second predetermined finger gesture 520 has been maintained, the processor 104 can continue to generate the second input signal corresponding to the second physical input element until the particular finger 51 ceases to make the second predetermined finger gesture 520.

[0075] In the embodiment in which the second physical input element is assumed to be the third physical button 494, it can be allowed for the user to perform a one-time trigger (e.g., firing a bullet in a VR game application) by sequentially making the first predetermined finger gesture 510, the second predetermined finger gesture 520, and the first predetermined finger gesture 510. In addition, it can also be allowed for the user to perform a continuous trigger (e.g., continuously firing bullets in a VR game application) by switching from the first predetermined finger gesture 510 to the second predetermined finger gesture 520 and maintaining the second predetermined finger gesture 520 until the user wants to stop, although the present disclosure is not limited thereto.

[0076] In addition, in response to determining that the particular finger 51 of the hand 30 has performed a particular action to become the first predetermined finger gesture 510, the processor 104 can also provide a vibration signal to a wearable device worn on the particular finger 51 (e.g., the wearable device 32 worn on the index finger), such that the wearable device can be controlled to provide a vibration action to notify that the first predetermined finger gesture 510 has been successfully detected, although the present disclosure is not limited thereto.

[0077] In some embodiments, the processor 104 can determine whether a gesture has changed from a target gesture to a cancel gesture (e.g., an open-palm gesture). If so, it means that the user can intentionally stop interacting with the visual content using the virtual input interface 41, and thus the processor 104 can stop displaying the virtual input interface 41, although the present disclosure is not limited thereto.

[0078] In addition, if the virtual input interface 41 is displayed after the user's eye gaze is determined to be on the fist gesture, the processor 104 may decide to stop displaying the virtual input interface 41 when the user's eye gaze is determined to be on the gesture again, but it is not limited to this.

[0079] As previously mentioned, the layout of the virtual input interface 41 (e.g., size, shape, and / or virtual input elements therein) can be customized / adjusted by the user, as detailed below.

[0080] Please refer to Figure 6 , Figure 6 This is a flowchart illustrating a method for configuring a virtual input interface according to an embodiment of the present invention. The method of this embodiment can be... Figure 1 The host 100 executes, and the following will be supplemented with Figure 1 To illustrate with components Figure 6 Details of each step in the process.

[0081] In step S610, the processor 104 tracks at least one feature point on the hand. In step S620, the processor 104 configures the layout of at least one virtual input element on the virtual input interface based on the at least one feature point.

[0082] For ease of understanding, the hand considered in step S610 can be assumed to be the hand 30 mentioned in the above embodiments, and at least one feature point of the hand can be correspondingly assumed to be a feature point on the hand 30, such as feature points I1 to I3 and / or T1, but the present invention is not limited thereto. In addition, the virtual input interface considered in step S620 can be assumed to be the virtual input interface 41 mentioned above, and therefore the virtual input elements thereon can include, for example, a virtual joystick 411, a first virtual button 412 and a second virtual button 412, but the present invention is not limited thereto.

[0083] In this embodiment of the invention, a reality service (e.g., a VR service) can provide virtual interface configuration functionality to allow users to configure the layout of the virtual input interface 41. For example, a reality service system (e.g., a VR system) can provide virtual interface configuration functionality in its system settings interface.

[0084] In one embodiment, in response to the determination that the virtual interface configuration function in the real service is triggered, the processor 104 can provide a configuration interface for configuring the layout of virtual input elements on the virtual input interface 41.

[0085] In this case, the processor 104 may begin executing step S610 after providing the setting interface, and thereby execute step S620, but the present invention is not limited thereto.

[0086] In one embodiment, prior to performing step S620, the processor 104 can request the hand 30 to present a target gesture. For example, once the processor 104 provides the configuration interface, the processor 104 can request the user to present a target gesture by, but not limited to, a textual / visual / audio notification, such that the user can be guided to present a fist gesture to enable the processor 104 to better track the feature points on the hand 30, but the present application is not limited thereto.

[0087] In one embodiment, the processor 104 can determine whether the hand 30 has presented the target gesture. In response to determining that the hand 30 has presented the target gesture, this represents that the current gesture is suitable for configuring the layout of the virtual input interface 41. Accordingly, the processor 104 can configure the layout of the virtual input elements on the virtual input interface 41 in accordance with the feature points on the hand 30.

[0088] On the other hand, in response to determining that the hand 30 has not presented the target gesture, this represents that the current gesture can not be suitable for configuring the layout of the virtual input interface 41, and thus the processor 104 can not configure the layout of the virtual input elements on the virtual input interface 41, but the present application is not limited thereto.

[0089] In various embodiments, step S620 can be implemented in different ways, and the following embodiments will provide further discussion.

[0090] In a fourth embodiment, the processor 104 can allow the user to manually establish an association between a certain feature point (e.g., a first feature point) on the hand 30 and a certain virtual input element (e.g., a first virtual input element) on the virtual input interface 41. Then, the processor 104 can determine to display the first virtual input element at a location where the first feature point is located.

[0091] For example, the processor 104 can display the tracked feature points on the hand 30 as shown in the middle of Figure 4 and ask the user to assign which feature points should correspond to which virtual input elements.

[0092] In one embodiment, the association between the feature points and the virtual input elements can be established as follows

[0093] as exemplified in Table 1.

[0094] Feature point I1 Feature point I2 Feature point I3 Virtual joystick 411 V First virtual button 412 V Second virtual button 413 V

[0095] Table 1

[0096] As can be seen from the above, the feature points I3, I1, I2 have been assigned to correspond to the virtual joystick 411, the first virtual button 412, and the second virtual button 413, respectively, but the present application is not limited thereto.

[0097] In this case, the processor 104 can determine to display the virtual joystick 411 at the position where the feature point I3 is located, the first virtual button 412 at the position where the feature point I1 is located, and the second virtual button 413 at the position where the feature point I2 is located. Accordingly, the layout of the virtual input interface 41 can be determined. More specifically, the positions of the virtual input elements in the virtual input interface 41 can be determined, but the present application is not limited thereto.

[0098] In an embodiment, the processor 104 can further display the virtual input interface 41 having the determined layout to the user for viewing. At this time, the user can see that the virtual joystick 411, the first virtual button 412, and the second virtual button 413 are displayed at the positions where the feature points I3, I1, and I2 are located, but the present application is not limited thereto.

[0099] In some embodiments, since the sizes of the hands of different users can be different, the embodiments of the present application can scale the size of the virtual input interface 41 accordingly.

[0100] More specifically, in the fifth embodiment, the processor 104 can determine the size information of the hand 30 based on the feature points on the hand 40. Then, the processor 104 can adjust the size of the virtual input interface 41 based on the size information of the hand 30.

[0101] In the fifth embodiment, the size information of the hand 30 can include at least one of a reference length and a reference area associated with the hand 30. For example, the reference length can include at least one of a finger length of the hand 30 (which can be derived from the distance between the feature points on the considered finger) and a palm width of the hand 30. In addition, the reference area can include the area of the tip of the thumb of the hand 30, but the present application is not limited thereto.

[0102] In the fifth embodiment, the processor 104 can determine the ratio by comparing the reference length with a predetermined length or comparing the reference area with a predetermined area. In an embodiment, the predetermined length can be, for example, the finger length and / or the palm width of an average person's hand. Similarly, the predetermined area can be the area of the tip of the thumb of an average person's hand.

[0103] After determining the ratio, the processor 104 can scale the size of the virtual input interface 41 by the ratio.

[0104] From another point of view, the processor 104 can determine whether the hand 30 is larger or smaller than an average person's hand and scale the size of the virtual input interface 41 accordingly. In this case, if the hand 30 is larger than an average person's hand, the processor 104 can enlarge the size of the virtual input interface 41, and vice versa.

[0105] Please refer to Figure 7 , Figure 7This is a schematic diagram illustrating virtual input interfaces of different sizes according to the fifth embodiment of the present invention.

[0106] like Figure 7 As shown on the left, the virtual input interface is sized to fit a standard-sized hand. On the other hand, Figure 7 The size of the virtual input interface on the right may be suitable for larger hands, but the invention is not limited thereto.

[0107] In the sixth embodiment, during the process of configuring the virtual input interface 41 (in the configuration interface), the processor 104 may display the virtual input interface 41 with a predetermined layout (e.g., virtual input elements with predetermined size / position) at a predetermined position in the visual content.

[0108] In the sixth embodiment, it can be assumed that the feature points include a first feature point and a second feature point, and it can be assumed that the virtual input element includes a first virtual input element and a second virtual input element.

[0109] In this embodiment, the processor 104 can determine whether the first feature point has been aligned with the first virtual input element.

[0110] In one embodiment, processor 104 can provide visual / audio guidance to request the user to align a first feature point with a first virtual input element.

[0111] In the sixth embodiment, in response to determining that the first feature point has been aligned with the first virtual input element, the processor 104 can determine a reference position associated with the second virtual input element within the virtual input interface 41 based on the first feature point and the second feature point, and display the second virtual input element at the reference position within the virtual input interface 41.

[0112] To better understand, the following will be based on Figure 8 For example, among which Figure 8 This is a schematic diagram illustrating the configuration of a virtual input interface based on the sixth embodiment.

[0113] exist Figure 8 In this context, the first virtual input element under consideration can be assumed to be a virtual joystick 411, and the second virtual input element under consideration can be assumed to be a first virtual button 412.

[0114] exist Figure 8 In the middle, processor 104 can display a conventional configuration (e.g., Figure 8 The virtual input interface 41 (layout on the left) prompts the user to align a feature point T1 (e.g., the tip of the thumb) with the virtual joystick 411 (center 411a). In other embodiments, the size of the virtual input interface 41 may be pre-adjusted using the mechanism described in the fifth embodiment, but the invention is not limited thereto.

[0115] In response to determining that the feature point T1 has been aligned with the virtual joystick 411, the processor 104 can determine a reference position associated with the first virtual button 412 within the virtual input interface 41 based on the feature points T1 and T3, and display the first virtual button 412 at the reference position within the virtual input interface 41, where the first virtual button 412 located at the reference position can be referred to as the first virtual button 412a on the right side, but the present application is not limited thereto. Figure 8 In response to determining that the feature point T1 has been aligned with the virtual joystick 411, the processor 104 can determine a reference position associated with the first virtual button 412 within the virtual input interface 41 based on the feature points T1 and T3, and display the first virtual button 412 at the reference position within the virtual input interface 41, where the first virtual button 412 located at the reference position can be referred to as the first virtual button 412a on the right side, but the present application is not limited thereto.

[0116] In Figure 8 In response to determining that the feature point T1 has been aligned with the virtual joystick 411, the processor 104 can determine a reference position associated with the first virtual button 412 within the virtual input interface 41 based on the feature points T1 and T3, and display the first virtual button 412 at the reference position within the virtual input interface 41, where the first virtual button 412 located at the reference position can be referred to as the first virtual button 412a on the right side, but the present application is not limited thereto.

[0117] In response to determining that the feature point T1 has been aligned with the virtual joystick 411, the processor 104 can determine a reference position associated with the first virtual button 412 within the virtual input interface 41 based on the feature points T1 and T3, and display the first virtual button 412 at the reference position within the virtual input interface 41, where the first virtual button 412 located at the reference position can be referred to as the first virtual button 412a on the right side, but the present application is not limited thereto.

[0118] In response to determining that the feature point T1 has been aligned with the virtual joystick 411, the processor 104 can determine a reference position associated with the first virtual button 412 within the virtual input interface 41 based on the feature points T1 and T3, and display the first virtual button 412 at the reference position within the virtual input interface 41, where the first virtual button 412 located at the reference position can be referred to as the first virtual button 412a on the right side, but the present application is not limited thereto. Figure 8 As can be seen from Figure 8 The first virtual button 412 on the left side can be considered to be located at a position that is difficult for the user to reach with the tip of the thumb. However, as can be seen from Figure 8 The first virtual button 412a on the right side can be considered to be located at a position that is difficult for the user to reach with the tip of the thumb. However, as can be seen from

[0119] Similarly, the second virtual button 413 can also be adjusted to be located at a position that is easy for the user to reach with the tip of the thumb, such as another point on the reference path 81. In an embodiment, the predetermined position of the second virtual button 413 can also be determined based on the original relative position between the first virtual button 412 and the second virtual button 413. For example, since the predetermined position of the second virtual button 413 is above the predetermined position of the first virtual button 412, the position of the second virtual button 413 on the reference path 81 can be determined, for example, to be at a certain position between the first virtual button 412a and the feature point T1, but the present application is not limited thereto.

[0120] From another point of view, the sixth embodiment can be understood as setting the first virtual button 412 at other positions that are easier for the user to reach according to the length of the fingers of the thumb, but the present application is not limited thereto.

[0121] In the seventh embodiment, the processor 104 can determine the first reference position and the second reference position within the virtual input interface 41 based on the first feature point and the second feature point. In this embodiment, the first reference position and the second reference position can be the positions where the first feature point and the second feature point are located, but the present application is not limited thereto.

[0122] In this embodiment, the size of the virtual input interface 41 can be adjusted in advance using the mechanism introduced in the fifth embodiment, but the present application is not limited thereto.

[0123] Next, the processor 104 can display the first virtual input element and the second virtual input element at the first reference position and the second reference position within the virtual input interface 41, respectively.

[0124] That is, when deciding the positions of the first virtual input element and the second virtual input element within the virtual input interface 41, the processor 104 can directly place the first virtual input element and the second virtual input element at the positions corresponding to the first virtual input element and the second virtual input element, respectively.

[0125] For example, assuming that the first feature point and the second feature point are feature points I3 and I1, respectively, the corresponding first virtual input element and the second virtual input element can be the virtual joystick 411 and the first virtual button 412, respectively. In this case, when deciding the positions of the virtual joystick 411 and the first virtual button 412 within the virtual input interface 41, the processor 104 can directly place the virtual joystick 411 and the first virtual button 412 at the positions corresponding to the feature points I3 and I1, respectively.

[0126] Similarly, the processor 104 can also directly place the second virtual button 413 at the position corresponding to the feature point I2 within the virtual input interface 41, but the present application is not limited thereto.

[0127] In an embodiment, if the user has completed the configuration of the layout of the virtual input interface 41, the system providing the reality service can allow the user to perform a confirmation operation.

[0128] Therefore, in response to determining that the confirmation operation has been detected, the processor 104 can fix the layout of the virtual input elements on the virtual input interface 41. For example, the processor 104 can store the size of the virtual input interface 41 and the sizes / relative positions of the virtual input elements therein for future use. For example, when an application program providing a reality service is executed, the virtual input interface 41 with the fixed layout can be applied.

[0129] In this case, in response to determining that an application program (e.g., a VR application program) has been executed, the processor 104 can apply the virtual input interface 41 with the fixed layout.

[0130] For example, the processor 104 can perform step S210 and display the virtual input interface 41 having the fixed layout in response to determining that the gesture of the hand 30 indicates that the hand 30 has assumed the target gesture, and the relevant details can refer to the above-described embodiments.

[0131] In one embodiment, Figure 6 The method of configuring the virtual input interface 41 can be performed before the method of providing the input interface. Figure 2 That is, the processor 104 can first let the user configure the layout required by the virtual input interface 41 based on the method of configuring the virtual input interface 41, and then, when the user assumes the target gesture in the process of interacting with the visual content of the real service, the processor 104 can display the virtual input interface 41 having the layout previously configured by the user, but the present application is not limited thereto. Figure 6

[0132] The present application also provides a computer-readable storage medium for performing the above-described method of providing the input interface and / or the method of configuring the virtual input interface. The computer-readable storage medium is composed of a plurality of program instructions (e.g., setting program instructions and deployment program instructions) contained therein. These program instructions can be loaded into the host 100 and executed by the host 100 to perform the above-described method of providing the input interface and / or the method of configuring the virtual input interface and the functions of the host 100.

[0133] In summary, the embodiments of the present application provide a scheme of displaying the virtual input interface in response to determining that the user has made the target gesture. In the embodiments in which the virtual input interface corresponds to the physical handheld controller, the user can be allowed to interact with the real service (e.g., a VR application) in a gesture similar to operating the physical handheld controller. Therefore, the process of teaching and / or guiding the user to learn the gesture-based interaction mode in the real service can be reduced.

[0134] In addition, the embodiments of the present application also provide a layout configuration scheme of the virtual input interface, which can improve the intuitiveness of the user operation.

[0135] Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present application, and not to limit them; although the present application has been described in detail with reference to the above embodiments, those skilled in the art should understand that: it can still modify the technical solutions recorded in the above embodiments, or make equivalent replacement for part or all of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the scope of the technical solutions of the embodiments of the present application.​

Claims

1. A method for providing an input interface, applied to a host computer, characterized in that, include: The host computer tracks hand gestures; as well as In response to determining that the hand gesture indicates that the hand has presented a target gesture, the host displays a virtual input interface in visual content associated with a real-world service, wherein the virtual input interface provides simulated controller functionality and moves with the hand.

2. The method according to claim 1, wherein the virtual input interface includes at least one of a virtual joystick, a first virtual button, and a second virtual button.

3. The method of claim 1, wherein tracking the gesture of the hand comprises at least one of the following: Receive first motion data from a first wearable device worn on a first finger of the hand, receive second motion data from a second wearable device worn on a second finger of the hand, and determine the gesture of the hand based on the first motion data and the second motion data; Receives first electromyography (EMG) data from a first wearable device worn on the first finger of the hand, receives second EMG data from a second wearable device worn on the second finger of the hand, and determines the hand gesture based on the first and second EMG data; and The image of the hand is captured and the hand gesture is determined accordingly.

4. The method according to claim 1, wherein the virtual input interface includes virtual input elements, and the method further includes: The system tracks feature points on the hand and maintains alignment between the virtual input elements on the virtual input interface and the feature points as the hand moves the virtual input interface.

5. The method according to claim 4, wherein the virtual input element on the virtual input interface includes one of a virtual joystick, a first virtual button, and a second virtual button, and the feature point on the hand corresponds to the knuckle of the index finger of the hand.

6. The method of claim 1, wherein in response to determining that the hand's gesture indicates that the hand has presented the target gesture, the method further comprises: In response to determining that the eye gaze is on the hand, the virtual input interface is displayed in the visual content associated with the real-world service; as well as In response to determining that the eye gaze is not on the hand, the virtual input interface is not displayed.

7. The method of claim 1, wherein the target gesture is a fist gesture in which the thumb of the hand is located on the side of the index finger of the hand, and the tip of the thumb substantially contacts the knuckle or tip of the index finger.

8. The method of claim 1, wherein the target gesture is a clenched fist gesture, wherein the thumb of the hand at least partially covers the knuckles of the clenched fist gesture.

9. The method according to claim 1, further comprising: The input operation is determined based on the interaction between the hand and the virtual input interface.

10. The method of claim 9, wherein the virtual input interface includes a virtual input element corresponding to the first physical input element, and determining the input operation based on the interaction between the hand and the virtual input interface includes: In response to the determination that the interaction between the hand and the virtual input interface indicates that the hand triggers the virtual input element, a first input signal corresponding to the first physical input element is generated.

11. The method of claim 10, comprising: In response to determining that the thumb of the hand triggers the virtual input element, it is determined that the hand triggers the virtual input element.

12. The method of claim 11, wherein the virtual input element comprises a virtual button, and the method comprises: In response to determining that the tip of the thumb of the hand is within the display range of the virtual button, it is determined that the thumb of the hand triggers the virtual input element; or In response to determining that the distance between the fingertip of the thumb and a feature point on the hand corresponding to the virtual button is less than a distance threshold, the thumb of the hand is determined to trigger the virtual input element.

13. The method of claim 11, wherein the virtual input element comprises a virtual joystick having a center, and the method comprises: In response to determining that the tip of the thumb of the hand is within the display range of the virtual input element, it is determined that the thumb of the hand triggers the virtual input element, wherein the first input signal is generated based on a first relative position between the tip of the thumb and the center of the virtual joystick; or In response to determining that the distance between the fingertip of the thumb and a feature point on the hand corresponding to the center of the virtual joystick is within a distance range, it is determined that the thumb of the hand triggers the virtual input element, wherein the first input signal is generated based on a second relative position between the fingertip of the thumb and the feature point.

14. The method of claim 1, wherein after displaying the virtual input interface, the method further comprises: In response to determining that a specific finger of the hand has performed a specific action and become a first predetermined finger gesture, it is determined whether the specific finger has changed from the first predetermined finger gesture to a second predetermined finger gesture; as well as In response to determining that the specific finger has changed from the first predetermined finger gesture to the second predetermined finger gesture, a second input signal corresponding to the second physical input element is generated.

15. The method of claim 14, comprising: In response to determining that the specific finger has been straightened and then partially bent, it is determined that the specific finger of the hand has performed the specific action to become the first predetermined finger gesture.

16. The method of claim 15, comprising: In response to determining that the specific finger has changed from partially bent to fully bent, it is determined that the specific finger has changed from the first predetermined finger gesture to the second predetermined finger gesture.

17. The method of claim 14, further comprising: In response to determining that the second predetermined finger gesture is maintained, the second input signal corresponding to the second physical input element is continuously generated until the specific finger stops presenting the second predetermined finger gesture.

18. The method of claim 14, wherein in response to determining that the particular finger of the hand has performed the particular movement to become the first predetermined finger gesture, a vibration signal is provided to a wearable device worn on the particular finger.

19. A host computer, characterized in that, include: Storage circuitry stores program code; as well as The processor is coupled to the storage circuitry and accesses the program code to execute: Track hand gestures; as well as In response to determining that the hand gesture indicates that the hand has presented a target gesture, a virtual input interface is displayed in visual content associated with a real-world service, wherein the virtual input interface provides simulated controller functionality and moves with the hand.

20. A computer-readable storage medium, characterized in that, The computer-readable storage medium records an executable computer program, which is loaded by a host computer to perform the following steps: Tracking hand gestures; and In response to determining that the hand gesture indicates that the hand has presented a target gesture, a virtual input interface is displayed in visual content associated with a real-world service, wherein the virtual input interface provides simulated controller functionality and moves with the hand.

Images