Head-mounted display device, interaction method thereof and computer readable storage medium

Abstract

The embodiment of the invention discloses a head-mounted display device, an interaction method thereof and a computer readable storage medium. The method comprises the steps that the head-mounted display device displays a view of a three-dimensional environment; if it is detected that a first part of a first finger of a first hand of the user collides with a second part of a second finger of a second hand, executing a first operation in response to collision of the first finger and the second finger; and / or if it is detected that the first finger slides on the second finger, the second operation is executed in response to sliding of the first finger on the second finger. According to the embodiment of the invention, the interactive operation with the head-mounted display device is carried out through the click gesture that the finger of one hand of the user collides with the finger of the other hand and / or the slide gesture that the finger of one hand slides on the finger of the other hand, so that the user can obtain stronger tactile feedback, the operation comfort is relatively high, and the user experience is improved. And the false touch probability is reduced, so that the user interaction operation experience is better.

Description

Technical Field

[0001] The present application relates to the field of extended reality (XR) technology, and in particular to a head-mounted display device and an interaction method thereof, and a computer-readable storage medium. Background Art

[0002] With the continuous development of XR technology and the increasing popularization of head-mounted display devices, more and more users are using head-mounted display devices to immersively experience virtual scenes.

[0003] At present, the three-dimensional environment displayed by the head-mounted display device includes a user interface (UI). The user can first select the UI to be operated by the eye gaze point, and then use a single-handed pinch gesture to perform basic operations such as clicking, sliding, zooming, and panning to operate the UI selected by the eye gaze point.

[0004] However, when performing interactive operations through a one-hand pinch gesture, the user obtains weak tactile feedback and the operation comfort is low; in addition, the user may unconsciously perform a one-hand pinch gesture, which results in false touches and a high false touch rate. These problems will lead to a poor user interactive operation experience. Summary of the invention

[0005] The embodiments of the present application provide a head-mounted display device and an interaction method thereof, and a computer-readable storage medium, which can solve the problem that when the existing head-mounted display device performs interaction operation through a single-hand pinch gesture, the user's interaction experience is poor.

[0006] In a first aspect, an embodiment of the present application provides an interaction method of a head-mounted display device, which is applied to the head-mounted display device, and the method includes:

[0007] Display a view of a three-dimensional environment; if it is detected that a first part of a first finger of a first hand of a user collides with a second part of a second finger of a second hand, perform a first operation in response to the collision of the first finger and the second finger, the first hand and the second hand are different; and / or if it is detected that the first finger slides over the second finger, perform a second operation in response to the sliding of the first finger over the second finger.

[0008] As can be seen from the above, the embodiments of the present application perform interactive operations with the head-mounted display device through a click gesture in which the fingers of one hand of the user touch the fingers of the other hand, and / or a sliding gesture in which the fingers of one hand slide on the fingers of the other hand. In this way, when performing interactive operations through two-hand gestures (including click gestures and / or sliding gestures), the user can obtain stronger tactile feedback through the contact and sliding between the fingers of the two hands, and the interactive operation is more comfortable; in addition, the probability of the user unconsciously touching and sliding the fingers of the two hands is low, which reduces the probability of false touches and provides a better user interactive operation experience.

[0009] In some possible implementations of the first aspect, the operation direction (or adjustment direction) of the second operation is related to the sliding direction of the first finger on the second finger, and / or the operation amount of the second operation is related to the displacement of the first finger on the second finger. In this way, associating the finger sliding direction with the operation direction, and associating the finger sliding displacement with the operation amount, further improves the interactive operation experience when interacting through the sliding gesture of the embodiment of the present application.

[0010] In some possible implementations of the first aspect, the view of the three-dimensional environment includes at least one interactive object; and the method further includes:

[0011] In response to one or more operations, the target object is displayed according to a target display mode, and the one or more operations are operations of selecting the target object from at least one interactive object. The target display mode is different from the display of non-target objects, that is, the display mode of the target object is different from that of other objects.

[0012] If it is detected that a first part of a first finger of a first hand of a user collides with a second part of a second finger of a second hand, in response to the collision of the first finger and the second finger, a first operation is performed, including:

[0013] If it is detected that the first part of the first finger and the second part of the second finger collide once, and the continuous contact time of the first finger and the second finger when they collide is less than the first time threshold, then in response to the collision of the first finger and the second finger, a confirmation operation is performed or a single-click operation is performed on the target object;

[0014] If it is detected that the first part of the first finger and the second part of the second finger collide once, and the continuous contact time of the first finger and the second finger during the collision is greater than the second time threshold, then in response to the collision of the first finger and the second finger, a long press operation is performed on the target object, and the second time threshold is greater than or equal to the first time threshold;

[0015] If it is detected that the first part of the first finger collides with the second part of the second finger twice, and the continuous contact time of the first finger and the second finger each time is less than the first time threshold, then in response to the collision of the first finger and the second finger, a double-click operation is performed on the target object; the first operation is a confirmation operation, a single-click operation, a long press operation or a double-click operation.

[0016] In this implementation, through a two-handed gesture of touching the fingers of one hand with the fingers of the other hand, confirmation, single-click, double-click or long-press operations are performed on objects in a three-dimensional environment, thereby reducing the probability of accidental touches in the confirmation, single-click, double-click or long-press operations, and the user can obtain stronger tactile feedback, the operation logic is more reasonable, and the user's interactive operation experience is further improved.

[0017] In some possible implementations of the first aspect, if it is detected that the first finger slides on the second finger, in response to the sliding of the first finger on the second finger, performing a second operation includes:

[0018] A first finger is detected sliding along an upper edge of a second finger; in response to the sliding direction and displacement of the first finger on the second finger, a second operation is performed on the target object; wherein the operation amount of the second operation is related to the displacement of the first finger on the second finger, and the adjustment direction of the second operation is related to the sliding direction of the first finger on the second finger, and the adjustment direction includes increase and decrease.

[0019] In this implementation, a two-handed gesture of sliding a first finger over a second finger is used to perform an adjustment operation on an object in a three-dimensional environment, and the operation amount and adjustment direction of the second operation are respectively related to the finger displacement and the sliding direction, so that the user can obtain stronger tactile feedback and can more conveniently perform adjustment operations on objects in the three-dimensional environment.

[0020] In some possible implementations of the first aspect, if the sliding direction of the first finger on the second finger is the first direction, the operation direction or adjustment direction of the second operation is to increase; if the sliding direction of the first finger on the second finger is the second direction, the operation direction or adjustment direction of the second operation is to decrease; wherein the first direction is the direction from the wrist of the second hand to the fingertip of the second finger, and the second direction is the direction from the fingertip of the second finger to the wrist of the second hand. In this way, the interactive operation logic is more reasonable and more in line with user habits, making the user interactive operation more comfortable.

[0021] In some possible implementations of the first aspect, the target object is a volume control, a brightness control, a window or a progress bar control, and the second operation is: increasing or decreasing the volume, increasing or decreasing the brightness, increasing or decreasing the window size, or increasing or decreasing the playback progress. In this way, through a two-handed gesture of sliding a finger of one hand on a finger of another hand, the user can more conveniently adjust objects that are difficult to operate in the view of the three-dimensional environment, such as the volume, brightness, window size, and playback progress bar, further improving the user's interactive operation experience.

[0022] In some possible implementations of the first aspect, if it is detected that the first finger slides on the second finger, in response to the sliding of the first finger on the second finger, performing a second operation includes:

[0023] If it is detected that the first finger slides on the second finger along the first direction, in response to the sliding of the first finger on the second finger, the page is slid in the first sliding direction, the first direction being the direction from the wrist of the second hand to the fingertip of the second finger;

[0024] If it is detected that the first finger slides on the second finger along the second direction, the page is slid in the second sliding direction in response to the sliding of the first finger on the second finger, and the second direction is the direction from the fingertip of the second finger to the wrist of the second hand, and the first sliding direction and the second sliding direction are different; wherein the second operation is a page sliding operation, and the sliding amount of the page is related to the displacement amount of the first finger on the second finger.

[0025] In this implementation, in response to the sliding of the first finger on the second finger, the page is controlled to slide (or scroll) in the first sliding direction or the second sliding direction, that is, when the finger of one hand of the user slides on the finger of the other hand, the page in the three-dimensional environment of the head-mounted display device slides in a specific direction, so that the user does not need to bend the wrist upward or bounce the wrist upward to slide the page, and the wrist pressure will not increase during long-term operation, which alleviates the wrist fatigue problem and improves the user's interactive operation comfort; in addition, when the finger of one hand slides on the finger of the other hand, the user can obtain stronger trigger feedback. In this way, the user's interactive operation experience is further improved.

[0026] In some possible implementations of the first aspect, the head mounted display device includes an optical display module, the optical display module includes a lens and a display screen; the method further includes: in response to one or more operations, entering a diopter adjustment mode;

[0027] If it is detected that the first finger slides on the second finger, in response to the sliding of the first finger on the second finger, a second operation is performed, including:

[0028] In the diopter adjustment mode, it is detected that the first finger slides on the second finger; in response to the sliding of the first finger on the second finger, the movable component in the optical display module is controlled to move; wherein the movement amount of the movable component is related to the displacement amount of the first finger on the second finger, and the movement direction of the movable component is related to the sliding direction of the first finger on the second finger; the movable component is a lens or a display screen; and the second operation is to control the movement of the movable component.

[0029] In this implementation, a two-handed gesture of sliding the first finger on the second finger is used to control the movable parts in the optical display module to be closer to or farther away from the human eye, so that when the user wears the head-mounted display device, the user can still conveniently adjust the refractive power (myopia / hyperopia degree) of the user's left and right eyes through two-handed gestures without removing the head-mounted display device and manually rotating the lens barrel for adjustment, further improving the user's interactive operation experience.

[0030] In some possible implementations of the first aspect, if the sliding direction of the first finger on the second finger is the first direction, the moving direction is the direction from the movable component to the human eye; if the sliding direction of the first finger on the second finger is the second direction, the moving direction is the direction from the human eye to the movable component. In this way, when the diopter is adjusted by the two-handed gesture of sliding the first finger on the second finger, the operation logic is more reasonable and more in line with user habits, making the user interaction operation more comfortable.

[0031] In some possible implementations of the first aspect, the first finger is a thumb or an index finger, the second finger is a thumb or an index finger, the first part is a fingertip, and the second part is a fingertip or a back of a finger. In this way, when the user performs interactive operations through two-handed gestures, the operation comfort is higher.

[0032] In a second aspect, an embodiment of the present application provides a head-mounted display device, comprising a processing module, an optical display module and a gesture tracking module; the optical display module is used to display a view of a three-dimensional environment; the gesture tracking module is used to recognize user gestures;

[0033] The processing module is used to: display a view of a three-dimensional environment through an optical display module; if a first part of a first finger of a first hand of a user collides with a second part of a second finger of a second hand through a gesture tracking module, perform a first operation in response to the collision of the first finger and the second finger, the first hand and the second hand are different; and / or if a first finger slides on a second finger through a gesture tracking module, perform a second operation in response to the sliding of the first finger on the second finger.

[0034] In some possible implementations of the second aspect, an operation direction of the second operation is related to a sliding direction of the first finger on the second finger, and / or an operation amount of the second operation is related to a displacement amount of the first finger on the second finger.

[0035] In some possible implementations of the second aspect, the head-mounted display device further includes an eye tracking module and / or a head movement aiming module, and the view of the three-dimensional environment includes at least one interactive object; the processing module is further used to: display the target object in a target display mode in response to one or more operations through the eye tracking module and / or the head movement aiming module, the one or more operations are operations of selecting the target object from the at least one interactive object, and the target display mode is different from the display mode of non-target objects;

[0036] The processing module is specifically used for:

[0037] If the gesture tracking module detects that the first part of the first finger and the second part of the second finger collide once, and the continuous contact time of the first finger and the second finger when they collide is less than the first time threshold, then in response to the collision of the first finger and the second finger, a confirmation operation is performed or a single-click operation is performed on the target object;

[0038] If it is detected through the gesture tracking module that the first part of the first finger and the second part of the second finger collide once, and the continuous contact time of the first finger and the second finger during the collision is greater than the second time threshold, then in response to the collision of the first finger and the second finger, a long press operation is performed on the target object, and the second time threshold is greater than or equal to the first time threshold;

[0039] If the gesture tracking module detects that the first part of the first finger collides with the second part of the second finger twice, and the continuous contact time of the first finger and the second finger each time is less than the first time threshold, then in response to the collision of the first finger and the second finger, a double-click operation is performed on the target object; the first operation is a confirmation operation, a single-click operation, a long press operation or a double-click operation.

[0040] In some possible implementations of the second aspect, the processing module is specifically used to: detect, through a gesture tracking module, a first finger sliding along the upper edge of a second finger; and perform a second operation on the target object in response to the sliding direction and displacement of the first finger on the second finger; wherein the operation amount of the second operation is related to the displacement of the first finger on the second finger, and the adjustment direction of the second operation is related to the sliding direction of the first finger on the second finger, and the adjustment direction includes increase and decrease.

[0041] In some possible implementations of the second aspect, if the sliding direction of the first finger on the second finger is a first direction, the adjustment direction of the second operation is increase; if the sliding direction of the first finger on the second finger is a second direction, the adjustment direction of the second operation is decrease; wherein the first direction is the direction from the wrist of the second hand to the fingertip of the second finger, and the second direction is the direction from the fingertip of the second finger to the wrist of the second hand.

[0042] In some possible implementations of the second aspect, the target object is a volume control, a brightness control, a window or a progress bar control, and the second operation is: increasing or decreasing the volume, increasing or decreasing the brightness, increasing or decreasing the window size, or increasing or decreasing the playback progress.

[0043] In some possible implementations of the second aspect, the processing module is specifically configured to:

[0044] If the gesture tracking module detects that the first finger slides on the second finger along the first direction, in response to the sliding of the first finger on the second finger, the page is slid in the first sliding direction, where the first direction is a direction from the wrist of the second hand to the fingertip of the second finger;

[0045] If the gesture tracking module detects that the first finger slides on the second finger along the second direction, the page is slid in the second sliding direction in response to the sliding of the first finger on the second finger, and the second direction is the direction from the fingertip of the second finger to the wrist of the second hand, and the first sliding direction and the second sliding direction are different; wherein the second operation is a page sliding operation, and the sliding amount of the page is related to the displacement of the first finger on the second finger.

[0046] In some possible implementations of the second aspect, the optical display module includes a lens and a display screen;

[0047] The processing module is further configured to: enter a diopter adjustment mode in response to one or more operations;

[0048] The processing module is specifically used for: in the diopter adjustment mode, detecting the sliding of the first finger on the second finger through the gesture tracking module; in response to the sliding of the first finger on the second finger, controlling the movable component in the optical display module to move; wherein the movement amount of the movable component is related to the displacement amount of the first finger on the second finger, and the movement direction of the movable component is related to the sliding direction of the first finger on the second finger; the movable component is a lens or a display screen; and the second operation is to control the movement of the movable component.

[0049] In some possible implementations of the second aspect, if the sliding direction of the first finger on the second finger is a first direction, the moving direction is from the movable component to the human eye; if the sliding direction of the first finger on the second finger is a second direction, the moving direction is from the human eye to the movable component.

[0050] In some possible implementations of the second aspect, the first finger is a thumb or an index finger, the second finger is a thumb or an index finger, the first part is a fingertip, and the second part is a fingertip or a back of a finger.

[0051] In a third aspect, an embodiment of the present application provides a head-mounted display device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the method described in any one of the first aspects above is implemented.

[0052] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method of any one of the above-mentioned first aspects is implemented.

[0053] In a fifth aspect, an embodiment of the present application provides a chip system, the chip system includes a processor, the processor is coupled to a memory, and the processor executes a computer program stored in the memory to implement any method as described in any one of the first aspects above. The chip system can be a single chip, or a chip module composed of multiple chips.

[0054] In a sixth aspect, an embodiment of the present application provides a computer program product. When the computer program product is run on a head-mounted display device, the head-mounted display device executes the method described in the first aspect above.

[0055] It can be understood that the beneficial effects of the second to sixth aspects mentioned above can be found in the relevant description of the first aspect mentioned above, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] Figure 1 A schematic diagram of an application scenario provided for an embodiment of the present application;

[0057] Figure 2A A schematic diagram of a thumb and index finger pinch gesture provided in an embodiment of the present application;

[0058] Figure 2B A schematic diagram of a wrist flick gesture provided in an embodiment of the present application;

[0059] Figure 3 A schematic block diagram of the hardware structure of a head mounted display device 100 provided in an embodiment of the present application;

[0060] Figure 4AA two-handed gesture application scenario provided in an embodiment of the present application;

[0061] Figure 4B A schematic diagram of a single-click gesture of a two-handed cross gesture provided in an embodiment of the present application;

[0062] Figure 4C A schematic diagram of a double-click gesture of a two-handed cross gesture provided in an embodiment of the present application;

[0063] Figure 4D A schematic diagram of a sliding gesture of a two-handed cross gesture provided in an embodiment of the present application;

[0064] Figure 4E A schematic diagram of a sliding gesture of a two-handed cross gesture provided in an embodiment of the present application;

[0065] Figure 4F Another gesture schematic diagram provided for an embodiment of the present application;

[0066] Figure 5A A schematic diagram of a click gesture in a two-handed gesture provided in an embodiment of the present application;

[0067] Figure 5B A schematic diagram of a sliding gesture in a two-handed gesture provided in an embodiment of the present application;

[0068] Fig. 6A A schematic diagram of interactive operations for movie viewing scenes provided in an embodiment of the present application;

[0069] Figure 6B A schematic diagram of adjusting the volume by sliding gestures provided in an embodiment of the present application;

[0070] Figure 6C A schematic diagram of adjusting the window size by sliding gestures provided in an embodiment of the present application;

[0071] Figure 7 A schematic diagram of controlling page sliding through sliding gestures provided in an embodiment of the present application;

[0072] Figure 8 A schematic diagram of the relationship between the eye and the optical display module 200 provided in an embodiment of the present application;

[0073] Fig. 9A A schematic diagram of a prompt operation provided in an embodiment of the present application;

[0074] Fig. 9B A schematic diagram of adjusting myopia degree through a sliding gesture provided in an embodiment of the present application;

[0075] Fig. 9C A schematic diagram of a diopter adjustment process provided in an embodiment of the present application;

[0076] Fig.10 A schematic diagram of an interaction method flow of the head mounted display device 100 provided in an embodiment of the present application;

[0077] Fig.11 A schematic flow chart of another interaction method of the head mounted display device 100 provided in an embodiment of the present application;

[0078] Fig.12 A flowchart of another interaction method of the head mounted display device 100 provided in an embodiment of the present application;

[0079] Fig.13 A flowchart of another interaction method of the head mounted display device 100 provided in an embodiment of the present application;

[0080] Fig.14 A flowchart of another interaction method of the head mounted display device 100 provided in an embodiment of the present application;

[0081] Fig.15 This is a schematic block diagram of the structure of a head mounted display device 100 provided in an embodiment of the present application. DETAILED DESCRIPTION

[0082] In the following description, for the purpose of explanation rather than limitation, specific details such as specific system structures and technologies are provided to facilitate a thorough understanding of the embodiments of the present application.

[0083] The following is an illustrative introduction to the relevant contents that may be involved in the embodiments of the present application.

[0084] (1) Physical three-dimensional environment.

[0085] The physical three-dimensional environment refers to the physical world (or real world, real world, physical three-dimensional space) that people can perceive without the help of computer systems. The physical three-dimensional environment includes real physical objects, such as physical trees, physical people, and physical buildings.

[0086] (2) The three-dimensional environment perceived by users through XR technology.

[0087] XR includes Virtual Reality (VR), Augmented Reality (AR), and Mixed Reality (MR).

[0088] VR system is a computer simulation system that can create and experience a virtual three-dimensional environment. VR technology can rely on computer graphics technology, computer simulation technology, sensor technology, and display technology to create a virtual three-dimensional environment (or virtual three-dimensional space, virtual world). A virtual three-dimensional environment is a virtual world completely simulated by a computer, including one or more virtual objects (such as virtual people, virtual trees, text, icons, buttons, and pages, etc.).

[0089] The virtual 3D environment includes three-dimensional realistic images generated by computers and played in real time, which brings visual perception to users. In addition to the visual perception generated by computer graphics technology, VR technology can also bring users hearing, touch, force, and movement perception, and even smell and taste, etc., to create a more realistic virtual 3D environment.

[0090] In a VR scenario, users can perceive a virtual three-dimensional environment by wearing a head-mounted display device. In addition, users can interact with virtual objects in the virtual three-dimensional environment by turning their heads, looking at their eyes, making gestures, or other human behaviors. For example, in a VR game scenario, after wearing a head-mounted display device, users can see a computer-simulated game interface and interact with the game interface through gestures or handles to gain an immersive gaming experience.

[0091] AR refers to a technology that superimposes computer-generated virtual objects on a physical three-dimensional environment to enhance the real world. For example, an AR head-mounted display device has a transparent or translucent display. After the user wears the head-mounted display device, he can directly see the physical three-dimensional environment through the display, and the head-mounted display device can display computer-simulated virtual objects on a transparent or translucent display, so that the user can perceive the virtual objects superimposed on the physical three-dimensional environment through the head-mounted display device. For another example, an AR head-mounted display device has an opaque display and an image sensor. The image sensor can capture images of the physical three-dimensional environment, and combine these images with computer-simulated virtual objects and display them on an opaque display, so that the user can perceive the virtual objects superimposed on the physical three-dimensional environment.

[0092] The difference between VR technology and AR technology is that the VR three-dimensional environment is a virtual environment completely simulated by a computer system, and what users see through head-mounted display devices are virtual objects; the AR three-dimensional environment includes virtual objects simulated by computers and real physical objects in the physical three-dimensional environment, that is, AR technology superimposes virtual objects on the real world, and the three-dimensional environment seen by users through head-mounted display devices includes both physical objects in the real world and virtual objects simulated by computers. For example, a user wears transparent glasses, through which he can see the real environment around him, and virtual objects can also be displayed on the glasses, so that the user can see both real objects and virtual objects.

[0093] MR introduces real scene information (or real scene information) into the virtual 3D environment, building a bridge of interactive feedback information between the virtual 3D environment, the physical 3D environment and the user, thereby enhancing the realism of the user experience. Specifically, the physical objects are virtualized (for example, using a camera to scan the physical objects in the real world for 3D reconstruction and generate virtual objects), and then the virtualized physical objects are introduced into the virtual 3D environment, so that the user can see the physical objects in the virtual 3D environment.

[0094] In AR and MR scenarios, users can also perceive the three-dimensional environment by wearing head-mounted display devices, and can also interact with objects in the three-dimensional environment through head movements, eye gaze, gestures, or other human behavioral movements.

[0095] As can be seen from the above, in VR, AR and MR scenarios, users can perceive the three-dimensional environment by wearing head-mounted display devices. Among them, the VR three-dimensional environment is a virtual environment completely simulated by a computer system, including one or more virtual objects. Part of the environment in the AR three-dimensional environment and the MR three-dimensional environment is simulated by a computer system, and part of the environment is a physical three-dimensional environment or a representation of a physical three-dimensional environment (such as an image or video of a physical three-dimensional environment).

[0096] After exemplarily introducing the relevant contents that may be involved in the embodiments of the present application, the application scenarios of the embodiments of the present application are exemplarily introduced below.

[0097] See also Figure 1 , is a schematic diagram of an application scenario provided by an embodiment of the present application. In a physical three-dimensional environment 10, a user wears a head-mounted display device 100. By wearing the head-mounted display device 100, the user can perceive a three-dimensional environment 20 displayed by the head-mounted display device 100. The three-dimensional environment 20 can be a VR three-dimensional environment, or an AR three-dimensional environment or an MR three-dimensional environment. The three-dimensional environment 20 includes one or more objects. Figure 1As shown, the three-dimensional environment 20 exemplarily includes multiple objects such as interface 21, interface 22 and interface 23.

[0098] The user can interact with the object in the three-dimensional environment 20 through gestures. Exemplarily, the user wears a head mounted display device, selects an object in the three-dimensional environment 20 through eye gaze or head movement, and then performs a gesture in the physical three-dimensional environment 10 to interact with the selected object through the gesture (e.g., perform an operation corresponding to the gesture on the selected object).

[0099] In the related art, a user interacts with an object in a three-dimensional environment 20 through a one-hand pinch gesture. The one-hand pinch gesture includes a finger pinch gesture and a wrist gesture. The finger pinch gesture refers to a gesture in which the thumb and index finger of the same hand are pinched, and the thumb and index finger tips touch each other; the wrist gesture refers to a gesture in which the wrist is flicked upward and bent downward.

[0100] For example, see Figure 2A The schematic diagram of the thumb and index finger pinch gesture is shown. Initially, the thumb tip and the index finger tip are not touching. The user pinches the thumb and index finger inwards so that the thumb tip and the index finger tip touch each other to perform the finger pinch gesture. Figure 2B Schematic diagram of wrist gestures shown, the left picture is a gesture of bending the wrist downward, and the right picture is a gesture of flicking the wrist upward.

[0101] Users perform a series of interactive operations through finger pinch gestures, wrist gestures, or a combination of the two. For example, Figure 2A The head mounted display device 100 detects that the user makes a pinch gesture (i.e., the thumb and index finger tips touch each other) to perform a click operation on an object in the three-dimensional environment 20. Figure 2A , and when the thumb and index finger tips of the same hand touch each other, it is considered that a click gesture is detected, and in response to the click gesture, a click operation is performed; Figure 2B The head mounted display device 100 detects that the user's wrist is flicked upward, and performs a sliding operation on the page in the three-dimensional environment 20. That is, if the head mounted display device 100 detects that the user's wrist is flicked upward, it is considered that an upward sliding gesture is detected, and in response to the upward sliding gesture, the page is slid upward. If the user's wrist is bent downward, it is considered that a downward sliding gesture is detected, and in response to the downward sliding gesture, the page is slid downward. That is, the user controls the page in the three-dimensional environment 20 to slide up or down by flicking his wrist upward or downward. When performing the upward flicking gesture and the downward bending gesture, the user's fingers will also appear as shown in the figure. Figure 2A The pinch state shown ( Figure 2B not shown).

[0102] The inventors found in the long-term research process that when interacting with objects in the three-dimensional environment 20 through a one-hand pinch gesture, although pinching the thumb and index fingertips of the same hand can allow the user to obtain tactile feedback, the tactile feedback is very weak. For example, when the thumb and index fingertips touch lightly, or the thumb and index fingertips gradually approach but do not touch, the tactile feedback obtained by the user is weak; and the user cannot obtain tactile feedback when the wrist bounces up and bends down. In summary, when the user performs a one-hand pinch gesture for interactive operation, the tactile feedback obtained by the user is weak; and when operating for a long time, the wrist pressure will increase, wrist fatigue will occur, and then affect the comfort of interactive operation.

[0103] In addition, the inventors have also found in the long-term research process that the head-mounted display device 100 has a camera placed downward for gesture recognition, so the user can comfortably place his hands on his legs or on supports such as chair armrests to perform a one-hand pinch gesture and operate with objects in the three-dimensional environment 20. However, when not performing a one-hand pinch gesture, the user will habitually place his hands on supports such as armrests or legs, and it is also a common and unintentional habit for the thumb and index finger of the same hand to touch or approach each other. In this way, the user will inadvertently make a one-hand pinch gesture, and the head-mounted display device 100 will also respond to the one-hand pinch gesture inadvertently performed by the user to perform the corresponding operation, thereby causing a false touch phenomenon, and the probability of false touch is high.

[0104] In response to the above-mentioned related problems that exist when performing interactive operations through one-hand pinch gestures, an embodiment of the present application provides an interactive solution for a head-mounted display device, which allows users to interact with the head-mounted display device through two-hand gestures, such as interacting with objects in a three-dimensional environment 20, so that users can obtain stronger tactile feedback, and long-term operation will not increase wrist pressure. The user's operation comfort is higher, and the probability of false touches can be reduced, thereby providing a better user interactive operation experience.

[0105] The hardware structure of the head mounted display device 100 is described below by way of example. Figure 3 The schematic block diagram of the hardware structure of the head-mounted display device 100 is shown. The head-mounted display device 100 may include a processor 110, a memory 120, a sensor module 130 (which can be used to obtain the user's posture), a microphone 140, a button 150, an input and output interface 160, a communication module 170, a camera 180, a battery 190, an optical display module 200, an eye tracking module 210, a gesture tracking module 220, and a head movement aiming module 230, etc.

[0106] It is understood that the structure illustrated in the embodiment of the present application does not constitute a specific limitation on the head mounted display device 100. In other embodiments of the present application, the head mounted display device 100 may include more or fewer components than shown in the figure, or combine some components, or split some components, or arrange the components differently. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.

[0107] The processor 110 is generally used to control the overall operation of the head mounted display device 100, and may include one or more processing units. For example, the processor 110 may include an application processor (AP), a modem processor, a graphics processor (GPU), an image signal processor (ISP), a video processing unit (VPU), a controller, a memory, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU), etc. Among them, different processing units may be independent devices or integrated in one or more processors. A memory may also be provided in the processor 110 for storing instructions and data.

[0108] In some embodiments of the present application, the processor 110 can be used to control the optical focal length of the head-mounted display device 100. Exemplarily, the processor 110 can be used to control the optical focal length of the optical display module 200 to achieve the function of adjusting the optical focal length of the head-mounted display device 100. For example, the processor 110 can adjust the relative positions between the various optical devices (such as lenses, etc.) in the optical display module 200 so that the optical focal length of the optical display module 200 is adjusted, and then when the optical display module 200 is imaged to the human eye, the position of the corresponding virtual image surface can be adjusted, thereby achieving the effect of controlling the optical focal length of the head-mounted display device 100.

[0109] In some embodiments, the processor 110 may include one or more interfaces. The interface may include an inter-integrated circuit (I2C) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface, a serial peripheral interface (SPI) interface, etc.

[0110] In some embodiments, the processor 110 may perform different degrees of blurring on objects at different depths of field, so that the clarity of objects at different depths of field is different.

[0111] The I2C interface is a bidirectional synchronous serial bus, including a serial data line (SDA) and a serial clock line (SCL). In some embodiments, the processor 110 may include multiple I2C buses.

[0112] The UART interface is a universal serial data bus for asynchronous communication. The bus can be a bidirectional communication bus. It converts the data to be transmitted between serial communication and parallel communication. In some embodiments, the UART interface is generally used to connect the processor 110 and the communication module 170. For example, the processor 110 communicates with the Bluetooth module in the communication module 170 through the UART interface to implement the Bluetooth function.

[0113] The MIPI interface can be used to connect the processor 110 with peripheral devices such as the display screen and camera 180 in the optical display module 200 .

[0114] The GPIO interface can be configured by software. The GPIO interface can be configured as a control signal or as a data signal. In some embodiments, the GPIO interface can be used to connect the processor 110 with the camera 180, the display screen in the optical display module 200, the communication module 170, the sensor module 130, and the microphone 140. The GPIO interface can also be configured as an I2C interface, an I2S interface, a UART interface, and a MIPI interface.

[0115] The camera 180 can be used to capture images including real objects, and the processor 110 can fuse the images captured by the camera with virtual objects, and display the fused images through the optical display module 200. For example, in an MR or AR scenario, the head mounted display device 100 captures images of a physical three-dimensional environment through the camera 180, and fuses the images of the physical three-dimensional environment with virtual objects simulated by a computer to construct an AR three-dimensional environment or a MR three-dimensional environment.

[0116] The camera 180 can also be used to collect images including human eyes. Through the eye tracking module 210, the processor 110 performs eye tracking based on the human eye image.

[0117] The camera 180 may also be used to collect a user's hand image. The processor 110 may also be used to process the user's hand image to obtain the hand key points, and based on the hand key points, perform gesture recognition and gesture tracking through the gesture tracking module 220.

[0118] It is understandable that the head mounted display device 100 of the embodiment of the present application may include multiple cameras 180. For example, the head mounted display device 100 may include one or more cameras for the eye tracking module 210, which are used to collect human eye images so that the eye tracking module 210 can perform eye tracking based on the human eye images; and may also include one or more cameras for the gesture tracking module 220, which are used to collect user hand images so that the gesture tracking module 220 can perform gesture recognition and gesture tracking based on the user hand images.

[0119] The USB interface is an interface that complies with the USB standard specification, and specifically can be a Mini USB interface, a Micro USB interface, a USB Type C interface, etc. The USB interface can be used to connect a charger to charge the head-mounted display device 100; it can be used to transmit data between the head-mounted display device 100 and peripheral devices; it can also be used to connect headphones to play audio through the headphones. The interface can also be used to connect other electronic devices, such as mobile phones. The USB interface can be USB3.0, which is used to be compatible with high-speed display port (DP) signal transmission, and can transmit high-speed video and audio data.

[0120] It is understandable that the interface connection relationship between the modules illustrated in the embodiment of the present application is only a schematic illustration and does not constitute a structural limitation on the head-mounted display device 100. In other embodiments of the present application, the head-mounted display device 100 may also adopt different interface connection methods in the above embodiments, or a combination of multiple interface connection methods.

[0121] In addition, the head-mounted display device 100 may include a communication module. For example, the head-mounted display device 100 may receive images from other electronic devices (eg, a host or a mobile phone) for display.

[0122] The communication module 170 may include a wireless communication module and a mobile communication module. The wireless communication function may be implemented through an antenna (not shown), a mobile communication module (not shown), a modem processor (not shown), and a baseband processor (not shown). The antenna is used to transmit and receive electromagnetic wave signals.

[0123] The mobile communication module can provide wireless communication solutions including the second generation (2G) network / third generation (3G) network / fourth generation (4G) network / fifth generation (5G) network applied to the head mounted display device 100.

[0124] The wireless communication module can provide wireless communication solutions for the head-mounted display device 100, including wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) networks), Bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication technology (NFC), infrared technology (IR), etc.

[0125] The head mounted display device 100 implements display functions through a GPU, an optical display module 200, and an application processor. The GPU is a microprocessor for image processing, which connects the optical display module 200 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The processor 110 may include one or more GPUs that execute program instructions to generate or change display information.

[0126] The memory 120 can be used to store computer executable program codes, which include instructions. The processor 110 executes various functional applications and data processing of the head-mounted display device 100 by running the instructions stored in the memory 120. The memory 120 may include a program storage area and a data storage area. Among them, the program storage area may store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc. The data storage area may store data created during the use of the head-mounted display device 100 (such as audio data, a phone book, etc.), etc. In addition, the memory 120 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, a universal flash storage (UFS), etc.

[0127] The head mounted display device 100 can implement audio functions through an audio module, a speaker, a microphone 140, a headphone interface, and an application processor. For example, music playback, recording, etc. The audio module is used to convert digital audio information into analog audio signal output, and is also used to convert analog audio input into digital audio signals. The audio module can also be used to encode and decode audio signals. In some embodiments, the audio module can be set in the processor 110, or some functional modules of the audio module can be set in the processor 110. The speaker, also called a "speaker", is used to convert audio electrical signals into sound signals.

[0128] Microphone 140, also called "microphone" or "microphone", is used to convert sound signals into electrical signals. The head-mounted display device 100 may be provided with at least one microphone 140. In other embodiments, the head-mounted display device 100 may be provided with two microphones 140, which can not only collect sound signals but also realize noise reduction function. In other embodiments, the head-mounted display device 100 may also be provided with three, four or more microphones 140 to realize sound signal collection, noise reduction, sound source identification, directional recording function, etc.

[0129] In some embodiments, the head-mounted display device 100 may include one or more buttons 150, which can control the head-mounted display device 100 and provide the user with access to functions on the head-mounted display device 100. The button 150 can be in the form of a button, a switch, a dial, and a touch or near-touch sensing device (such as a touch sensor). Specifically, for example, a user can turn on the optical display module 200 of the head-mounted display device 100 by pressing a button. The button 150 includes a power button, a volume button, etc. The button 150 can be a mechanical button. It can also be a touch button. The head-mounted display device 100 can receive a button input and generate a key signal input related to the user settings and function control of the head-mounted display device 100.

[0130] In some embodiments, the head mounted display device 100 may include an input / output interface 160, which may connect other devices to the head mounted display device 100 through appropriate components. The components may include, for example, an audio jack, a video jack, and / or a data connector.

[0131] The sensor module 130 may include, but is not limited to, a gyroscope sensor, an acceleration sensor, a distance sensor, and a touch sensor. The gyroscope sensor may be used to determine the motion posture of the head-mounted display device 100. In some embodiments, the angular velocity of the head-mounted display device 100 around three axes (i.e., x, y, and z axes) may be determined by a gyroscope sensor. The gyroscope sensor may also be used for navigation and somatosensory game scenes. The acceleration sensor may detect the magnitude of the acceleration of the head-mounted display device 100 in all directions (generally three axes). When the head-mounted display device 100 is stationary, the magnitude and direction of gravity may be detected, and it may also be used to identify the posture of the head-mounted display device 100.

[0132] The optical display module 200 is used to present images to the user under the control of the processor 110. The optical display module 200 can convert the real pixel image display into a virtual image display of near-eye projection through one or more optical devices such as a reflector, a transmission mirror or an optical waveguide, so as to realize a virtual interactive experience or an interactive experience combining virtual and reality. For example, the optical display module 200 receives the image data information sent by the processor 110 and presents the corresponding image to the user.

[0133] In some embodiments, the optical display module 200 may be a Pancake optical module. The Pancake optical module may include, but is not limited to: a polarizer, a lens group consisting of one or more lenses, and a display screen. The lens group or display screen of the Pancake optical module may be used as a movable component, that is, the lens group or display screen may be moved along the propagation direction of the optical path to adjust the virtual image distance. Exemplarily, the processor 110 controls the lens group or display screen to move along the propagation direction of the optical path (close to or away from the human eye) in response to the detected two-handed gesture to adjust the diopter (myopia or hyperopia).

[0134] In some embodiments, the head-mounted display device 100 may also include an eye tracking module 210, which is used to track the movement of the human eye and determine the gaze point of the human eye. For example, the pupil position can be located through image processing technology, the coordinates of the pupil center can be obtained, and then the gaze point of the person can be calculated. Exemplarily, the eye tracking module 210 includes a photosensitive unit and an infrared unit. The infrared unit is used to emit infrared light or near-infrared light toward the user's eyes; the photosensitive unit is used to collect the position of the infrared light reflected by the eye in the eye, and infer the user's line of sight. The photosensitive unit can be implemented using devices such as cameras.

[0135] In some embodiments, the head mounted display device 100 may further include a gesture tracking module 220, which is used to collect images or video streams including hands through the camera 180, and process the images or video streams through a gesture detection algorithm, identify user gestures, and track the user's hands. Exemplarily, the image or video stream including the hand is identified to obtain the coordinate information of the hand key points; based on the hand key point coordinate information, the gesture composed of these hand key points is obtained. In addition, the hand position or gesture is tracked based on the hand key points in different image frames.

[0136] In some embodiments, the head mounted display device 100 may further include a head motion aiming module 230 for detecting head swing or movement, and performing corresponding operations according to the swing or movement of the head. Exemplarily, when the user's head swings or moves, the head mounted display device 100 obtains the detection amount fed back by the gyroscope sensor or the acceleration sensor, and determines the direction and amount of the head movement or swing according to the detection amount, and controls the displacement of the mouse or cursor in the three-dimensional environment 20 according to the direction and amount of the head movement, or controls the screen to move in the direction corresponding to the head movement direction.

[0137] In some embodiments, the head mounted display device 100 may further include a motor. The motor may generate a vibration prompt to give vibration feedback to the user. For example, when a click operation is detected by a click gesture, vibration feedback may be given to the user by the motor vibration.

[0138] It should be noted that the head-mounted display device 100 can be an integrated head-mounted display system, that is, the computing unit (such as a processor, etc.) and the display unit (such as an optical display module) of the head-mounted display system are integrated into one physical device, such as a VR all-in-one machine. The head-mounted display system with an integrated design does not need to obtain image data from an external device, has image data processing capabilities and image display capabilities, and can be used independently.

[0139] Of course, the head-mounted display device 100 may also be a head-mounted display system with a split design. In this case, the display unit and the computing unit of the head-mounted display device 100 may not be integrated into one physical device. For example, VR glasses and VR host, the VR host is integrated with a computing unit, has image data processing capabilities, but does not have image display capabilities. VR glasses are integrated with a display unit, do not have image processing capabilities, have image display capabilities, and need to obtain image data from an external device. VR glasses and VR host need to be used in conjunction.

[0140] After exemplarily introducing the hardware architecture of the head-mounted display device 100, the interaction solution provided in the embodiment of the present application is introduced and explained by taking the head-mounted display device 100 as an example.

[0141] In the embodiment of the present application, the head mounted display device 100 can perform corresponding operations on objects in the three-dimensional environment 20 in response to a two-handed gesture made by the user. A two-handed gesture refers to a gesture that relies on the user's two hands (including the left hand and the right hand). The embodiment of the present application provides two basic two-handed gestures: a click gesture and a slide gesture.

[0142] A click gesture refers to: a first part of a first finger of a first hand of a user touches a second part of a second finger of a second hand. A slide gesture refers to: a first finger of a first hand slides on a second finger of a second hand.

[0143] The first hand and the second hand are different. For example, the first hand and the second hand are the left and right hands of the user, respectively. Usually, the first hand and the second hand both include a thumb, an index finger, a middle finger, a ring finger, and a little finger. The first finger of the first hand can be any one of the thumb, the index finger, the middle finger, the ring finger, and the little finger, and the second finger of the second hand can be any one of the thumb, the index finger, the middle finger, the ring finger, and the little finger.

[0144] Fingers may include parts such as the base and the fingertip. The base refers to the place where the finger is connected to the palm, that is, the part of the metacarpophalangeal joint. The fingertip refers to the top or end of the finger. The first part of the first finger may be the base of the first finger, or the fingertip of the first finger, or the part between the base and the fingertip. Similarly, the first part of the second finger may be the base of the second finger, or the fingertip of the second finger, or the part between the base and the fingertip. The first part and the second part may be any part of the finger, and are not limited here.

[0145] For the click gesture, the click gesture can be further divided into a single-click gesture, a double-click gesture, and a long press gesture according to the number of times the first finger and the second finger touch each other and the continuous contact time of the first finger and the second finger during each touch.

[0146] Exemplarily, a single-click gesture means that the first part of the first finger touches the second part of the second finger once, and the continuous contact time of the first finger and the second finger when touching is less than the first time threshold. In some scenarios, the single-click gesture can also be regarded as a confirmation gesture.

[0147] The long press gesture means that the first part of the first finger touches the second part of the second finger once, and the continuous contact time of the first finger and the second finger when they touch is greater than the second time threshold, and the second time threshold is greater than or equal to the first time threshold.

[0148] The double-click gesture means that the first part of the first finger touches the second part of the second finger twice, and the continuous contact time of the first finger and the second finger each time is less than the first time threshold.

[0149] It can be understood that the specific values ​​of the first time threshold and the second time threshold can be set according to actual needs and are not limited here.

[0150] For the sliding gesture, when the first finger of the first hand slides on the second finger of the second hand, the sliding can be performed in the first direction or in the second direction. The first direction refers to the direction from the wrist of the second hand to the fingertip of the second finger, or from the base of the second finger to the fingertip of the second finger; the second direction refers to the direction from the fingertip of the second finger to the wrist of the second hand, or from the fingertip of the second finger to the base of the second finger.

[0151] For example, the first finger and the second finger are both thumbs, and the first part of the first finger and the second part of the second finger are both fingertips. In this case, the user can make a click gesture by touching the fingertip of the left thumb with the fingertip of the right thumb; the user can make a slide gesture by sliding the thumb of the first hand on the thumb of the second hand. Figure 4A to Figure 4D This example of two-handed gesture is introduced and explained as an example. Figure 4A A two-handed gesture application scenario provided in an embodiment of the present application, Figure 4B A schematic diagram of a single-click gesture of a two-handed cross gesture provided in an embodiment of the present application, Figure 4C A schematic diagram of a double-click gesture of a two-handed cross gesture provided in an embodiment of the present application, Figure 4D and Figure 4E A schematic diagram of a sliding gesture of a two-handed cross gesture provided in an embodiment of the present application.

[0152] like Figure 4AAs shown, in a physical three-dimensional environment 10, the user wears a head-mounted display device 100. The head-mounted display device 100 has one or more cameras facing downward, which can capture hand images within the acquisition range 30. The user places the first hand 41 and the second hand 42 on a physical support plane 11 (such as a table surface) and makes a hand-crossing gesture. At this time, the first hand 41 and the second hand 42 are both within the acquisition range 30, so the head-mounted display device 100 can capture hand images including the first hand 41 and the second hand 42 through the camera facing downward, and perform hand key point detection on the hand image to obtain the hand key points of the first hand 41 and the second hand 42, and then perform gesture recognition and gesture tracking based on the hand key points through the gesture tracking module 220; after recognizing the hand-crossing gesture, the corresponding operation is performed in response to the detected hand-crossing gesture. As shown Figure 4A to Figure 4D As shown, the two-handed crossed gesture means that, except for the thumb, the other four fingers of the first hand 41 and the second hand 42 are in a crossed state, and the clicking gesture is performed by touching the fingertips of the thumbs of the two hands, and the sliding gesture is performed by sliding the thumb of one hand on the thumb of the other hand.

[0153] like Figure 4B As shown, looking from left to right, initially the thumb 43 of the first hand 41 and the thumb 44 of the second hand 42 are not in contact (as shown in the leftmost figure); the thumb 43 of the first hand 41 and the thumb 44 of the second hand 42 are gradually brought closer until the fingertips of the thumb 43 and the thumb 44 touch each other (as shown in the middle figure); the fingertips of the thumb 43 and the thumb 44 touch each other and gradually separate within a period of contact (as shown in the rightmost figure), thus completing the touching of the thumb 43 of the first hand 41 and the thumb 44 of the second hand once.

[0154] exist Figure 4B If the user needs to perform a click gesture or a confirmation gesture, the fingertips of thumb 43 and 44 can be kept in contact for a time shorter than a first time threshold; if the user needs to perform a long press gesture, the fingertips of thumb 43 and 44 can be kept in contact for a time longer than a second time threshold.

[0155] like Figure 4CAs shown, initially, the thumb 43 of the first hand 41 and the thumb 44 of the second hand 42 are not in contact, and then the thumb 43 of the first hand 41 and the thumb 44 of the second hand 42 are gradually brought closer until the fingertips of the thumb 43 and the thumb 44 touch each other; after the fingertips of the thumb 43 and the thumb 44 are in contact for a period of time (the time is less than the first time threshold), the fingertips of the thumb 43 and the thumb 44 are gradually separated, and when the fingertips of the thumb 43 and the thumb 44 are separated to a certain distance, the thumb 43 of the first hand 41 and the thumb 44 of the second hand 42 are gradually brought closer until the fingertips of the thumb 43 and the thumb 44 touch each other again; after the fingertips of the thumb 43 and the thumb 44 are in contact for a period of time (the time is less than the first time threshold), the fingertips of the thumb 43 and the thumb 44 are gradually separated again. In this way, the user completes the double-click gesture.

[0156] That is, the user can make the thumb 43 of the first hand 41 and the thumb 44 of the second hand touch each other twice, and each time when they touch, the fingertip of the thumb 43 and the fingertip of the thumb 44 are in continuous contact for a time period less than the first time threshold, so as to perform a double-click gesture. It is understandable that when the user performs a double-click gesture, the interval time between the first and second touches should be less than the preset time threshold. That is, the user can make the fingertip of the thumb 43 touch the fingertip of the thumb 44 twice in a short time to perform a double-click gesture.

[0157] like Figure 4D As shown, the user can slide the fingertips of the thumb 43 of the first hand 41 on the back of the thumb 44 of the second hand 42 to perform a sliding gesture. The sliding direction can be a first direction or a second direction. The first direction refers to the direction from the phalanges of the thumb 44 of the second hand 42 (the end close to the wrist) to the fingertips of the thumb 44, and the first direction is the direction gradually away from the wrist; the second direction refers to the direction from the fingertips of the thumb 44 of the second hand 42 (the end away from the wrist) to the base of the thumb 44, and the second direction is the direction gradually approaching the wrist.

[0158] like Figure 4E As shown, the fingertip of the thumb 43 of the first hand 41 contacts the back of the thumb 44 of the second hand 44, and the contact position of the thumb 43 and the thumb 44 is position 400. Taking position 400 as the starting point, the fingertip of the thumb 43 slides on the back of the thumb 44 along the first direction, and during the sliding process, the fingertip of the thumb 43 and the back of the thumb 44 continue to contact; the fingertip of the thumb 43 slides from position 400 to position 410, and position 410 is a position on the back of the thumb 44. In some embodiments, the head mounted display device 100 can also obtain the sliding displacement of the thumb 43 on the thumb 44, and at this time, the distance between position 400 and position 410 can be used as the sliding displacement or finger displacement.

[0159] It is understandable that in other embodiments, Figure 4A to Figure 4E The first hand 41 and the second hand 42 in the figure may not be in the crossed state. In this case, except for the thumb, the remaining four fingers of the first hand 41 and the second hand 42 may be in any state. Figure 4F Another hand gesture diagram is shown, except for the thumb 44, the other four fingers of the second hand 42 are in a naturally bent state, and these four fingers are bent and closely together; similarly, the other four fingers of the first hand 41 can also be in a similar state. Figure 4F The state shown, and relying on thumb 44 and thumb 43 to perform a click gesture and a slide gesture.

[0160] In addition, Figure 4A to Figure 4E In the embodiment, the user can perform a click gesture by touching the fingertips of thumb 43 and thumb 44. In other embodiments, the user can also touch the pulp of thumb 43 with the corresponding position on the back of thumb 44 to perform a click gesture. Specifically, a single-click gesture refers to: thumb 43 taps once on the back of thumb 44, and the continuous contact time of thumb 43 and thumb 44 is less than a first time threshold. A double-click gesture refers to: thumb 43 taps twice on the back of thumb 44, and each time the thumb 43 and thumb 44 are in contact for less than the first time threshold. A long press gesture refers to: thumb 43 taps once on the back of thumb 44, and the continuous contact time of thumb 43 and thumb 44 is greater than or equal to a second time threshold.

[0161] In addition to relying on the thumbs of two hands to perform click gestures and slide gestures, two-handed gestures can also be performed through other finger combinations. For example, one finger combination is the index fingers of two hands, that is, the first finger and the second finger are both index fingers, and the first part of the first finger and the second part of the second finger are both fingertips. For another example, another finger combination is the thumb and the index finger. In this case, the first finger is the thumb and the second finger is the index finger; the first part is the tip of the thumb, and the second part is the part of the index finger between the base and the tip of the finger. In this case, the user can make a click gesture by tapping the index finger with the tip of the thumb, and can make a slide gesture by sliding the thumb of one hand on the index finger of the other hand.

[0162] For example, see Figure 5AIn the schematic diagram of the click gesture in the two-hand gesture provided by the embodiment of the present application, the index finger 45 of the first hand 41 is in contact with the position 420 of the thumb 44 of the second hand 42. The user can use the index finger 45 to tap once at the position 420 on the thumb 44 to perform a single-click gesture. The continuous contact time of the index finger 45 and the thumb 44 during the tap is less than the first time threshold; the index finger 45 is used to tap twice at the position 420 on the thumb 44 to perform a double-click gesture. During each tap, the continuous contact time of the index finger 45 and the thumb 44 is less than the first time threshold; the index finger 45 is allowed to separate from the position 420 on the thumb 44 after contacting once, and the continuous contact time is greater than the second time threshold to perform a long press gesture.

[0163] See also Figure 5B In the schematic diagram of the sliding gesture in the two-hand gesture provided in the embodiment of the present application, the user can slide the index finger 45 of the first hand 41 on the back of the thumb 44 of the second hand 42 to perform the sliding gesture. The sliding direction can be the first direction or the second direction.

[0164] After exemplarily introducing the click gesture and the slide gesture provided in the embodiment of the present application, the process of using the click gesture and the slide gesture to interact with objects in the three-dimensional environment 20 is exemplarily introduced below.

[0165] After the user wears the head-mounted display device 100, he selects the object to be operated in the three-dimensional environment 20 by gazing with his eyes or turning his head, and then performs a click gesture or a slide gesture in the physical three-dimensional environment 10 with his hands to perform confirmation operations, single-click operations, double-click operations, long press operations, and slide operations on the object to be operated in the three-dimensional environment 20.

[0166] The head-mounted display device 100 determines the object to be operated in the three-dimensional environment 20 according to the user's eye gaze point or head rotation movement through the eye tracking module 210 or the head aiming module 230, and displays the object to be operated in a display method that is different from other objects. For example, the object to be operated is highlighted so that the user can easily perceive the selected object; in addition, the hand image is captured by a downward-facing camera, and gesture recognition and gesture tracking are performed based on the hand image; when a single-click gesture is recognized, a single-click operation is performed on the object to be operated in the three-dimensional environment 20 in response to the single-click gesture; when a double-click gesture is recognized, a double-click operation is performed on the object to be operated in the three-dimensional environment 20 in response to the double-click gesture; when a long press gesture is recognized, a long press operation is performed on the object to be operated in the three-dimensional environment 20; when a sliding gesture is recognized, a corresponding operation is performed in response to the sliding gesture.

[0167] For example, see Fig. 6AThe schematic diagram of the interactive operation of the movie-watching scene provided by the embodiment of the present application is shown. In the physical three-dimensional environment 10, the user wears the head-mounted display device 100 and places the first hand 41 and the second hand 42 on the physical support 11 to perform two-hand gestures.

[0168] In the movie watching scene, after the user wears the head mounted display device 100, he can see the main console interface 61 in the three-dimensional environment 20. The main console interface 61 includes icons such as the gallery control 62, the music control 63, the movie watching control 64, the note control 65 and the chat control 66. At this time, the user needs to select the movie watching control 64, and let the eye gaze point stay on the movie watching control 64. The head mounted display device 100 determines that the eye gaze point is on the movie watching control 64 according to the eye tracking module 210, and highlights the movie watching control 64 (highlighting is indicated by bold and black in the figure). In addition to highlighting, the movie watching control 64 can also be pulled closer or the movie watching control 64 can be flashed to make the display mode of the selected object different from the display mode of other unselected objects, so that the user can easily perceive the currently selected object.

[0169] After the head mounted display device 100 highlights the viewing control 64, the user can perform a single-click gesture (such as Figure 4B The head mounted display device 100 collects the hand images of the first hand 41 and the second hand 42, processes the hand images based on the gesture tracking module 220, and recognizes the single-click gesture, then performs a click operation or a confirmation operation on the viewing control 64, enters (or displays) the program menu interface 67, and displays the main console interface 61. The program menu interface 67 includes windows 68 to 610 corresponding to programs 1 to 7.

[0170] After the head mounted display device 100 displays the program menu interface 67 in the three-dimensional environment 20, the user can select the program to be watched by the eye gaze point. The head mounted display device 100 determines that the eye gaze point is on the window 610 corresponding to the program 3 through the eye tracking module 210, and then highlights the window 610.

[0171] After the head mounted display device 100 highlights the window 610, the user can perform a single click gesture (such as Figure 4BThe head mounted display device 100 collects the hand images of the first hand 41 and the second hand 42, processes the hand images based on the gesture tracking module 220, and recognizes the single-click gesture, and then performs a single-click operation on the window 610 to enter (or display) the viewing interface 615 by performing the single-click operation to display the program menu interface 67. The viewing interface 615 includes the movie content, the progress bar 616, and the volume control 617. In this way, the user can watch the movie content corresponding to the program 3 through the head mounted display device 100.

[0172] When the user watches the film content corresponding to the program 3 through the head mounted display device 100, the user can perform a sliding gesture in the physical three-dimensional environment 10 to adjust the volume, progress bar, screen brightness, or window size, etc. The sliding direction is related to the adjustment direction, and the sliding amount is related to the adjustment amount.

[0173] For example, see Figure 6B The schematic diagram of adjusting the volume by sliding gesture provided by the embodiment of the present application is shown. After the head-mounted display device 100 displays the viewing interface 615 in the three-dimensional environment 20, the user gazes at the volume control 617 with his eyes. The head-mounted display device 100 determines that the eye gaze point is on the volume control 617 through the eye tracking module 210, flashes or highlights the volume control 617, and waits for the user to confirm whether to adjust the volume. At this time, the user can perform a single-click gesture by touching the thumb 43 of the first hand 41 and the thumb 44 of the second hand 42 in the physical three-dimensional environment 10; after the head-mounted display device 100 recognizes the single-click gesture, it is considered that the user confirms to enter the volume adjustment state, and then the dot 618 and the adjustment bar 619 of the volume control 617 are displayed. The dot 618 can slide on the adjustment bar 619, and the dot 618 is at different positions on the adjustment bar 619 to represent different volume levels.

[0174] like Figure 6B As shown, after the head mounted display device 100 displays the dot 618 and the adjustment bar 619, the dot 618 is initially located at the position 620 of the adjustment bar 619. At this time, the user wants to increase the volume, and performs a sliding gesture in the physical three-dimensional environment 10. Specifically, with the position 400 as the starting point, the finger pulp of the thumb 43 slides along the first direction on the back of the thumb 44 to the position 410; the head mounted display device 100 recognizes the sliding gesture, and after recognizing the finger sliding direction and the finger sliding amount (i.e., the distance between the position 400 and the position 410), in response to the sliding gesture, the dot 618 is slid from the position 620 to the position 621 according to the finger sliding direction and the finger sliding amount to increase the volume.

[0175] exist Figure 6BIn the figure, the sliding amount (or displacement) of thumb 43 on thumb 44 is the distance between position 400 and position 410. The head mounted display device 100 can acquire hand images, obtain hand key points from the hand images, determine the coordinate change of thumb 43 according to the hand key points, and then obtain the displacement and sliding direction.

[0176] The displacement of thumb 43 on thumb 44 is related to the volume adjustment amount, and the sliding direction is related to the volume adjustment direction. The volume adjustment direction includes increase and decrease. Figure 6B When the sliding direction is the first direction, the volume adjustment direction is increasing; when the sliding direction is the second direction, the volume adjustment direction is decreasing. After obtaining the displacement, the head mounted display device 100 can convert the displacement into a volume adjustment amount, and determine the operation amount of the current volume adjustment operation according to the volume adjustment amount, such as Figure 6B As shown, the difference between the volume corresponding to the dot 618 at position 620 and the volume corresponding to the dot 618 at position 621 is the operation amount. In other words, the magnitude of the operation amount is related to the displacement, but the positive or negative of the operation amount (ie, the adjustment direction) is related to the sliding direction.

[0177] Similarly, if the user needs to reduce the volume, the thumb 43 can slide on the back of the thumb 44 along the second direction, so that the head mounted display device 100 can reduce the corresponding volume in response to the sliding direction and displacement of the sliding gesture.

[0178] For example, see Figure 6C The schematic diagram of adjusting the window size by sliding gesture provided by the embodiment of the present application is shown. After the head-mounted display device 100 displays the viewing interface 615 in the three-dimensional environment 20, the user gazes at one of the four corners of the viewing interface 615. The head-mounted display device 100 determines that the eye gaze point is at position 622 through the eye tracking module 210, and then flashes or highlights the viewing interface 615, waiting for the user to confirm whether to adjust the size of the window (or interface, screen). At this time, the user can perform a single-click gesture by touching the thumb 43 of the first hand 41 and the thumb 44 of the second hand 42 in the physical three-dimensional environment 10; after the head-mounted display device 100 recognizes the single-click gesture, it is considered that the user confirms to enter the window size adjustment, and displays a pattern 623 to indicate that the window size adjustment state is currently in progress.

[0179] After the head mounted display device 100 enters the window size adjustment state, if the user wants to increase the window size of the viewing interface 615, a sliding gesture is performed in the physical three-dimensional environment 10. Specifically, with position 400 as the starting point, the fingertip of the thumb 43 slides along the first direction on the back of the thumb 44 to position 410; after the head mounted display device 100 recognizes the sliding gesture, the sliding direction and the sliding amount of the sliding gesture, the window size of the viewing interface 615 is increased according to the sliding direction and the sliding amount in response to the sliding gesture.

[0180] The displacement is related to the window size adjustment amount, and the sliding direction is related to the window size adjustment direction. The window size adjustment direction includes increase and decrease. Figure 6C In the example, when the sliding direction is the first direction, the window size adjustment direction is increasing; when the sliding direction is the second direction, the window size adjustment direction is decreasing. After obtaining the displacement amount, the head mounted display device 100 can convert the displacement amount into the window size adjustment amount, and determine the operation amount of the current window adjustment operation according to the window size adjustment amount.

[0181] Similarly, if the user needs to reduce the window size, the user can slide thumb 43 on the back of thumb 44 along the second direction, so that the head-mounted display device 100 can reduce the window size of the viewing interface 615 in response to the sliding direction and displacement of the sliding gesture.

[0182] It should be pointed out that the progress bar adjustment of video and audio playback, as well as the brightness adjustment, are similar to the volume adjustment and window size adjustment processes mentioned above. Users can first select the progress bar or brightness control by gazing at a point with their eyes or aiming with their head, and then adjust the progress bar or brightness through sliding gestures. I will not go into details here.

[0183] As can be seen from the above, the user can use the sliding gesture to conveniently and quickly adjust the volume, brightness, window size, progress bar and other objects that are difficult to operate in the three-dimensional environment 20. In addition, the user can also control the sliding of the page (or interface, webpage, etc.) in the three-dimensional environment 20 through the sliding gesture.

[0184] For example, see Figure 7 The schematic diagram of controlling page sliding by sliding gesture provided by an embodiment of the present application is shown, and the head mounted display device 100 displays page 71 in the three-dimensional environment 20. At this time, if the user wants to slide up (or scroll) page 71, the thumb 43 can slide along the first direction on the back of the thumb 44 in the physical three-dimensional environment 10 to perform the sliding gesture. When the head mounted display device 100 recognizes the sliding gesture and recognizes that the sliding direction is the first direction, the page 71 is controlled to slide up (or scroll).

[0185] Similarly, if the user wants to slide (or scroll) page 71 downward, the user can slide thumb 43 on the back of thumb 44 along the second direction in the physical three-dimensional environment 10 to perform a sliding gesture. When the head mounted display device 100 recognizes the sliding gesture and recognizes that the sliding direction is the second direction, the page 71 is controlled to slide downward (or scroll).

[0186] Furthermore, the sliding amount of the page may be related to the displacement amount of the finger. At this time, after recognizing the sliding gesture, the head mounted display device 100 further recognizes the sliding direction and the finger displacement amount, converts the finger displacement amount into the sliding amount of the page sliding up or down; then, in response to the sliding gesture, according to the sliding direction, controls the page 71 to slide up or down by the corresponding sliding amount.

[0187] It is understandable that, in addition to controlling the same page to slide up or down, the sliding gesture can also be used to slide to the previous page or slide to the next page (i.e., turn the page). For example, when the thumb 43 slides along the first direction on the back of the thumb 44, the page is turned up; when the thumb 43 slides along the second direction on the back of the thumb 44, the page is turned down.

[0188] It is worth pointing out that compared with the method of controlling page sliding through gestures such as flicking the wrist upward and bending the wrist downward, the sliding gesture provided in the embodiment of the present application controls page sliding. Long-term operation will not increase wrist pressure, alleviate wrist fatigue problems, and provide better user operation comfort.

[0189] In addition to controlling objects in the three-dimensional environment 20 through sliding gestures, movable parts of the optical display module 200 in the head-mounted display device 100 can also be controlled to move through sliding gestures to perform diopter adjustment.

[0190] The display clarity of the head-mounted display device 100 is related to the screen quality in the optical display module, the user's interpupillary distance (IPD), and vision. In the related art, the head-mounted display device 100 supports diopter adjustment, but the user needs to take off the head-mounted display device 100 before officially using the head-mounted display device 100, and manually rotate the lens barrel to adjust the diopter. In this diopter adjustment method, the user needs to take off the device when adjusting, so that the user cannot adjust by observing the screen imaging, which is very troublesome and affects the user's interactive operation experience.

[0191] The technical solution for adjusting the diopter through sliding gestures provided in the embodiments of the present application allows the user to conveniently adjust the diopter (myopia / hyperopia degree) of the user's left and right eyes through two-handed gestures when wearing a head-mounted display device, without having to remove the head-mounted display device and manually rotate the lens barrel for adjustment, thereby further improving the user's interactive operation experience.

[0192] See also Figure 8 The schematic diagram of the relationship between the eye and the optical display module 200 provided in the embodiment of the present application is shown. The optical display module 200 includes a lens group 82 and a screen 83. The optical path between the human eye 81 and the lens group 82 and the screen 83 can be as follows: Figure 8 As shown. The movable component in the optical display module 200 can move in a direction close to the human eye or in a direction away from the human eye under the control of the sliding gesture. The movable component in the optical display module 200 can be a lens group 82 or a screen 83.

[0193] In a specific application, after the user wears the head-mounted display device 100, the head-mounted display device 100 can perform a prompt operation in the three-dimensional environment 20 to prompt the user whether to enter the diopter adjustment mode. If the user confirms to enter the diopter adjustment mode, the diopter adjustment process for the left eye and the right eye will be entered respectively.

[0194] The prompt operation can be a voice prompt and / or a text prompt. For example, see Fig. 9A In the prompt operation diagram shown, the user wears the head mounted display device 100 in the physical three-dimensional environment 10, and after adjusting the pupil distance, the head mounted display device 100 presents the three-dimensional environment 20 to the user, and the three-dimensional environment 20 includes a text prompt interface 91. The text prompt interface 91 includes a prompt text "Do you need myopia / hyperopia adjustment to see the pictures and text on the screen clearly", and the options of "yes" and "no". Optionally, in addition to displaying the prompt text in the three-dimensional environment 20, the head mounted display device 100 can also play a prompt voice 92 in the physical three-dimensional environment 10.

[0195] The user can confirm whether to adjust myopia / hyperopia by voice, eye gaze or head movement. Fig. 9A In the scenario shown, the user fixes his eyes on the "Yes" option in the text prompt interface 91 and touches the thumb tip of the first hand 41 and the thumb tip of the second hand 42 together once in the physical three-dimensional environment 10 to perform a confirmation operation and confirm the selection of the "Yes" option.

[0196] After entering the diopter adjustment process for the left eye and the right eye, you can adjust the diopter of the left eye or the right eye first, and then adjust the diopter of the other eye. Fig. 9A , Fig. 9B and Fig. 9C The myopia adjustment process of the left eye is introduced and explained. Fig. 9B A schematic diagram of adjusting myopia through a sliding gesture provided in an embodiment of the present application, Fig. 9C A schematic diagram of the diopter adjustment process provided in an embodiment of the present application.

[0197] like Fig. 9A and Fig. 9C As shown, the first hand 41 and the second hand 42 of the user are within the acquisition range 30 of the binocular camera of the head mounted display device 100. The binocular camera acquires the user's hand image including the first hand 41 and the second hand 42 in real time; the gesture tracking module 220 processes the user's hand image, obtains the 3D key points of both hands, and performs gesture recognition and gesture tracking based on the 3D key points of both hands, and identifies whether to long press, single click, double click, slide up, and slide down. Sliding up means that the first finger slides on the second finger along the first direction, and sliding down means that the first finger slides on the second finger along the second direction. If the diopter adjustment module determines the sliding gesture, it further determines the sliding direction, and obtains the finger displacement according to the coordinates of the 3D key points of both hands; finally, according to the sliding direction and the finger displacement, in response to the sliding gesture, the movable component in the optical display module 200 is controlled to move. Among them, the moving direction of the movable component is related to the sliding direction, and the movement amount is related to the finger displacement.

[0198] exist Fig. 9A In the example, if the user's eyes focus on the "Yes" option in the text prompt interface 91 and the thumb tip of the first hand 41 touches the thumb tip of the second hand 42 once to perform a click gesture, the head mounted display device 100 captures the user's hand image, recognizes the click gesture based on the user's hand image, and then responds to the click gesture to enter the left eye myopia degree adjustment process.

[0199] like Fig. 9B As shown, after entering the left eye myopia adjustment process, the head mounted display device 100 displays the left eye myopia adjustment interface 93 in the three-dimensional environment 20. The pointing line 94 in the left eye myopia adjustment interface 93 rotates around the center of the circle, and represents different myopia degrees at different positions. In the left eye myopia adjustment interface 93, when the pointing line 94 rotates in the clockwise direction, it represents an increase in myopia; when the pointing line 94 rotates in the counterclockwise direction, it represents a decrease in myopia.

[0200] If the user slides the thumb 43 on the thumb 44 in a first direction (i.e., upward) to adjust the degree of myopia, the head-mounted display device 100 can control the lens group 82 or the screen 83 to move in a direction close to the human eye and allow the pointing line 94 to rotate in a clockwise direction.

[0201] If the user slides the thumb 43 on the thumb 44 in the second direction (i.e., downward) to adjust the myopia degree to reduce, the head-mounted display device 100 can control the lens group 82 or the screen 83 to move in a direction away from the human eye and allow the pointing line 94 to rotate in a counterclockwise direction.

[0202] In addition, the head-mounted display device 100 can also obtain the displacement (or sliding amount) of the thumb 43 on the thumb 44 according to the 3D key points of both hands, and convert the thumb displacement into the displacement of the movable part and the change in myopia degree; according to the displacement of the movable part, control the movable part to move a corresponding distance; according to the change in myopia degree, control the pointing line 94 to rotate a corresponding angle.

[0203] The right eye myopia adjustment process and the hyperopia adjustment process can be similar to the left eye myopia adjustment process mentioned above, and will not be repeated here.

[0204] The user wears the head mounted display device 100, and after adjusting the diopter conveniently and quickly by sliding the first finger on the second finger along the first direction or the second direction, the head mounted display device 100 can display the main console interface 61, so that the user can continue to use the head mounted display device. In this way, when the user wears the head mounted display device 100, the user can also adjust the myopia / hyperopia degree of the user's left eye and right eye through hand gestures, without removing the product and manually rotating the lens barrel for adjustment, and the head mounted display device 100 can be used directly after adjustment.

[0205] After exemplarily introducing various interactive operation scenarios based on two-handed gestures, the interactive solution provided in the embodiment of the present application is introduced and explained in conjunction with a flowchart.

[0206] See also Fig.10 , is a flow chart of an interaction method of the head mounted display device 100 provided in an embodiment of the present application, the method may include the following steps:

[0207] Step S101 : The head mounted display device 100 displays a view of the three-dimensional environment 20 .

[0208] It is understandable that the three-dimensional environment 20 displayed by the head mounted display device 100 may be an AR three-dimensional environment, a VR three-dimensional environment or an MR three-dimensional environment. The view of the three-dimensional environment includes one or more interactive objects, which may be interfaces, pages, controls, icons, etc.

[0209] Step S102: If the head-mounted display device 100 detects that the first part of the first finger of the user's first hand collides with the second part of the second finger of the second hand, the first operation is performed in response to the collision of the first finger and the second finger, and the first hand and the second hand are different; and / or if the head-mounted display device 100 detects that the first finger slides on the second finger, the second operation is performed in response to the sliding of the first finger on the second finger.

[0210] It can be understood that the head-mounted display device 100 may only support a click gesture, that is, a gesture in which the first part of the first finger of the user's first hand touches the second part of the second finger of the second hand; it may also only support a sliding gesture, that is, a gesture in which the first finger slides over the second finger; of course, it may also support both a click gesture and a sliding gesture.

[0211] Further, the click gesture may also include one or more of a single-click gesture, a double-click gesture, and a long-press gesture. In this case, the first operation may be a click operation, a confirmation operation, a double-click operation, or a long-press operation. The first operation is an operation on an object in the three-dimensional environment 20, which is usually a virtual object. For example, a click operation on a control, a click operation on a window, etc.

[0212] The second operation may be an operation on an object in the three-dimensional environment 20, for example, the second operation may be a volume adjustment operation, a brightness adjustment operation, a window size adjustment operation, or a progress bar adjustment operation, etc. The volume control, the brightness control, the library, the progress bar, etc. are all virtual objects in the three-dimensional environment 20.

[0213] The second operation may also be an operation of a movable component in the optical display module 200 . In this case, the second operation may be an operation of controlling the movement of the movable component.

[0214] Optionally, the operation direction of the second operation is related to the sliding direction of the first finger on the second finger, and / or the operation amount of the second operation is related to the displacement of the first finger on the second finger. In this way, associating the finger sliding direction with the operation direction, and associating the finger sliding displacement with the operation amount, further improves the interactive operation experience when interacting through the sliding gesture of the embodiment of the present application.

[0215] Optionally, the first finger is a thumb or an index finger, the second finger is a thumb or an index finger, the first part is a fingertip, and the second part is a fingertip or a back of a finger. In this way, when the user performs interactive operations through two-handed gestures, the operation comfort is higher.

[0216] As can be seen from the above, when the embodiments of the present application perform interactive operations through two-handed gestures (including click gestures and / or sliding gestures), the user can obtain stronger tactile feedback through the contact and sliding between the fingers of the two hands, and the interactive operation is more comfortable; in addition, the probability of the user unconsciously touching and sliding the fingers of the two hands is low, which reduces the probability of false touches and provides a better user interactive operation experience.

[0217] Combine the following Fig.11 , which provides an exemplary introduction to the process of interacting with the head mounted display device 100 through a click gesture. Fig.11 , is a flow chart of another interaction method of the head mounted display device 100 provided in an embodiment of the present application, the method may include the following steps:

[0218] Step S201: The head mounted display device 100 displays a view of a three-dimensional environment 20, where the view of the three-dimensional environment 20 includes at least one interactive object.

[0219] Interactive objects include, but are not limited to: application icons, application interfaces, windows, and controls, etc. For example, a movie viewing window, a volume control, a playback progress bar, and a brightness control, etc.

[0220] Step S202: The head mounted display device 100 displays the target object in a target display mode in response to one or more operations, wherein the one or more operations are operations of selecting the target object from at least one interactive object, and the target display mode is different from the display mode of other objects.

[0221] The target display mode is used to highlight the target object selected by the user so that the user can perceive which object is currently selected. Exemplarily, the target display mode is highlight display. For example, Fig. 6A As shown, window 610 and movie viewing controls 64 are highlighted.

[0222] The one or more operations may be, for example, eye gazing or head turning, that is, a target object may be selected from multiple objects by operations such as eye gazing or head movement aiming.

[0223] Step S203: If the head mounted display device 100 detects that the first part of the first finger collides with the second part of the second finger once, and the continuous contact time of the first finger and the second finger when they collide is less than the first time threshold, then in response to the collision between the first finger and the second finger, a single-click operation is performed on the target object; if the first part of the first finger collides with the second part of the second finger once, and the continuous contact time of the first finger and the second finger when they collide is greater than the second time threshold, then in response to the collision between the first finger and the second finger, a long press operation is performed on the target object; if the first part of the first finger collides with the second part of the second finger twice, and the continuous contact time of the first finger and the second finger each time is less than the first time threshold, then in response to the collision between the first finger and the second finger, a double-click operation is performed on the target object. The second time threshold is greater than or equal to the second time threshold.

[0224] For example, Fig. 6A As shown, after the user selects the movie viewing control 64 by gazing at the movie viewing control 64, the user touches the two fingers of the first hand 41 and the second hand 42 together once in the physical three-dimensional environment 10 to confirm entering the movie viewing control 64. The head-mounted display device 100 responds to the detected single-click gesture, performs a single-click operation on the movie viewing control 64, and displays the program menu interface 67 of the movie viewing control 64.

[0225] It can be understood that, through the two-handed gesture of touching the fingers of one hand with the fingers of another hand, a confirmation operation, a single-click operation, a double-click operation or a long press operation is performed on a target object selected in a three-dimensional environment, so that the user can obtain stronger tactile feedback; and the probability of the user inadvertently making the two-handed gesture is low, thereby reducing the probability of false touches and further improving the user's interactive operation experience.

[0226] Optionally, the first part is the fingertip, and the second part is the fingertip or the back of the finger.

[0227] See also Fig.12 , is a flow chart of another interaction method of the head mounted display device 100 provided in an embodiment of the present application, the method may include the following steps:

[0228] Step S301: The head mounted display device 100 displays a view of a three-dimensional environment 20, where the view of the three-dimensional environment includes at least one interactive object.

[0229] Step S302: The head mounted display device 100 selects a target object from at least one interactive object in response to one or more operations.

[0230] The one or more operations may be, for example, eye gaze or head rotation, that is, a target object may be selected from multiple objects by eye gaze or head movement.

[0231] For example, see Figure 6B The user looks at the volume control 617 to select the volume control 617 as the target object for interaction.

[0232] Step S303: If the head mounted display device 100 detects that the first part of the first finger collides with the second part of the second finger, it performs a confirmation operation, a single click operation, a double click operation, or a long press operation.

[0233] For example, see Figure 6B , the user gazes at the volume control 617 to select the volume control 617 as the target object for interaction. Further, after gazing at the volume control 617, the user performs a click gesture in the physical three-dimensional environment 10 to confirm the operation of the volume control 617, that is, enters the volume adjustment state.

[0234] Among them, steps S301 to S303 can refer to Fig.11 The relevant content will not be repeated here.

[0235] Step S304: The head mounted display device 100 detects that the first finger slides on the second finger.

[0236] Step S305: The head-mounted display device 100 performs a second operation on the target object in response to the sliding of the first finger on the second finger; wherein the operation amount of the second operation is related to the displacement of the first finger on the second finger, and the adjustment direction of the second operation is related to the sliding direction of the first finger on the second finger, and the adjustment direction includes increase and decrease.

[0237] Optionally, if the sliding direction of the first finger on the second finger is the first direction, the adjustment direction of the second operation is to increase; if the sliding direction of the first finger on the second finger is the second direction, the adjustment direction of the second operation is to decrease. In this way, the interactive operation logic is more reasonable and more in line with user habits, making the user interactive operation more comfortable.

[0238] Of course, if the sliding direction of the first finger on the second finger is the first direction, the adjustment direction of the second operation may also be decreasing. Correspondingly, when the sliding direction is the second direction, the adjustment direction of the second operation is increasing.

[0239] Optionally, the target object may be a volume control, a brightness control, a window or a progress bar control, and in this case, the second operation is correspondingly: increasing or decreasing the volume, increasing or decreasing the brightness, increasing or decreasing the window size, or increasing or decreasing the playback progress. It is understandable that the leftmost side of the playback progress bar is the playback start time point, and the rightmost side is the playback end time point. Increasing the playback progress means adjusting the progress bar towards the playback end time point, and decreasing the progress bar means adjusting the progress bar towards the playback start time point.

[0240] It is worth pointing out that it is not convenient to operate objects that are not easy to adjust, such as volume, brightness, window size, and playback progress bar, through a single-handed pinch gesture, and the interactive operation is cumbersome. However, the embodiment of the present application can conveniently adjust these objects through a gesture of sliding a finger of one hand on the finger of another hand, further improving the user's interactive operation experience.

[0241] For example, see Figure 6B After entering the volume adjustment state, the user slides the thumb 43 on the thumb 44 along the first direction in the physical three-dimensional environment 10. After the head-mounted display device 100 detects the sliding gesture, it responds to the sliding gesture and performs a volume adjustment operation on the volume control.

[0242] The operation amount of the volume adjustment operation is related to the displacement amount (or sliding amount) of the thumb 43 on the thumb 44, and the adjustment direction of the volume adjustment operation is related to the sliding direction of the thumb 43 on the thumb 44. At this time, the sliding direction is the first direction, that is, increasing the volume. The head-mounted display device 100 can determine the displacement amount of the thumb 43 on the thumb 44 based on the 3D key point coordinate information of both hands, and then convert the displacement amount into the operation amount, and increase the volume to a corresponding value based on the operation amount and the sliding direction.

[0243] Among them, you can refer to the adjustment of window size by sliding gestures. Figure 6C The examples shown will not be described in detail here.

[0244] It can be seen that the gesture of sliding the first finger on the second finger can be used to control the increase or decrease of target objects such as volume, brightness, myopia, screen size, video / music progress bar, etc. in the three-dimensional environment 20, and the operation amount and adjustment direction are respectively related to the finger displacement and sliding direction, so that the user can obtain stronger tactile feedback and can operate the target objects in the three-dimensional environment 20 more conveniently.

[0245] In other embodiments, if the head mounted display device 100 only supports the sliding gesture, does not support the click gesture, or supports both the sliding gesture and the click gesture, the user can interact with the head mounted display device 100 in other ways (different from the click gesture) to make the head mounted display device 100 perform a confirmation operation, a single click operation, a long press operation, and a double click operation; then, the user interacts with the head mounted display device 100 through the sliding gesture, so that the head mounted display device 100 performs a second operation in response to the sliding gesture.

[0246] It is understandable that the object of the second operation may be different from the object of the first operation. For example, the first operation is a click operation on a control of a movie viewing scene, and the second operation may be an operation on a page.

[0247] See also Fig.13 , is a flow chart of an interaction method of the head mounted display device 100 provided in an embodiment of the present application, the method may include the following steps:

[0248] Step S401: The head mounted display device 100 displays a view of a three-dimensional environment 20, where the view of the three-dimensional environment includes a page.

[0249] It should be noted that the page may refer to a web page or various interfaces.

[0250] Step S402: If the head-mounted display device 100 detects that a first finger of a first hand of a user slides on a second finger of a second hand along a first direction, then in response to the sliding of the first finger on the second finger, the page is slid in the first sliding direction, where the first direction is from the wrist of the second hand to the fingertip of the second finger, and the first hand and the second hand are different.

[0251] The first hand and the second hand may refer to the left hand and the right hand of the same user respectively. The first hand may be the left hand or the right hand, and correspondingly, the second hand is the other hand.

[0252] The first finger and the second finger can be any one of the fingers. Optionally, the first finger is a thumb or an index finger, and the second finger is a thumb or an index finger. In this way, when the user interacts with an object in a three-dimensional environment through a two-handed gesture, the operation comfort is higher.

[0253] Step S403: If the head-mounted display device 100 detects that the first finger slides on the second finger along the second direction, then in response to the sliding of the first finger on the second finger, the page is slid in the second sliding direction, and the second direction is the direction from the fingertip of the second finger to the wrist of the second hand, and the first sliding direction and the second sliding direction are different.

[0254] The first sliding direction may be upward, downward, left or right. Similarly, the second sliding direction may also be upward, downward, left or right, but the first sliding direction and the second sliding direction are different. Figure 7 As shown, the first sliding direction is upward, and the second sliding direction is downward; when the thumb 43 slides on the thumb 44 along the first direction, the page slides upward; when the thumb 43 slides on the thumb 44 along the second direction, the page slides downward.

[0255] It is understandable that, through the gesture of sliding the first finger on the second finger, the same page can be controlled to slide up or down (or scroll), and in the scenario where multiple pages exist simultaneously in the three-dimensional environment 20, the page can be controlled to slide left, right, up and down, for example Figure 1 In the scene shown, there are interfaces 21 to 23 in the three-dimensional environment 20, and the user controls the interface to slide forward, backward, upward or downward through sliding gestures. Alternatively, the page can be turned forward or backward.

[0256] The embodiment of the present application controls the page to slide (or scroll) in the first sliding direction or the second sliding direction in response to the sliding of the first finger of the first hand on the second finger of the second hand, so that the user does not need to bend the wrist downward and bounce the wrist upward to slide the page, and the wrist pressure will not increase during long-term operation, which alleviates the wrist fatigue problem and improves the user's interactive operation comfort; in addition, when the fingers of one hand slide on the fingers of the other hand, the user can obtain stronger trigger feedback and the user interactive operation experience is better.

[0257] Optionally, the sliding amount of the page is related to the displacement of the first finger on the second finger. The head mounted display device 100 can obtain the displacement (or sliding amount) of the first finger on the second finger according to the coordinate information of the 3D key points of both hands; convert the displacement into the sliding amount, and control the page to slide in the first sliding direction or the second sliding direction according to the corresponding displacement.

[0258] See also Fig.14 , is a flow chart of another interaction method of the head mounted display device 100 provided in an embodiment of the present application, the method may include the following steps:

[0259] Step S501 : The head mounted display device 100 displays a view of the three-dimensional environment 20 .

[0260] For example, Fig. 9A As shown, the view of the three-dimensional environment 20 may include a prompt interface 91 and the like.

[0261] Step S502: The head mounted display device 100 enters a diopter adjustment mode in response to one or more operations.

[0262] One or more operations are used to cause the head mounted display device 100 to enter a diopter adjustment mode. For example, see Fig. 9A After the head mounted display device 100 executes the text prompt and the semantic prompt, the user can confirm entering the diopter adjustment mode by performing eye gaze operations and single-click gestures.

[0263] Step S503: The head mounted display device 100 detects that the first finger slides on the second finger in the diopter adjustment mode.

[0264] Step S504: In response to the sliding of the first finger on the second finger, the head mounted display device 100 controls the movable component in the optical display module 200 to move. The movement amount of the movable component is related to the displacement amount of the first finger on the second finger, and the movement direction of the movable component is related to the sliding direction of the first finger on the second finger; the movable component is a lens or a display screen.

[0265] Optionally, if the sliding direction of the first finger on the second finger is the first direction, the moving direction is from the movable part to the human eye, that is, the direction close to the human eye; if the sliding direction of the first finger on the second finger is the second direction, the moving direction is from the human eye to the movable part, that is, the direction away from the human eye. In this way, when the diopter is adjusted by the two-handed gesture of sliding the first finger on the second finger, the operation logic is more reasonable and more in line with user habits, making the user interaction operation more comfortable.

[0266] Of course, if the sliding direction of the first finger on the second finger is the first direction, the moving direction is the direction away from the human eye; if the sliding direction of the first finger on the second finger is the second direction, the moving direction is the direction close to the human eye.

[0267] Through a two-handed gesture of sliding the first finger on the second finger to control the movable parts in the optical display module to be closer to or farther away from the human eye, the user can still conveniently adjust the refractive power (myopia / hyperopia degree) of the left and right eyes through two-handed gestures when wearing a head-mounted display device, without having to remove the head-mounted display device and manually rotate the lens barrel for adjustment, further improving the user's interactive operation experience.

[0268] Corresponding to the method described in the above embodiment, Fig.15 The schematic block diagram of the structure of the head mounted display device 100 provided in the embodiment of the present application is shown. For the convenience of description, only the parts related to the embodiment of the present application are shown.

[0269] Reference Fig.15 , the head mounted display device 100 may include: a processing module 1501, an optical display module 1502 and a gesture tracking module 1503;

[0270] The optical display module 1502 is used to display a view of a three-dimensional environment;

[0271] The gesture tracking module 1503 is used to recognize user gestures;

[0272] The processing module 1501 is used to: display a view of a three-dimensional environment through the optical display module 1502; if it is detected through the gesture tracking module 1503 that the first part of the first finger of the user's first hand collides with the second part of the second finger of the second hand, then in response to the collision of the first finger and the second finger, perform a first operation, the first hand and the second hand are different; and / or if it is detected through the gesture tracking module 1503 that the first finger slides on the second finger, then in response to the sliding of the first finger on the second finger, perform a second operation.

[0273] In some possible implementations, the operation direction of the second operation is related to the sliding direction of the first finger on the second finger, and / or the operation amount of the second operation is related to the displacement amount of the first finger on the second finger.

[0274] In some possible implementations, the head mounted display device 100 further includes an eye tracking module and / or a head aiming module. The view of the three-dimensional environment includes at least one interactive object; the processing module 1501 is further used to: display the target object in a target display mode in response to one or more operations through the eye tracking module and / or the head aiming module, wherein the one or more operations are operations of selecting the target object from at least one interactive object, and the target display mode is different from the display mode of non-target objects;

[0275] The processing module 1501 is specifically used for: if it is detected through the gesture tracking module 1503 that the first part of the first finger collides with the second part of the second finger once, and the continuous contact time of the first finger and the second finger when they collide is less than the first time threshold, then in response to the collision between the first finger and the second finger, a confirmation operation is performed or a single-click operation is performed on the target object; if it is detected through the gesture tracking module 1503 that the first part of the first finger collides with the second part of the second finger once, and the continuous contact time of the first finger and the second finger when they collide is greater than the second time threshold, then in response to the collision between the first finger and the second finger, a long press operation is performed on the target object, and the second time threshold is greater than or equal to the first time threshold; if it is detected through the gesture tracking module 1503 that the first part of the first finger collides with the second part of the second finger twice, and the continuous contact time of the first finger and the second finger each time is less than the first time threshold, then in response to the collision between the first finger and the second finger, a double-click operation is performed on the target object; the first operation is a confirmation operation, a single-click operation, a long press operation, or a double-click operation.

[0276] In some possible implementations, the processing module 1501 is specifically used to: detect, through the gesture tracking module 1503, a first finger sliding along the upper edge of a second finger; and perform a second operation on the target object in response to the sliding direction and displacement of the first finger on the second finger; wherein the operation amount of the second operation is related to the displacement of the first finger on the second finger, and the adjustment direction of the second operation is related to the sliding direction of the first finger on the second finger, and the adjustment direction includes increase and decrease.

[0277] In some possible implementations, if the sliding direction of the first finger on the second finger is the first direction, the adjustment direction of the second operation is increase; if the sliding direction of the first finger on the second finger is the second direction, the adjustment direction of the second operation is decrease; wherein the first direction is the direction from the wrist of the second hand to the fingertip of the second finger, and the second direction is the direction from the fingertip of the second finger to the wrist of the second hand.

[0278] In some possible implementations, the target object is a volume control, a brightness control, a window or a progress bar control, and the second operation is: increasing or decreasing the volume, increasing or decreasing the brightness, increasing or decreasing the window size, or increasing or decreasing the playback progress.

[0279] In some possible implementations, the processing module 1501 is specifically used for: if it is detected through the gesture tracking module 1503 that the first finger slides on the second finger along the first direction, then in response to the sliding of the first finger on the second finger, sliding the page in the first sliding direction, the first direction is the direction from the wrist of the second hand to the fingertip of the second finger; if it is detected through the gesture tracking module 1503 that the first finger slides on the second finger along the second direction, then in response to the sliding of the first finger on the second finger, sliding the page in the second sliding direction, the second direction is the direction from the fingertip of the second finger to the wrist of the second hand, and the first sliding direction and the second sliding direction are different; wherein, the second operation is a page sliding operation, and the sliding amount of the page is related to the displacement of the first finger on the second finger.

[0280] In some possible implementations, the optical display module 1502 includes a lens and a display screen; the processing module 1501 is further configured to: enter a diopter adjustment mode in response to one or more operations;

[0281] The processing module 1501 is specifically used for: in the diopter adjustment mode, detecting the sliding of the first finger on the second finger through the gesture tracking module 1503; in response to the sliding of the first finger on the second finger, controlling the movable component in the optical display module 1502 to move; wherein the movement amount of the movable component is related to the displacement amount of the first finger on the second finger, and the movement direction of the movable component is related to the sliding direction of the first finger on the second finger; the movable component is a lens or a display screen; and the second operation is to control the movement of the movable component.

[0282] In some possible implementations, if the sliding direction of the first finger on the second finger is a first direction, the moving direction is from the movable component to the human eye; if the sliding direction of the first finger on the second finger is a second direction, the moving direction is from the human eye to the movable component.

[0283] In some possible implementations, the first finger is a thumb or an index finger, the second finger is a thumb or an index finger, the first part is a fingertip, and the second part is a fingertip or a back of a finger.

[0284] The above-mentioned head-mounted display device 100 has the function of implementing the above-mentioned interaction method, which can be implemented by hardware or by hardware executing corresponding software. The hardware or software includes one or more modules corresponding to the above-mentioned functions, and the modules can be software and / or hardware.

[0285] It should be noted that the information interaction, execution process, etc. between the above-mentioned head-mounted display devices 100 are based on the same concept as the method embodiment of the present application. Their specific functions and technical effects can be found in the method embodiment part and will not be repeated here.

[0286] The technicians in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In practical applications, the above-mentioned function allocation can be completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated in a processing unit, or each unit can exist physically separately, or two or more units can be integrated in one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, which will not be repeated here.

[0287] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present application implements all or part of the processes in the above-mentioned embodiment method, which can be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned various method embodiments when executed by the processor. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium can at least include: any entity or device that can carry the computer program code to the camera / terminal device, recording medium, computer memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, RandomAccess Memory), electric carrier signal, telecommunication signal and software distribution medium. For example, a USB flash drive, a mobile hard disk, a magnetic disk or an optical disk. In some jurisdictions, according to legislation and patent practice, computer-readable media cannot be electric carrier signals and telecommunication signals.

[0288] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0289] In the embodiments provided in the present application, it should be understood that the disclosed devices, electronic devices and methods can be implemented in other ways. For example, the device / electronic device embodiments described above are merely schematic. For example, the division of the modules or units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0290] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0291] The head-mounted display device provided in an embodiment of the present application may include a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, a method as described in any one of the above method embodiments is implemented.

[0292] The embodiment of the present application further provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments can be implemented.

[0293] An embodiment of the present application provides a computer program product. When the computer program product is run on a head-mounted display device, the head-mounted display device can implement the steps in the above-mentioned method embodiments when executing the computer program product.

[0294] The present application also provides a chip system, which includes a processor coupled to a memory, and the processor executes a computer program stored in the memory to implement the methods described in the above method embodiments. The chip system can be a single chip or a chip module composed of multiple chips.

[0295] In the above embodiments, the description of each embodiment has its own emphasis. For the part that is not described or recorded in detail in a certain embodiment, please refer to the relevant description of other embodiments. It should be understood that the size of the sequence number of each step in the above embodiments does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiment of the present application. In addition, in the description of the present application specification and the attached claims, the terms "first", "second", "third", etc. are only used to distinguish the description, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Therefore, the features defined as "first", "second", "third", and "fourth" can explicitly or implicitly include one or more of the features. In addition, it should be understood that at least one of the embodiments of the present application includes one or more; wherein, multiple means greater than or equal to two. In the embodiment of the present application, "and / or" is only a kind of association relationship describing the associated objects, indicating that there can be three relationships, for example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character “ / ” in this article generally indicates that the previous and next associated objects are in an “or” relationship.

[0296] References to "one embodiment" or "some embodiments" etc. described in the specification of this application mean that one or more embodiments of the present application include specific features, structures or characteristics described in conjunction with the embodiment. Therefore, the phrases "in one embodiment", "in some embodiments", "in some other embodiments", "in some other embodiments", etc. appearing in different places in this specification do not necessarily refer to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized in other ways.

[0297] Finally, it should be noted that the above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions within the technical scope disclosed in the present application should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. An interactive method for a head mounted display device, characterized in that: Applied to a head mounted display device, the method comprises: Displaying a view of a three-dimensional environment; If a first portion of a first finger of a first hand of a user is detected to collide with a second portion of a second finger of a second hand, performing a first operation in response to the collision of the first finger and the second finger, the first hand and the second hand being different; And / or if it is detected that the first finger slides on the second finger, a second operation is performed in response to the sliding of the first finger on the second finger.

2. The method according to claim 1, characterized in that The view of the three-dimensional environment includes at least one interactable object; the method further comprising: In response to one or more operations, displaying the target object in a target display mode, wherein the one or more operations are operations of selecting the target object from the at least one interactive object, and the target display mode is different from a display mode of non-target objects; If it is detected that a first part of a first finger of a first hand of a user collides with a second part of a second finger of a second hand, in response to the collision of the first finger and the second finger, a first operation is performed, including: If it is detected that the first part of the first finger and the second part of the second finger collide once, and the continuous contact time of the first finger and the second finger when they collide is less than a first time threshold, then in response to the collision of the first finger and the second finger, a confirmation operation is performed or a single-click operation is performed on the target object; If it is detected that the first part of the first finger and the second part of the second finger collide once, and the continuous contact time of the first finger and the second finger during the collision is greater than the second time threshold, then in response to the collision of the first finger and the second finger, a long press operation is performed on the target object, and the second time threshold is greater than or equal to the first time threshold; If it is detected that the first part of the first finger collides with the second part of the second finger twice, and the continuous contact time of the first finger and the second finger during each collision is less than the first time threshold, in response to the collision between the first finger and the second finger, a double-click operation is performed on the target object; The first operation is the confirmation operation, the single-click operation, the long-press operation or the double-click operation.

3. The method according to claim 2, characterized in that If it is detected that the first finger slides on the second finger, in response to the sliding of the first finger on the second finger, performing a second operation includes: Detecting that the first finger slides along the upper edge of the second finger; In response to the sliding direction and displacement of the first finger on the second finger, performing the second operation on the target object; The operation amount of the second operation is related to the displacement amount of the first finger on the second finger, the adjustment direction of the second operation is related to the sliding direction of the first finger on the second finger, and the adjustment direction includes increase and decrease.

4. The method according to claim 3, characterized in that If the sliding direction of the first finger on the second finger is the first direction, the adjustment direction of the second operation is to increase; if the sliding direction of the first finger on the second finger is the second direction, the adjustment direction of the second operation is to decrease; The first direction is a direction from the wrist of the second hand to the fingertip of the second finger, and the second direction is a direction from the fingertip of the second finger to the wrist of the second hand.

5. The method according to claim 3, characterized in that: The target object is a volume control, a brightness control, a window or a progress bar control, and the second operation is: increasing or decreasing the volume, increasing or decreasing the brightness, increasing or decreasing the window size, or increasing or decreasing the playback progress.

6. The method according to claim 1, characterized in that If it is detected that the first finger slides on the second finger, in response to the sliding of the first finger on the second finger, performing a second operation includes: If it is detected that the first finger slides on the second finger along a first direction, in response to the sliding of the first finger on the second finger, the page is slid in the first sliding direction, where the first direction is a direction from the wrist of the second hand to the fingertip of the second finger; If it is detected that the first finger slides on the second finger along a second direction, in response to the sliding of the first finger on the second finger, the page is slid in a second sliding direction, the second direction is a direction from the fingertip of the second finger to the wrist of the second hand, and the first sliding direction and the second sliding direction are different; The second operation is a page sliding operation, and the sliding amount of the page is related to the displacement amount of the first finger on the second finger.

7. The method according to claim 1, characterized in that The head mounted display device comprises an optical display module, wherein the optical display module comprises a lens and a display screen; The method further comprises: In response to one or more operations, entering a diopter adjustment mode; If it is detected that the first finger slides on the second finger, in response to the sliding of the first finger on the second finger, performing a second operation includes: In the diopter adjustment mode, detecting that the first finger slides on the second finger; In response to the sliding of the first finger on the second finger, controlling the movable component in the optical display module to move; Among them, the movement amount of the movable component is related to the displacement amount of the first finger on the second finger, and the movement direction of the movable component is related to the sliding direction of the first finger on the second finger; the movable component is the lens or the display screen; and the second operation is to control the movement of the movable component.

8. The method according to claim 7, characterized in that If the sliding direction of the first finger on the second finger is the first direction, the moving direction is the direction from the movable component to the human eye; if the sliding direction of the first finger on the second finger is the second direction, the moving direction is the direction from the human eye to the movable component.

9. The method according to any one of claims 1 to 8, characterized in that: The first finger is a thumb or an index finger, the second finger is a thumb or an index finger, the first part is a fingertip, and the second part is a fingertip or a back of a finger.

10. A head mounted display device, characterized in that: Includes processing module, optical display module and gesture tracking module; The optical display module is used to display a view of a three-dimensional environment; The gesture tracking module is used to recognize user gestures; The processing module is used to: display the view of the three-dimensional environment through the optical display module; If the gesture tracking module detects that the first part of the first finger of the user's first hand collides with the second part of the second finger of the second hand, a first operation is performed in response to the collision of the first finger and the second finger, and the first hand and the second hand are different; and / or if the gesture tracking module detects that the first finger slides on the second finger, a second operation is performed in response to the sliding of the first finger on the second finger.

11. The device according to claim 10, characterized in that The head mounted display device further comprises an eye tracking module and / or a head aiming module, and the view of the three-dimensional environment comprises at least one interactive object; the processing module is further used for: displaying the target object in a target display mode in response to one or more operations through the eye tracking module and / or the head aiming module, wherein the one or more operations are operations of selecting the target object from the at least one interactive object, and the target display mode is different from a display mode of non-target objects; The processing module is specifically used for: If it is detected by the gesture tracking module that the first part of the first finger and the second part of the second finger collide once, and the continuous contact time of the first finger and the second finger when they collide is less than a first time threshold, then in response to the collision of the first finger and the second finger, a confirmation operation is performed or a single-click operation is performed on the target object; If it is detected by the gesture tracking module that the first part of the first finger and the second part of the second finger collide once, and the continuous contact time of the first finger and the second finger during the collision is greater than the second time threshold, then in response to the collision of the first finger and the second finger, a long press operation is performed on the target object, and the second time threshold is greater than or equal to the first time threshold; If the gesture tracking module detects that the first part of the first finger collides with the second part of the second finger twice, and the continuous contact time between the first finger and the second finger each time is less than the first time threshold, then in response to the collision between the first finger and the second finger, a double-click operation is performed on the target object; the first operation is the confirmation operation, the single-click operation, the long press operation or the double-click operation.

12. The device according to claim 11, characterized in that The processing module is specifically used for: Detecting, by the gesture tracking module, that the first finger slides along the upper edge of the second finger; In response to the sliding direction and displacement of the first finger on the second finger, performing the second operation on the target object; The operation amount of the second operation is related to the displacement of the first finger on the second finger, and the adjustment direction of the second operation is related to the sliding direction of the first finger on the second finger, and the adjustment direction includes increase and decrease.

13. The device according to claim 12, characterized in that If the sliding direction of the first finger on the second finger is the first direction, the adjustment direction of the second operation is to increase; if the sliding direction of the first finger on the second finger is the second direction, the adjustment direction of the second operation is to decrease; The first direction is a direction from the wrist of the second hand to the fingertip of the second finger, and the second direction is a direction from the fingertip of the second finger to the wrist of the second hand.

14. The device according to claim 12, characterized in that The target object is a volume control, a brightness control, a window or a progress bar control, and the second operation is: increasing or decreasing the volume, increasing or decreasing the brightness, increasing or decreasing the window size, or increasing or decreasing the playback progress.

15. The device according to claim 10, characterized in that The processing module is specifically used for: If the gesture tracking module detects that the first finger slides on the second finger along a first direction, in response to the sliding of the first finger on the second finger, the page is slid in the first sliding direction, where the first direction is a direction from the wrist of the second hand to the fingertip of the second finger; If the gesture tracking module detects that the first finger slides on the second finger along a second direction, in response to the sliding of the first finger on the second finger, the page is slid in a second sliding direction, the second direction being a direction from the fingertip of the second finger to the wrist of the second hand, and the first sliding direction and the second sliding direction are different; The second operation is a page sliding operation, and the sliding amount of the page is related to the displacement amount of the first finger on the second finger.

16. The device according to claim 10, characterized in that The optical display module includes a lens and a display screen; The processing module is also used for: In response to one or more operations, entering a diopter adjustment mode; The processing module is specifically used for: In the diopter adjustment mode, the gesture tracking module detects that the first finger slides on the second finger; In response to the sliding of the first finger on the second finger, controlling the movable component in the optical display module to move; Among them, the movement amount of the movable component is related to the displacement amount of the first finger on the second finger, and the movement direction of the movable component is related to the sliding direction of the first finger on the second finger; the movable component is the lens or the display screen; and the second operation is to control the movement of the movable component.

17. The device according to claim 16, characterized in that If the sliding direction of the first finger on the second finger is the first direction, the moving direction is the direction from the movable component to the human eye; if the sliding direction of the first finger on the second finger is the second direction, the moving direction is the direction from the human eye to the movable component.

18. The device according to any one of claims 10 to 17, characterized in that The first finger is a thumb or an index finger, the second finger is a thumb or an index finger, the first part is a fingertip, and the second part is a fingertip or a back of a finger.

19. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 9 is implemented.

Images