Two-handed gesture recognition method, interactive interface display method, and head-mounted display device

Abstract

This disclosure presents a two-handed gesture recognition method, an interactive interface display method, and a head-mounted display device. One specific implementation of the two-handed gesture recognition method includes: acquiring a video stream using a monocular camera; detecting two-handed gestures in the video stream to obtain two-handed gesture position information; generating two-handed gesture key point information; determining camera pose information; and determining predicted two-handed gesture position information. Another specific implementation of the interactive interface display method includes: determining the gesture key point information; determining whether the gesture key point information indicates that the gesture is in a tap-activated gesture state; in response to determining that the gesture key point information indicates that the gesture is in a tap-activated gesture state, determining the gesture dwell time; and in response to the gesture dwell time meeting a preset gesture dwell time condition, displaying the interactive interface corresponding to the gesture key point information. This implementation reduces the size and power consumption of the camera mounted on the head-mounted display device, improving the user experience.

Description

Technical Field

[0001] The embodiments disclosed herein relate to the field of computer technology, specifically to a two-hand gesture recognition method, an interactive interface display method, and a head-mounted display device. Background Technology

[0002] With the development of augmented reality technology, mounting cameras on head-mounted display devices (such as smart glasses) for human-computer interaction has become a new mainstream approach. Currently, existing solutions for human-computer interaction using a monocular camera determine whether the user has performed a click operation based on the interaction between the screen plane and the user's pinch gesture. In addition, there are also methods that utilize binocular cameras for human-computer interaction.

[0003] However, when using the above methods for human-computer interaction, the following technical problems often arise:

[0004] Binocular cameras are large and consume a lot of power; in addition, monocular cameras require users to keep their arms raised for a long time due to the interaction based on the screen plane, which causes arm fatigue and results in a poor user experience. Summary of the Invention

[0005] The summary portion of this disclosure is intended to provide a brief overview of the concepts, which will be described in detail in the detailed description portion. This summary portion is not intended to identify key or essential features of the claimed technical solutions, nor is it intended to limit the scope of the claimed technical solutions.

[0006] Some embodiments of this disclosure provide a two-hand gesture recognition method, an interactive interface display method, and a head-mounted display device to solve one or more of the technical problems mentioned in the background section above.

[0007] In a first aspect, some embodiments of this disclosure provide a method for recognizing two-handed gestures. The method includes: acquiring a video stream through the monocular camera; performing two-handed gesture detection on the video stream to obtain two-handed gesture position information; generating two-handed gesture key point information based on the two-handed gesture position information; determining camera pose information based on the video stream and inertial measurement unit information; and determining predicted two-handed gesture position information based on the camera pose information and the video stream.

[0008] Optionally, the above-mentioned two-hand gesture detection of the video stream to obtain two-hand gesture position information includes: performing two-hand gesture detection on the video stream using a two-hand gesture detection algorithm to obtain two-hand gesture bounding box position information and two-hand gesture initial coordinate information; and determining the two-hand gesture bounding box position information and the two-hand gesture initial coordinate information as the two-hand gesture position information.

[0009] Optionally, before determining the bounding box position information and the initial coordinate information of the two-hand gestures as the position information of the two-hand gestures, the method further includes: aligning the initial coordinate information of the two-hand gestures to obtain aligned initial coordinate information of the two-hand gestures; and performing registration processing on the aligned initial coordinate information of the two-hand gestures to update the initial coordinate information of the two-hand gestures.

[0010] Optionally, the above two-hand gesture recognition method further includes: in response to the fact that the last video frame included in the video stream does not contain gesture information, determining the relative position information of the predicted gesture position information with respect to the last video frame; and prompting the user to move the monocular camera based on the relative position information.

[0011] Secondly, some embodiments of this disclosure provide an interactive interface display method, the method comprising: determining gesture key point information using the method described in any implementation of the first aspect above; determining whether the gesture key point information indicates that the gesture is in a tap-activated gesture state; in response to determining that the gesture key point information indicates that the gesture is in a tap-activated gesture state, determining a gesture dwell time based on the gesture key point information; and in response to the gesture dwell time satisfying a preset gesture dwell time condition, displaying the interactive interface corresponding to the gesture key point information.

[0012] Optionally, the above interactive interface display method further includes: closing the above interactive interface in response to detecting that the above gesture key point information indicates that the gesture is in a clenched fist state.

[0013] Optionally, the above-mentioned interactive interface display method further includes: in response to detecting that the gesture key point information indicates that the gesture is in a wrist-raised state, displaying the interactive interface at the wrist corresponding to the gesture key point information in a preset wrist display mode.

[0014] Optionally, the above-mentioned interactive interface display method further includes: moving the interactive interface in response to detecting that the gesture key point information indicates that the gesture is in a preset mobile interface gesture state.

[0015] Optionally, the above interactive interface display method further includes: in response to detecting that the above gesture key point information indicates that the gesture is in a preset mobile application gesture state, moving the application within the above interactive interface.

[0016] Thirdly, some embodiments of this disclosure provide a head-mounted display device, including: a monocular camera for acquiring video streams; one or more processors; and a storage device storing one or more programs thereon, which, when executed by one or more processors, cause the one or more processors to implement the method described in any implementation of the first aspect above.

[0017] The various embodiments disclosed above have the following beneficial effects: the two-hand gesture recognition method of some embodiments of this disclosure reduces the size and power consumption of the camera mounted on the head-mounted display device, and improves the user experience. Specifically, the reason for the large size and power consumption of the camera mounted on the head-mounted display device and the poor user experience is that: binocular cameras are large and consume a lot of power; in addition, the interaction of a monocular camera based on the screen plane requires the user's arm to be raised continuously for a long time, causing arm fatigue and a poor user experience. Based on this, the two-hand gesture recognition method of some embodiments of this disclosure first acquires a video stream through a monocular camera. Then, two-hand gesture detection is performed on the video stream to obtain two-hand gesture position information. Thus, two-hand gesture position information representing the position of the user's hands in each video frame included in the video stream can be obtained. Then, two-hand gesture key point information is generated based on the two-hand gesture position information. Thus, two-hand gesture key point information representing the coordinates of the joint points of the two palms can be obtained. Afterwards, the camera pose information is determined based on the video stream and inertial measurement unit information. Thus, the camera pose information representing the position of the monocular camera can be obtained. Finally, the predicted two-hand gesture position information is determined based on the camera pose information and the video stream. Therefore, the predicted hand gesture position information, representing the predicted position of both hands relative to the monocular camera in the next video frame, can be obtained. Because human-computer interaction is achieved by using a monocular camera, and by obtaining the predicted hand gesture position information, the loss of gesture or hand tracking caused by directly replacing a binocular camera with a monocular camera can be reduced. This reduces the size of the camera while maintaining a similar human-computer interaction effect for the user, thereby improving the user experience. Attached Figure Description

[0018] The above and other features, advantages, and aspects of the embodiments of this disclosure will become more apparent from the accompanying drawings and the following detailed description. Throughout the drawings, the same or similar reference numerals denote the same or similar elements. It should be understood that the drawings are schematic, and elements are not necessarily drawn to scale.

[0019] Figure 1 This is a flowchart of some embodiments of the two-handed gesture recognition method according to the present disclosure;

[0020] Figure 2 This is a flowchart of some embodiments of the interactive interface display method according to the present disclosure;

[0021] Figure 3 This is a schematic diagram of a head-mounted display device suitable for implementing some embodiments of the present disclosure. Detailed Implementation

[0022] Embodiments of this disclosure will now be described in more detail with reference to the accompanying drawings. While some embodiments of this disclosure are shown in the drawings, it should be understood that this disclosure can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this disclosure. It should be understood that the accompanying drawings and embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of protection of this disclosure.

[0023] It should also be noted that, for ease of description, only the parts relevant to the invention are shown in the accompanying drawings. Unless otherwise specified, the embodiments and features described in this disclosure can be combined with each other.

[0024] It should be noted that the concepts of "first" and "second" mentioned in this disclosure are used only to distinguish different devices, modules or units, and are not used to limit the order of functions performed by these devices, modules or units or their interdependencies.

[0025] It should be noted that the terms "a" and "a plurality of" used in this disclosure are illustrative rather than restrictive, and those skilled in the art should understand that, unless otherwise expressly indicated in the context, they should be understood as "one or more".

[0026] The names of messages or information exchanged between multiple devices in the embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of such messages or information.

[0027] This disclosure will now be described in detail with reference to the accompanying drawings and embodiments.

[0028] Figure 1 A flow 100 of some embodiments of a two-handed gesture recognition method according to the present disclosure is shown. The two-handed gesture recognition method includes the following steps:

[0029] Step 101: Acquire the video stream using a monocular camera.

[0030] In some embodiments, the entity executing the two-hand gesture recognition method (e.g., a head-mounted display device) can acquire a video stream via a monocular camera. The head-mounted display device may include the monocular camera. In practice, the entity can acquire the video stream using a monocular camera that is in an active state.

[0031] Step 102: Detect two hand gestures in the video stream to obtain the position information of the two hand gestures.

[0032] In some embodiments, the execution entity may perform two-hand gesture detection on the video stream to obtain two-hand gesture position information. This two-hand gesture position information can characterize the position of the user's hands within each video frame of the video stream. For example, the two-hand gesture position information may include the coordinates of the four vertices of the minimum bounding rectangle corresponding to both hands and the initial coordinates of the palms. Here, the initial coordinates of the palms can be the coordinates of the palm's center. Alternatively, the initial coordinates of the palm's base can be the coordinates of the base of the palm. For example, the two-hand gesture position information could be: [First video frame: vertex coordinates (12, 25), vertex coordinates (30, 25), vertex coordinates (12, 60), vertex coordinates (30, 60), initial palm coordinates (16, 33), initial palm coordinates (22, 34)], [Second video frame: vertex coordinates (12, 25), vertex coordinates (30, 25), vertex coordinates (12, 60), vertex coordinates (30, 60), initial palm coordinates (16, 33), initial palm coordinates (22, 34)]. In practice, the aforementioned execution entity can use a dynamic filtering algorithm (such as the Kalman filter) to detect hand gestures in the video stream and obtain hand gesture position information. This allows for the acquisition of hand gesture position information that characterizes the positions of the user's hands within each video frame of the video stream.

[0033] In some optional implementations of certain embodiments, firstly, the execution entity can perform two-handed gesture detection on the video stream using a two-handed gesture detection algorithm to obtain the bounding box position information and initial coordinate information of the two-handed gestures. The two-handed gesture detection algorithm can be a Hidden Markov Model (HMM) method. The bounding box position information can represent the coordinates of the minimum bounding rectangles of each hand in each video frame included in the video stream. For example, for each video frame included in the video stream, the bounding box position information can include the four vertices of the minimum bounding rectangle of the left hand and the four vertices of the minimum bounding rectangle of the right hand in that video frame. The initial coordinate information of the two-handed gestures can represent the initial coordinates of the two palms. Then, the bounding box position information and the initial coordinate information of the two-handed gestures can be determined as the two-handed gesture position information. Therefore, since the two-handed gesture detection detects both palms simultaneously, the accuracy of detecting whether a video frame contains gesture information (information representing the presence of the user's palm within the video frame) is improved for the same video frame. This allows for continuous tracking of the user's palms in consecutive video frames, reducing the probability of losing track of the palms.

[0034] Optionally, before determining the bounding box position information and the initial coordinate information of the two-hand gestures as the position information of the two-hand gestures, the executing entity can first perform alignment processing on the initial coordinate information of the two-hand gestures to obtain aligned initial coordinate information of the two-hand gestures. In practice, the executing entity can perform alignment processing on the initial coordinates of each hand included in the initial coordinate information of the two-hand gestures to obtain aligned initial coordinate information of the two-hand gestures. This ensures that the initial coordinates of each hand are located in the same coordinate system. Then, registration processing can be performed on the aligned initial coordinate information of the two-hand gestures to update the initial coordinate information of the two-hand gestures. This can improve the accuracy of the coordinates represented by the initial coordinate information of the two-hand gestures.

[0035] Step 103: Generate key point information of the two hand gestures based on the position information of the two hand gestures.

[0036] In some embodiments, the execution entity can generate key point information for the two-hand gestures based on the aforementioned hand gesture position information. This key point information can be information representing the coordinates of the joints of the two hands. For example, it can be information representing the coordinates of 42 joints of the two hands. In practice, the execution entity can input the video stream and the hand gesture position information into a Landmark model to obtain the key point information. Thus, key point information representing the coordinates of the joints of the two hands can be obtained. Step 104: Determine the camera pose information based on the video stream and inertial measurement unit information.

[0037] In some embodiments, the execution entity can determine the camera pose information based on the video stream and the inertial measurement unit (IMU) information. The IMU information can be information measured by an IMU sensor included in the head-mounted display device. In practice, VIO (Visual-Inertial Odometry) technology can be used to determine the camera pose information based on the video stream and the IMU information. Thus, camera pose information characterizing the position of the monocular camera can be obtained.

[0038] Step 105: Determine the predicted hand gesture position information based on the camera pose information and video stream.

[0039] In some embodiments, the execution entity can determine the predicted hand gesture position information based on the camera pose information and the video stream. The hand gesture position information represents the predicted position of the hands relative to the monocular camera in the next video frame. The next video frame can be the frame following the last frame in the video stream. In practice, the execution entity can input the camera pose information and the video stream into a pre-trained hand gesture position prediction model to obtain the predicted hand gesture position information. The hand gesture position prediction model can be a neural network model that takes the camera pose information and the video stream as input and outputs the predicted hand gesture position information. For example, the neural network model can be a CNN (Convolutional Neural Network) model. The hand gesture position prediction model can include a low-pass convolutional filter. This low-pass convolutional filter can correspond to low-pass convolutional filter coefficients. Specifically, firstly, the video stream can be input into the hand gesture position prediction model to update the low-pass convolutional filter coefficients (the update of the low-pass convolutional filter coefficients has been pre-trained). Then, the camera pose information can be input into the hand gesture position prediction model updated with low-pass convolutional filter coefficients to obtain the predicted hand gesture position information. Thus, the predicted hand gesture position information, representing the predicted position of the hands relative to the monocular camera in the next video frame, can be obtained.

[0040] Optionally, firstly, in response to the fact that the last video frame included in the video stream does not contain gesture information, the executing entity can determine the relative position information of the predicted gesture position information with respect to the last video frame. The gesture information can be information representing the presence of a user's palm within the video frame. The relative position information can represent the orientation of the position represented by the predicted gesture position information relative to the position represented by the last video frame. For example, the relative position information could be: upper left. In practice, since the orientation of the predicted gesture position information relative to the last video frame is the same as the orientation of the predicted gesture position information relative to the monocular camera, the orientation of the predicted gesture position information relative to the monocular camera can be determined as the relative position information. Then, based on the relative position information, the user can be prompted to move the monocular camera. As an example, if the relative position information represents the predicted gesture position information being in the upper left of the last video frame, the executing entity can prompt the user to move the monocular camera to the upper left. Thus, when the palm leaves the frame, the user can be prompted to adjust the position of the monocular camera in time so that the palm re-enters the frame. Furthermore, using inertial measurement unit information can further reduce the probability of losing hand tracking.

[0041] Optionally, the executing entity can generate gesture depth information from the video stream acquired by the monocular camera. This gesture depth information can characterize the distance between the hand in the video stream and the monocular camera. For example, the gesture depth information can characterize the distance between the hand in the video stream and the monocular camera as: [first video frame: 1 unit depth], [second video frame: 1 unit depth]. Here, the unit depth can be preset by relevant personnel. Alternatively, the unit depth can be the distance between the hand and the monocular camera in the first video frame. In practice, firstly, the executing entity can determine gesture scale information based on the video frames included in the video stream. This gesture scale information can characterize the size of the hand relative to each video frame included in the video stream. For example, the gesture scale information can include: [first video frame: hand occupies 100 pixels], [second video frame: hand occupies 100 pixels]. Here, the specific method for determining the gesture scale information is not limited. For example, an object detection algorithm (such as a convolutional neural network algorithm) can be used to determine the gesture scale information from the video frames included in the video stream. Then, gesture depth information can be determined based on the gesture scale information mentioned above. For example, gesture depth information can be determined using a scale-depth lookup table. This table represents a one-to-one correspondence between gesture scale and gesture depth. For example, the table could include: 100 pixels - one unit depth, 200 pixels - two units depth. Furthermore, when the unit depth is the distance between the hand and the monocular camera in the first video frame, the gesture depth can be determined by the multiple of the hand pixels in each video frame relative to the first video frame. The hand pixel multiple is the quotient of the number of hand pixels in each video frame and the number of hand pixels in the first video frame. As an example, in the first video frame, the hand occupies 100 pixels, and in the second video frame, the hand occupies 100 pixels. Therefore, the number of hand pixels in the second video frame is 1 compared to the first video frame, and the gesture depth corresponding to the second video frame is 1 unit depth. The gesture depth information at this time is: [first video frame: 1 unit depth], [second video frame: 1 unit depth].

[0042] Therefore, existing solutions for human-computer interaction using monocular cameras often struggle to acquire gesture depth information, leading to difficult interactive operations and a poor user experience. The aforementioned invention, as an embodiment of this disclosure, generates gesture depth information from the video stream acquired by a monocular camera. This allows for the determination of the direction and distance of virtual object movement in various user interaction scenarios via head-mounted display devices, such as grasping and dragging virtual objects. Consequently, more precise interactive operations can be achieved using head-mounted display devices including monocular cameras, improving the user experience.

[0043] The various embodiments disclosed above have the following beneficial effects: the two-hand gesture recognition method of some embodiments of this disclosure reduces the size and power consumption of the camera mounted on the head-mounted display device, and improves the user experience. Specifically, the reason for the large size and power consumption of the camera mounted on the head-mounted display device and the poor user experience is that: binocular cameras are large and consume a lot of power; in addition, the interaction of a monocular camera based on the screen plane requires the user's arm to be raised continuously for a long time, causing arm fatigue and a poor user experience. Based on this, the two-hand gesture recognition method of some embodiments of this disclosure first acquires a video stream through a monocular camera. Then, two-hand gesture detection is performed on the video stream to obtain two-hand gesture position information. Thus, two-hand gesture position information representing the position of the user's hands in each video frame included in the video stream can be obtained. Then, two-hand gesture key point information is generated based on the two-hand gesture position information. Thus, two-hand gesture key point information representing the coordinates of the joint points of the two palms can be obtained. Afterwards, the camera pose information is determined based on the video stream and inertial measurement unit information. Thus, the camera pose information representing the position of the monocular camera can be obtained. Finally, the predicted two-hand gesture position information is determined based on the camera pose information and the video stream. Therefore, the predicted hand gesture position information, representing the predicted position of both hands relative to the monocular camera in the next video frame, can be obtained. Because human-computer interaction is achieved by using a monocular camera, and by obtaining the predicted hand gesture position information, the loss of gesture or hand tracking caused by directly replacing a binocular camera with a monocular camera can be reduced. This reduces the size of the camera while maintaining a similar human-computer interaction effect for the user, thereby improving the user experience.

[0044] The following is for reference. Figure 2 The diagram illustrates a flow 200 of some embodiments of an interactive interface display method according to the present disclosure. This interactive interface display method includes the following steps:

[0045] Step 201: Determine the key information of the gesture.

[0046] In some embodiments, the specific implementation of step 201 and its resulting technical effects can be found in [reference needed]. Figure 1 Steps 101-103 in the corresponding embodiments will not be repeated here.

[0047] Step 202: Determine whether the key information of the two hand gestures indicates that the gesture is in a tap-activated gesture state.

[0048] In some embodiments, the executing entity of the interactive interface display method (e.g., a head-mounted display device) can determine whether the aforementioned two-hand gesture key point information indicates that the gesture is in a tap-activated gesture state. The aforementioned tap-activated gesture state can be a gesture in a pre-defined hand posture indicating tap activation. For example, the hand posture corresponding to the aforementioned tap-activated gesture state could be a left hand with the palm facing upwards and the fingers naturally spread, a right hand with the back of the hand facing upwards after making a fist, and the right index finger pressed against the center of the left wrist. In practice, whether the gesture is in a tap-activated gesture state can be determined by comparing the relative positional relationship between the 42 joints included in the aforementioned two-hand gesture key point information with the relative positional relationship between the 42 joints included in the preset tap-activated gesture key point information. The aforementioned preset tap-activated gesture key point information can be pre-defined two-hand gesture key point information indicating that the gesture is in a tap-activated gesture state. If they are the same, the gesture is in a tap-activated gesture state. If they are different, the gesture is not in a tap-activated gesture state. Thus, the state of the gesture can be determined.

[0049] Step 203: In response to determining the key point information of the two-hand gestures to indicate that the gestures are in a tap-activated gesture state, determine the gesture dwell time based on the key point information of the two-hand gestures.

[0050] In some embodiments, in response to determining that the key points of the two-hand gesture represent a tap-activated gesture state, the executing entity can determine the gesture dwell time based on the key points of the two-hand gesture. In practice, in response to determining that the key points of the two-hand gesture represent a tap-activated gesture state, the duration for which the key points of the two-hand gesture represent a tap-activated gesture state can be determined as the gesture dwell time. This avoids accidental touches caused by the gesture being accidentally in a tap-activated gesture state.

[0051] Step 204: In response to the gesture dwell time meeting the preset gesture dwell time condition, the interactive interface corresponding to the key point information of the two hand gestures is displayed.

[0052] In some embodiments, the execution entity may display the interactive interface corresponding to the key point information of the two-hand gestures in response to the gesture dwell time meeting a preset gesture dwell time condition. The preset gesture dwell time condition may be that the gesture dwell time is greater than or equal to a preset duration threshold. The interactive interface may be a pre-defined interface triggered when the gesture is in a tap-activated gesture state and the gesture dwell time meets the preset gesture dwell time condition. For example, the interactive interface may be a desktop interface.

[0053] Optionally, in response to detecting that the gesture key information indicates the gesture is in a clenched fist state, the interactive interface is closed. The clenched fist state can be the left hand's gesture being clenched. The specific method for detecting whether the gesture key information indicates the gesture is in a clenched fist state can be found in the implementation of "Step 202, determining whether the hand gesture key information indicates the gesture is in a tap-activated gesture state," and will not be elaborated here. This further enables user interaction with the head-mounted display device.

[0054] Optionally, in response to detecting that the gesture key point information indicates the gesture is in a wrist-raised state, the interactive interface can be displayed at the wrist corresponding to the gesture key point information using a preset wrist display mode. Here, the wrist-raised state can be the state where the user's wrist is already raised. The wrist corresponding to the gesture key point information can be the location of the user's raised wrist. The preset wrist display mode can be a pre-defined method for displaying the interactive interface when the gesture is in a wrist-raised state. As an example, the preset wrist display mode can be a mode in which a circular dial is displayed in the center of the wrist to indicate the time, and application icons are displayed sequentially in a circular pattern around the circular dial. Therefore, the interactive interface can be displayed in a preset wrist display mode when the wrist is raised, thereby enriching the triggering and display methods of the interactive interface.

[0055] Optionally, the above-mentioned response to detecting that the gesture key point information indicates the gesture is in a wrist-raised state, and displaying the interactive interface at the wrist corresponding to the gesture key point information in a preset wrist display mode, may include: responding to detecting that the gesture key point information indicates the gesture is in a wrist-raised state, and responding to detecting gesture depth information, displaying the interactive interface at the wrist corresponding to the gesture key point information in a preset wrist display mode and scale display information corresponding to the gesture depth information. The gesture scale display information can be information about the scale (size) of the interactive interface. For example, a standard display size of the interactive interface can be preset. The corresponding scale display information can be determined by the gesture depth of the last video frame included in the gesture depth information and a preset depth scale coefficient. The preset depth scale coefficient can be a preset ratio coefficient between the gesture depth and the standard display size. As an example, the gesture depth of the last video frame included in the gesture depth information can be 1 unit depth, the preset depth scale coefficient can be 1, and the scale display information can be 1 times the standard display size. As another example, the gesture depth of the last video frame included in the gesture depth information can be 2 units depth, the preset depth scale coefficient can be 1, and the scale display information can be 2 times the standard display size. Therefore, the size of the interactive interface can be adjusted according to the depth information of the gesture. That is, a larger interactive interface can be displayed for the wrist that is closer to the monocular camera, while the interactive interface can be gradually reduced as the wrist moves away from the camera, thereby enhancing the user's sense of immersion and engagement during interaction.

[0056] Optionally, in response to detecting that the aforementioned gesture key point information indicates that the gesture is in a preset mobile interface gesture state, the aforementioned interactive interface is moved. The aforementioned preset mobile interface gesture state can be that the right hand gesture is in a state of spreading all five fingers. The specific method for detecting whether the gesture key point information indicates that the gesture is in a preset mobile interface gesture state can be found in the specific implementation of "Step 202, determining whether the key point information of both hands indicates that the gesture is in a tap-activated gesture state", which will not be repeated here. Thus, the interaction between the user and the head-mounted display device can be further realized.

[0057] Optionally, in response to detecting that the aforementioned gesture key point information indicates that the gesture is in a preset mobile application gesture state, the application within the aforementioned interactive interface is moved. The aforementioned preset mobile application gesture state can be a right-hand gesture with all five fingers pinched together. For a detailed method of detecting whether the gesture key point information indicates that the gesture is in a preset mobile application gesture state, please refer to the specific implementation of "Step 202, determining whether the key point information of both hands indicates that the gesture is in a tap-to-activate gesture state," which will not be elaborated here. This further enables user interaction with the head-mounted display device.

[0058] The above-described embodiments of this disclosure have the following beneficial effects: The interactive interface display method of some embodiments of this disclosure reduces the size and power consumption of the camera mounted on the head-mounted display device, thereby improving the user experience. Specifically, the large size and power consumption of the camera mounted on the head-mounted display device, resulting in a poor user experience, are due to: binocular cameras being large and consuming high power; furthermore, monocular cameras, based on the screen plane, require the user to continuously raise their arm for extended periods, causing arm fatigue and a poor user experience. Therefore, the interactive interface display method of some embodiments of this disclosure first determines the key gesture information. Then, it determines whether the key gesture information of both hands indicates that the gesture is in a tap-activated gesture state. This determines the gesture state. Then, in response to determining that the key gesture information of both hands indicates that the gesture is in a tap-activated gesture state, the gesture dwell time is determined based on the key gesture information of both hands. This avoids accidental touches caused by the gesture being accidentally in a tap-activated gesture state. Finally, in response to the gesture dwell time meeting a preset gesture dwell time condition, the interactive interface corresponding to the key gesture information of both hands is displayed. Because the interactive interface is displayed by activating the gesture state through tapping and setting the preset gesture dwell time, users do not need to raise their arms for a long time during human-computer interaction, reducing arm fatigue and improving user experience.

[0059] The following is for reference. Figure 3 It shows a schematic diagram of the structure of a head-mounted display device 300 suitable for implementing some embodiments of the present disclosure. Figure 3 The head-mounted display device shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments of this disclosure.

[0060] like Figure 3 As shown, the head-mounted display device 300 may include a processing unit (e.g., a central processing unit, a graphics processing unit, etc.) 301, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 302 or a program loaded from a storage device 308 into a random access memory (RAM) 303. The RAM 303 also stores various programs and data required for the operation of the head-mounted display device 300. The processing unit 301, ROM 302, and RAM 303 are interconnected via a bus 304. An input / output (I / O) interface 305 is also connected to the bus 304.

[0061] Typically, the following devices can be connected to the I / O interface 305: input devices 306 including, for example, a touchscreen, touchpad, keyboard, mouse, camera, microphone, accelerometer, gyroscope, etc.; output devices 307 including, for example, a liquid crystal display (LCD), speaker, vibrator, etc.; storage devices 308 including, for example, magnetic tape, hard disk, etc.; and communication devices 309. The communication device 309 allows the head-mounted display device 300 to communicate wirelessly or wiredly with other devices to exchange data. Furthermore, the head-mounted display device 300 may also include a monocular camera 310. The monocular camera 310 is used to acquire video streams. Although... Figure 3 A head-mounted display device 300 with various devices is shown; however, it should be understood that it is not required to implement or have all of the devices shown. More or fewer devices may be implemented or included alternatively. Figure 3 Each box shown can represent a device or multiple devices as needed.

[0062] In particular, according to some embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, some embodiments of this disclosure include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device 309, or installed from a storage device 308, or installed from a ROM 302. When the computer program is executed by the processing device 301, it performs the functions defined in the methods of some embodiments of this disclosure.

[0063] Computer program code for performing operations of some embodiments of this disclosure can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0064] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0065] The functions described above in this document can be performed, at least in part, by one or more hardware logic components. For example, exemplary types of hardware logic components that can be used, without limitation, include: Field Programmable Gate Arrays (FPGAs), Application-Specific Integrated Circuits (ASICs), Application Standard Products (ASSPs), System-on-Chip (SoCs), Complex Programmable Logic Devices (CPLDs), and so on.

[0066] The above description is merely a selection of preferred embodiments of this disclosure and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of the invention involved in the embodiments of this disclosure is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-described inventive concept. For example, technical solutions formed by substituting the above-described features with (but not limited to) technical features with similar functions disclosed in the embodiments of this disclosure.

Claims

1. A two-handed gesture recognition method, applied to a head-mounted display device including a monocular camera, wherein, The method includes: The video stream is acquired using the monocular camera; Perform two-hand gesture detection on the video stream to obtain the position information of the two-hand gestures; Based on the position information of the two hand gestures, generate key point information of the two hand gestures; Based on the video stream and inertial measurement unit information, determine the camera pose information; Based on the camera pose information and the video stream, the predicted hand gesture position information is determined, wherein the hand gesture position information represents the predicted position of the hands relative to the monocular camera in the next video frame.

2. The method according to claim 1, wherein, The step of detecting two hand gestures in the video stream to obtain the position information of the two hand gestures includes: The video stream is subjected to two-handed gesture detection using a two-handed gesture detection algorithm to obtain the position information of the two-handed gesture bounding box and the initial coordinate information of the two-handed gesture; The bounding box position information of the two-hand gesture and the initial coordinate information of the two-hand gesture are determined as the position information of the two-hand gesture.

3. The method according to claim 2, wherein, Before determining the bounding box position information of the two-hand gesture and the initial coordinate information of the two-hand gesture as the position information of the two-hand gesture, the method further includes: The initial coordinate information of the two hand gestures is aligned to obtain the aligned initial coordinate information of the two hand gestures; The initial coordinate information of the two-handed gestures after alignment is registered to update the initial coordinate information of the two-handed gestures.

4. The method according to claim 1, wherein, The method further includes: In response to the fact that the last video frame included in the video stream does not contain gesture information, the relative position information of the predicted hand gesture position information with respect to the last video frame is determined; Based on the relative position information, the user is prompted to move the monocular camera.

5. An interactive interface display method, applied to a head-mounted display device including a monocular camera, wherein, The method includes: Using the method described in any one of claims 1-4, determine the key point information of the two-hand gestures; Determine whether the key point information of the two-hand gesture indicates that the gesture is in a tap-activated gesture state; In response to determining that the key point information of the two hand gestures indicates that the gesture is in a tap-activated gesture state, the gesture dwell time is determined based on the key point information of the two hand gestures. In response to the gesture dwell time meeting the preset gesture dwell time condition, the interactive interface corresponding to the key point information of the two hand gestures is displayed.

6. The method according to claim 5, wherein, The method further includes: In response to detecting that the gesture key point information indicates that the gesture is in a clenched fist state, the interactive interface is closed.

7. The method according to claim 5, wherein, The method further includes: In response to detecting that the gesture key point information indicates that the gesture is in a wrist-raised state, the interactive interface is displayed at the wrist corresponding to the gesture key point information in a preset wrist display mode.

8. The method according to claim 5, wherein, The method further includes: In response to detecting that the gesture key point information indicates that the gesture is in a preset mobile interface gesture state, the interactive interface is moved.

9. The method according to claim 5, wherein, The method further includes: In response to detecting that the gesture key point information indicates that the gesture is in a preset mobile application gesture state, the application within the interactive interface is moved.

10. A head-mounted display device, comprising: A monocular camera is used to acquire video streams; One or more processors; A storage device on which one or more programs are stored; When the one or more programs are executed by the one or more processors, the one or more processors implement the method as described in any one of claims 1-9.

Images