Gesture recognition method, device, head-mounted device, head-mounted system and medium

Abstract

The present application discloses a gesture recognition method, device, head-mounted device, head-mounted system and medium, and relates to the field of head-mounted technology. The method is applied to a head-mounted device, which is adapted to be equipped with at least one handle. The method comprises: obtaining hand images of a hand performing a reference gesture captured by the head-mounted device and any handle at the same time, where the reference gesture is an occlusion gesture; determining a recognition gesture based on the hand images; determining a gesture recognition result based on the recognition gesture and the reference gesture; outputting a prompt message for adjusting the handle position when the gesture recognition result indicates that there is a deviation in the recognition gesture; repeatedly obtaining hand images of the hand performing the reference gesture captured by the head-mounted device and any handle at the same time, until the gesture recognition result indicates that the gesture recognition is accurate. Accurate recognition of occlusion gestures can be achieved through this method.

Description

Technical Field

[0001] The present application relates to the field of head-mounted technology, and more specifically, to a gesture recognition method, a gesture recognition apparatus, a head-mounted device, a head-mounted system, and a computer-readable storage medium. Background Art

[0002] With the development of science and technology and economy, head-mounted devices have been increasingly used. In addition, gesture recognition is one of the important functions of head-mounted devices.

[0003] Currently, headsets are equipped with two cameras to capture the external environment. These cameras capture images of the user's hand and determine the user's gesture. However, if there's an obstruction between the camera and the user's hand, or if one hand blocks the other's gesture, the headset cannot determine the correct gesture based on these two cameras. Summary of the Invention

[0004] One purpose of this application is to provide a new technical solution for gesture recognition.

[0005] According to a first aspect of the present application, a gesture recognition method is provided, which is applied to a head-mounted device, wherein the head-mounted device is adapted to be equipped with at least one handle, and the method comprises:

[0006] Acquire hand images of a hand performing a reference gesture captured by the head mounted device and any of the handles at the same moment, where the reference gesture is an occlusion gesture;

[0007] Determining a recognized gesture according to the hand image;

[0008] Determining a gesture recognition result according to the recognized gesture and a reference gesture;

[0009] If the gesture recognition result indicates that there is a deviation in the recognized gesture, outputting prompt information for adjusting the handle position;

[0010] Furthermore, the step of obtaining the hand images of the hand performing the reference gesture respectively captured by the head mounted device and any one of the handles at the same moment is repeated until the gesture recognition result indicates that the gesture recognition is accurate.

[0011] Optionally, before acquiring hand images of a hand performing a reference gesture captured by the head mounted device and any of the handles at the same moment, the method further includes:

[0012] At least one of an image or a descriptive voice of the reference gesture is output.

[0013] Optionally, the method further includes:

[0014] When the head mounted device runs the target application and receives notification information indicating that there is a deviation in gesture recognition, the acquisition is triggered to simultaneously capture the hand image of the hand performing the reference gesture by the head mounted device and any of the handles.

[0015] Optionally, the method further includes a step of obtaining the notification information, the step including:

[0016] Detect whether the set trigger event occurs;

[0017] If this happens, a notification message is generated indicating that there is a deviation in gesture recognition;

[0018] The set triggering event includes at least one of: failure to recognize a gesture, an abnormal gesture being recognized, and receiving user indication information indicating deviation in gesture recognition.

[0019] Optionally, after acquiring the hand images of the hand performing the reference gesture captured by the head mounted device and any of the handles at the same moment, the method further includes:

[0020] For any of the hand images, remove the background image in the hand image;

[0021] Converting the hand image after removing the background image into a grayscale image;

[0022] The step of determining a recognized gesture based on the hand image includes:

[0023] A recognized gesture is determined according to the grayscale image.

[0024] According to a second aspect of the present application, a gesture recognition device is provided, characterized in that the device is applied to a head-mounted device, the head-mounted device is equipped with at least one handle, and the device includes:

[0025] An acquisition module, configured to acquire hand images of a hand performing a reference gesture captured by the head mounted device and any of the handles at the same moment, where the reference gesture is an occlusion gesture;

[0026] A first determining module, configured to determine a recognized gesture based on the hand image;

[0027] a second determining module, configured to determine a gesture recognition result based on the recognized gesture and a reference gesture;

[0028] a first output module, configured to output a prompt message for adjusting the handle position if the gesture recognition result indicates that the recognized gesture has a deviation;

[0029] A repetition module is used to repeatedly obtain the hand images of the hand performing the reference gesture collected by the head mounted device and any of the handles at the same time, until the gesture recognition result indicates that the gesture recognition is accurate.

[0030] Optionally, the device further comprises:

[0031] The second output module is configured to output at least one of an image of the reference gesture or a description voice.

[0032] According to a third aspect of the present application, a head-mounted device is provided, the head-mounted device comprising the apparatus according to the second aspect; or,

[0033] The system comprises a memory and a processor, wherein the memory is used to store computer instructions, and the processor is used to call the computer instructions from the memory to execute the gesture recognition method as described in any one of the first aspects.

[0034] According to a fourth aspect of the present application, a head-mounted system is provided, comprising the head-mounted device as described in the third aspect and at least one handle.

[0035] According to a fifth aspect of the present application, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the gesture recognition method according to any one of the first aspects is implemented.

[0036] An embodiment of the present application provides a gesture recognition method, which is applied to a head-mounted device, wherein the head-mounted device is adapted to have at least one handle. The method comprises: obtaining hand images of a hand that is performing a reference gesture captured by the head-mounted device and any handle at the same time, wherein the reference gesture is an occlusion gesture; determining a recognition gesture based on the hand images; determining a gesture recognition result based on the recognized gesture and the reference gesture; outputting a prompt message for adjusting the handle position when the gesture recognition result indicates that there is a deviation in the recognized gesture; and repeatedly obtaining hand images of the hand that is performing the reference gesture captured by the head-mounted device and any handle at the same time, until the gesture recognition result indicates that the gesture recognition is accurate. In this method, by assisting the head-mounted device with the handle, the head-mounted device can obtain more accurate gesture recognition without introducing additional hardware costs. Furthermore, the gesture recognition method provided in the embodiment of the present application can achieve accurate recognition of occlusion gestures.

[0037] Other features and advantages of the present application will become apparent from the following detailed description of exemplary embodiments of the present application with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] The accompanying drawings, which are incorporated in and constitute a part of the specification, illustrate embodiments of the application and, together with the description, serve to explain the principles of the application.

[0039] Figure 1 is a block diagram of a hardware configuration of a head-mounted device for implementing a gesture recognition method according to an embodiment of the present application;

[0040] Figure 2 is a flowchart of a method for implementing gesture recognition according to an embodiment of the present application;

[0041] Figure 3 is a schematic diagram of an example of a blocking gesture provided according to an embodiment of the present application;

[0042] Figure 4 This is a structural diagram of a gesture recognition device provided according to an embodiment of the present application;

[0043] Figure 5 This is a structural diagram of a head-mounted device provided according to an embodiment of the present application. DETAILED DESCRIPTION

[0044] Various exemplary embodiments of the present application will now be described in detail with reference to the accompanying drawings. It should be noted that unless otherwise specifically stated, the relative arrangements of components and steps, numerical expressions and numerical values ​​set forth in these embodiments do not limit the scope of the present application.

[0045] The following description of at least one exemplary embodiment is merely illustrative in nature and is in no way intended to limit the present disclosure, its application, or uses.

[0046] Technologies, methods, and equipment known to ordinary technicians in the relevant art may not be discussed in detail, but where appropriate, the technologies, methods, and equipment should be considered part of the specification.

[0047] In all examples shown and discussed herein, any specific values ​​should be interpreted as merely exemplary and not limiting. Therefore, other examples of the exemplary embodiments may have different values.

[0048] It should be noted that like reference numerals and letters refer to like items in the following figures, and therefore, once an item is defined in one figure, it need not be further discussed in subsequent figures.

[0049] Figure 1 This is a block diagram of the hardware configuration of a head-mounted device for implementing a gesture recognition method provided in an embodiment of the present application.

[0050] The head mounted device 1000 may be, for example, an AR device, an MR device, or a VR device. Furthermore, the head mounted device 1000 is adapted to be equipped with at least one handle, and the handle is provided with at least one camera.

[0051] The head-mounted device 1000 may include a processor 1100, a memory 1200, an interface device 1300, a communication device 1400, a display device 1500, an input device 1600, a speaker 1700, a microphone 1800, and an image acquisition device 1900, among others. The processor 1100 may be a central processing unit (CPU), a microprocessor (MCU), or the like. The memory 1200 may include, for example, ROM (read-only memory), RAM (random access memory), or a non-volatile memory such as a hard disk. The interface device 1300 may include, for example, a USB port or a headphone jack. The communication device 1400 may be capable of wired or wireless communication. The display device 1500 may be, for example, an LCD display or a touchscreen display. The input device 1600 may include, for example, a touchscreen or a keyboard. A user may input / output voice information via the speaker 1700 and the microphone 1800. The image acquisition device 1900 may be at least one camera of various types.

[0052] Despite Figure 1 Multiple devices are shown for the head-mounted device 1000, but the present application may only involve some of the devices, for example, the head-mounted device 1000 only involves the memory 1200 and the processor 1100.

[0053] In the embodiment of the present application, the memory 1200 of the head mounted device 1000 is used to store instructions, which are used to control the processor 1100 to execute the gesture recognition method provided in the embodiment of the present application.

[0054] In the above description, a person skilled in the art can design instructions according to the solution disclosed in this application. How instructions control the operation of a processor is well known in the art and will not be described in detail here.

[0055] The embodiment of the present application provides a gesture recognition method, such as Figure 2 As shown, the method includes the following S2100-S2500:

[0056] S2100: Obtain hand images of the hand performing the reference gesture captured by the head mounted device and any handle at the same moment.

[0057] The reference gesture is an occlusion gesture.

[0058] In the embodiment of the present application, the gesture recognition method provided in the embodiment of the present application is applied to the above Figure 1The head-mounted device 1000 shown in FIG. 1 is adapted to be equipped with at least one handle. Both the head-mounted device 1000 and the at least one handle are provided with cameras capable of capturing images. The cameras may be dual cameras or TOF cameras.

[0059] Before executing the gesture recognition method provided in the embodiments of the present application, the user first positions the controller so that the controller can capture hand images of the user performing a reference gesture from different perspectives. In one example, the head-mounted device is adapted to have two controllers, one of which is positioned so that its camera faces directly in front of the user, and the other is positioned so that its camera faces to the side of the user.

[0060] In one embodiment of the present application, in order to make the position of the handle flexible, the handle can be set on the bracket.

[0061] In the embodiment of the present application, the reference gesture is a blocking gesture. The blocking gesture includes: a gesture blocked by an external object, or a gesture in which one hand blocks another hand. In one example, the blocking gesture can be as follows: Figure 3 shown.

[0062] In one embodiment of the present application, the reference gesture can be in the form of a picture, attached to the head-mounted device or any controller, or attached to the instruction manual of the head-mounted device or any controller.

[0063] In another embodiment of the present application, the reference gesture may also be output by a head-mounted device. On this basis, the gesture recognition method provided in the embodiment of the present application further includes the following S2110 before the above S2100:

[0064] S2110: Output at least one of an image or a description voice of the reference gesture.

[0065] In the embodiment of the present application, the above-mentioned method of S2110 can enable the user to intuitively obtain the reference gesture and then execute the reference gesture.

[0066] In one embodiment of the present application, after performing a reference gesture, the user can instruct the head-mounted device, for example, by voice, to capture a hand image of the user's hand performing the reference gesture. Based on this, the head-mounted device can obtain a hand image of the user's hand performing the reference gesture. Furthermore, upon receiving the aforementioned instruction, the head-mounted device sends an instruction to any controller to capture a hand image of the user's hand performing the reference gesture. Based on this, each controller captures an image of the user's hand performing the reference gesture, obtains a hand image, and sends the obtained hand image to the head-mounted device. Based on this, the head-mounted device obtains a hand image of the hand performing the reference gesture captured by any controller.

[0067] It is understandable that any hand image has attribute information corresponding to the shooting time. Based on this, after the head-mounted device obtains the hand image, it selects the hand image at the same time to implement the above S2100.

[0068] S2200: Determine a recognized gesture based on the hand image.

[0069] In the embodiment of the present application, the recognized gesture is a gesture estimated based on the hand image.

[0070] In one embodiment of the present application, the above S2200 may be specifically implemented by performing a 3D reconstruction of the hand gesture based on the hand image and a 3D reconstruction algorithm (e.g., an algorithm for calculating depth of field based on perspective difference), and determining the recognized hand gesture based on the result of the 3D reconstruction.

[0071] Of course, the above S2200 can also be implemented by machine learning. It should be noted that the embodiment of the present application does not limit the specific implementation of the above S2200.

[0072] In the embodiment of the present application, the hand image in S2200 includes hand images collected jointly by the head-mounted device and the controller, that is, the hand image in S2200 includes hand images collected from multiple perspectives. In this way, compared to obtaining gesture recognition based on hand images collected by the head-mounted device, more accurate gesture recognition can be obtained based on S2200. In other words, in the present application, the controller assists the head-mounted device to enable the head-mounted device to obtain more accurate gesture recognition. This does not incur additional hardware costs.

[0073] S2300: Determine a gesture recognition result according to the recognized gesture and the reference gesture.

[0074] In an embodiment of the present application, the reference gesture is pre-stored in the head-mounted device.

[0075] In one embodiment of the present application, the specific implementation of the above S2300 may be: calculating the similarity value between the recognized gesture and the reference gesture according to a similarity algorithm. Based on this, the calculated similarity value may be used as the gesture recognition result.

[0076] S2400: When the gesture recognition result indicates that there is a deviation in the recognized gesture, output a prompt message for adjusting the handle position.

[0077] In one embodiment of the application, when a gesture recognition result is a similarity value between a recognized gesture and a reference gesture, if the similarity value is less than a preset threshold, the gesture recognition result indicates that the head-mounted device has a deviation in recognizing the gesture, that is, the recognized gesture cannot reflect the reference gesture. The preset threshold is the minimum similarity value allowed between two similar gestures.

[0078] Since the position of the head-mounted device usually changes during the use of the head-mounted device, and there are some application scenarios that do not require the use of a handle and require gesture recognition (such as a boxing game scenario), therefore, when the gesture recognition result indicates that there is a deviation in the gesture recognition by the head-mounted device, the position of the handle can be adjusted and then the above step S2100 can be repeated to obtain a recognition result in which there is no deviation in the gesture recognition.

[0079] To remind the user to adjust the handle position, if the gesture recognition results indicate a deviation, a prompt to adjust the handle position is output. For example, a text or voice message such as "Please adjust the handle position" is output. The user then randomly adjusts the handle position. Alternatively, the user requests a recognized gesture, and the handle position is adjusted based on the recognized gesture.

[0080] Alternatively, if the gesture recognition result indicates a deviation in the recognized gesture, the headset may provide a suggestion for adjusting the handle position based on the recognized gesture and output a prompt for adjusting the handle position based on the suggestion. The user may then adjust the handle position based on the suggestion in the prompt.

[0081] In one example, when the gesture recognition is incomplete, the head-mounted device may output a text or voice message of "Please move the handle away."

[0082] On the contrary, when the similarity value is greater than or equal to the preset threshold, it means that the gesture recognition result indicates that the recognized gesture is accurate, that is, the recognized gesture can reflect the reference gesture.

[0083] S2500: Repeatedly obtain hand images of the hand performing the reference gesture collected by the head mounted device and any handle at the same time, until the gesture recognition result indicates that the gesture recognition is accurate.

[0084] In conjunction with S2400 above, after the user adjusts the position of the controller, S2100 above is repeated. At this point, hand images of the hand performing the reference gesture, captured by the headset and either controller at the same moment after the controller adjustment, are obtained. S2200-S2500 above are repeatedly executed until the gesture recognition result indicates accurate gesture recognition.

[0085] If the gesture recognition result indicates accurate gesture recognition, it means that the headset combined with the controller can accurately recognize the user's occlusion gesture. Based on this, it can be understood that the headset combined with the controller can also accurately recognize the user's non-occlusion gesture, which is simpler than the occlusion gesture.

[0086] Based on the above, the head-mounted device can accurately recognize the user's gestures during subsequent use, thereby improving the user experience.

[0087] An embodiment of the present application provides a gesture recognition method, which is applied to a head-mounted device, wherein the head-mounted device is adapted to have at least one handle. The method comprises: obtaining hand images of a hand that is performing a reference gesture captured by the head-mounted device and any handle at the same time, wherein the reference gesture is an occlusion gesture; determining a recognition gesture based on the hand images; determining a gesture recognition result based on the recognized gesture and the reference gesture; outputting a prompt message for adjusting the handle position when the gesture recognition result indicates that there is a deviation in the recognized gesture; and repeatedly obtaining hand images of the hand that is performing the reference gesture captured by the head-mounted device and any handle at the same time, until the gesture recognition result indicates that the gesture recognition is accurate. In this method, by assisting the head-mounted device with the handle, the head-mounted device can obtain more accurate gesture recognition without introducing additional hardware costs. Furthermore, the gesture recognition method provided in the embodiment of the present application can achieve accurate recognition of occlusion gestures.

[0088] In one embodiment of the present application, the gesture recognition method provided in the embodiment of the application further includes the following S2120 and S2121 after the above S2100:

[0089] S2120: For any hand image, remove the background image in the hand image.

[0090] In the embodiment of the present application, invalid image portions in the hand image can be extracted based on the above S2120, thereby reducing the data volume of the hand image.

[0091] S2121. Convert the hand image after removing the background image into a grayscale image.

[0092] In the embodiment of the present application, since the color information in the hand image is useless information for gesture recognition, the hand image after background removal is converted into a grayscale image to further reduce the data volume of the hand image.

[0093] Based on the above S2120 and S2121, the above S2200 is implemented through the following S2210:

[0094] S2210: Determine a recognized gesture based on the grayscale image.

[0095] In the embodiment of the present application, since the data volume of the grayscale image is small, determining the recognized gesture based on the grayscale image can greatly improve the speed of determining the recognized gesture.

[0096] In one embodiment of the present application, after the above S2500, the gesture recognition method provided in the embodiment of the present application further includes the following S2600:

[0097] S2600: When the head-mounted device runs the target application and receives notification information that there is a deviation in gesture recognition, trigger the acquisition of hand images of the hand performing the reference gesture captured by the head-mounted device and any handle at the same time.

[0098] The target application is an application that needs to recognize user gestures, such as a boxing game. When the head-mounted device runs the target application, it means that the head-mounted device has entered normal working mode.

[0099] Furthermore, gesture recognition deviation refers to the inability to accurately recognize the user's current gesture. The most likely cause of gesture recognition deviation is that the handle is accidentally touched, causing its position to change. To solve this problem, the above step S2100 is triggered. Based on this, the above steps S2100-S2500 are repeated to readjust the handle position so that the head-mounted device can accurately recognize the user's gesture.

[0100] Based on the embodiment shown in S2600 above, the gesture recognition method provided in this embodiment of the application further includes the step of obtaining notification information. This step includes the following S2610 and S2611:

[0101] S2610: Detect whether the set trigger event occurs.

[0102] The set triggering event includes one of: failure to recognize a gesture, an abnormal gesture being recognized, and receiving user indication information indicating deviation in gesture recognition.

[0103] Among them, the abnormal gesture is an abnormal gesture that the user cannot perform.

[0104] In the embodiment of the present application, when the head-mounted device runs the target application, it indicates that user gesture recognition is required. In this case, if the gesture is not recognized or the recognized gesture is abnormal, it indicates that there is a deviation in the gesture recognition of the head-mounted device. In this case, the head-mounted device detects the occurrence of the set trigger event. Conversely, the head-mounted device does not detect the occurrence of the set trigger event.

[0105] Also, when the user performs the correct gesture corresponding to the target application, if the head-mounted device frequently prompts the user that it cannot recognize the gesture, or the head-mounted device determines that an error occurs in the operation of the target application (for example, a game fails), etc., then the user determines that there is a deviation in the gesture recognition of the head-mounted device. When the user determines that there is a deviation in gesture recognition, the user can indicate to the head-mounted device that there is a deviation in gesture recognition by, for example, voice notification. Based on this, the head-mounted device receives the user's indication information that there is a deviation in gesture recognition. Further, the head-mounted device detects the occurrence of a set trigger event. Contrary to the above, the head-mounted device does not detect the occurrence of the set trigger event.

[0106] S2611. If this occurs, generate notification information indicating that there is a deviation in gesture recognition.

[0107] In an embodiment of the present application, when notification information indicating that there is a deviation in gesture recognition is generated, the head-mounted device receives the notification information indicating that there is a deviation in gesture recognition.

[0108] The embodiment of the present application further provides a gesture recognition device 400, which is applied to a head-mounted device. The head-mounted device is equipped with at least one handle, such as Figure 4 As shown, the apparatus 400 includes:

[0109] An acquisition module 410 is configured to acquire hand images of a hand performing a reference gesture captured by the head mounted device and any of the handles at the same moment, where the reference gesture is an occlusion gesture;

[0110] A first determining module 420 is configured to determine a recognized gesture based on the hand image;

[0111] A second determination module 430 is configured to determine a gesture recognition result based on the recognized gesture and a reference gesture;

[0112] A first output module 440 is configured to output a prompt message for adjusting the handle position if the gesture recognition result indicates that the recognized gesture has a deviation;

[0113] The repetition module 450 is configured to repeatedly obtain the hand images of the hand performing the reference gesture captured by the head mounted device and any of the handles at the same time, until the gesture recognition result indicates that the gesture recognition is accurate.

[0114] In this device, the handle assists the head-mounted device, allowing the head-mounted device to more accurately recognize gestures without introducing additional hardware costs. Furthermore, the gesture recognition device provided in the embodiment of the application can achieve accurate recognition of occlusion gestures.

[0115] In one embodiment of the present application, the gesture recognition device 400 provided in the embodiment of the present application further includes:

[0116] The second output module is configured to output at least one of an image of the reference gesture or a description voice.

[0117] In one embodiment of the present application, the gesture recognition device 400 provided in the embodiment of the present application further includes:

[0118] The trigger module is used to trigger the acquisition of hand images of the hand performing the reference gesture captured by the head-mounted device and any of the handles at the same time when the head-mounted device runs the target application and receives notification information that there is a deviation in gesture recognition.

[0119] In one embodiment of the present application, the gesture recognition device 400 provided in the embodiment of the present application further includes:

[0120] A detection module is used to detect whether a set trigger event occurs;

[0121] A generation module, used to generate a notification message of a deviation in gesture recognition when such deviation occurs;

[0122] The set triggering event includes at least one of: failure to recognize a gesture, an abnormal gesture being recognized, and receiving user indication information indicating deviation in gesture recognition.

[0123] In one embodiment of the present application, the gesture recognition device 400 provided by the present application embodiment further includes:

[0124] An image processing module, configured to remove a background image from any of the hand images;

[0125] and, converting the hand image after removing the background image into a grayscale image;

[0126] In the embodiment of the present application, the first determining module 420 is specifically configured to:

[0127] A recognized gesture is determined according to the grayscale image.

[0128] The embodiment of the present application further provides a head mounted device 500, wherein the electronic device includes any one of the devices provided in the above device embodiments; or

[0129] like Figure 5 As shown, it includes a memory 510 and a processor 520, the memory 510 is used to store computer instructions, and the processor 520 is used to call the computer instructions from the memory 510 to execute the gesture recognition method as described in any one of the above method embodiments.

[0130] An embodiment of the present application also provides a head-mounted system, which includes any head-mounted device 500 provided in the above-mentioned device embodiments and at least one handle.

[0131] The present application also provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the gesture recognition method according to any one of the above method embodiments.

[0132] The present application may be a system, method and / or computer program product. The computer program product may include a computer-readable storage medium carrying computer-readable program instructions for causing a processor to implement various aspects of the present application.

[0133] A computer-readable storage medium can be a tangible device that can hold and store instructions for use by an instruction execution device. A computer-readable storage medium can be, for example, but not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination thereof. More specific examples (a non-exhaustive list) of computer-readable storage media include: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disc read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanical encoding device, such as a punch card or a raised structure in a groove on which instructions are stored, and any suitable combination thereof. As used herein, a computer-readable storage medium is not to be construed as a transient signal per se, such as a radio wave or other freely propagating electromagnetic wave, an electromagnetic wave propagating through a waveguide or other transmission medium (e.g., a light pulse through a fiber optic cable), or an electrical signal transmitted through an electrical wire.

[0134] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to each computing / processing device, or downloaded to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network can include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. The network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions to be stored in the computer-readable storage medium in each computing / processing device.

[0135] The computer program instructions for performing the operation of the present application can be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-related instructions, microcode, firmware instructions, state setting data or source code or object code written in any combination of one or more programming languages, wherein the programming language includes object-oriented programming languages ​​such as Smalltalk, C++, and conventional procedural programming languages ​​such as "C" language or similar programming languages. Computer-readable program instructions can be executed completely on the user's computer, partially on the user's computer, executed as an independent software package, partially on the user's computer and partially on a remote computer, or executed completely on a remote computer or server. In the case of a remote computer, the remote computer can be connected to the user's computer by any type of network including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computer (such as by using an Internet service provider to connect to the Internet). In certain embodiments, by utilizing the state information of computer-readable program instructions to personalize electronic circuits, such as programmable logic circuits, field programmable gate arrays (FPGAs) or programmable logic arrays (PLAs), the electronic circuits can execute computer-readable program instructions, thereby realizing various aspects of the present application.

[0136] Various aspects of the present application are described herein with reference to flowcharts and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the present application. It should be understood that each block of the flowcharts and / or block diagrams, and combinations of blocks in the flowcharts and / or block diagrams, can be implemented by computer-readable program instructions.

[0137] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, thereby producing a machine, so that when these instructions are executed by the processor of the computer or other programmable data processing device, a device is generated that implements the functions / actions specified in one or more blocks in the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium, where these instructions cause the computer, programmable data processing device, and / or other device to operate in a specific manner. Thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing various aspects of the functions / actions specified in one or more blocks in the flowchart and / or block diagram.

[0138] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device so that a series of operational steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to implement the functions / actions specified in one or more blocks in the flowchart and / or block diagram.

[0139] The flowcharts and block diagrams in the accompanying drawings show the possible architecture, functions and operations of the systems, methods and computer program products according to multiple embodiments of the present application. In this regard, each box in the flowchart or block diagram can represent a part of a module, program segment or instruction, and the part of the module, program segment or instruction contains one or more executable instructions for realizing the specified logical function. In some alternative implementations, the functions marked in the box can also occur in an order different from that marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of the boxes in the block diagram and / or flowchart, can be implemented by a dedicated hardware-based system that performs the specified function or action, or can be implemented by a combination of dedicated hardware and computer instructions. It is well known to those skilled in the art that implementation by hardware, implementation by software, and implementation by a combination of software and hardware are all equivalent.

[0140] The embodiments of the present application have been described above. The above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terms used herein are selected to best explain the principles of the embodiments, practical applications, or technical improvements to technologies in the market, or to enable other persons skilled in the art to understand the embodiments disclosed herein. The scope of this application is defined by the appended claims.

Claims

1. A gesture recognition method, characterized in that: Applied to a head-mounted device, the head-mounted device is adapted to be equipped with at least one handle, and the method includes: Acquire hand images of a hand performing a reference gesture captured by the head mounted device and any of the handles at the same moment, where the reference gesture is an occlusion gesture; Determining a recognized gesture based on the hand image; Determining a gesture recognition result according to the recognized gesture and a reference gesture; If the gesture recognition result indicates that there is a deviation in the recognized gesture, outputting prompt information for adjusting the handle position; Furthermore, the step of obtaining the hand images of the hand performing the reference gesture respectively captured by the head mounted device and any one of the handles at the same moment is repeated until the gesture recognition result indicates that the gesture recognition is accurate.

2. The method according to claim 1, characterized in that Before acquiring hand images of a hand performing a reference gesture captured by the head mounted device and any of the handles at the same moment, the method further includes: At least one of an image or a descriptive voice of the reference gesture is output.

3. The method according to claim 1, characterized in that The method further comprises: When the head mounted device runs the target application and receives notification information indicating that there is a deviation in gesture recognition, the acquisition is triggered to simultaneously capture the hand image of the hand performing the reference gesture by the head mounted device and any of the handles.

4. The method according to claim 3, characterized in that The method further includes a step of obtaining the notification information, the step comprising: Detect whether the set trigger event occurs; If this happens, a notification message is generated indicating that there is a deviation in gesture recognition; The set triggering event includes at least one of: failure to recognize a gesture, an abnormal gesture being recognized, and receiving user indication information indicating deviation in gesture recognition.

5. The method according to claim 1, wherein After acquiring hand images of a hand performing a reference gesture captured by the head mounted device and any of the handles at the same moment, the method further includes: For any of the hand images, remove the background image in the hand image; Converting the hand image after removing the background image into a grayscale image; The step of determining a recognized gesture based on the hand image includes: A recognized gesture is determined according to the grayscale image.

6. A gesture recognition device, characterized in that: The device is applied to a head-mounted device, wherein the head-mounted device is equipped with at least one handle, and the device comprises: An acquisition module, configured to acquire hand images of a hand performing a reference gesture captured by the head mounted device and any of the handles at the same moment, where the reference gesture is an occlusion gesture; A first determining module, configured to determine a recognized gesture based on the hand image; a second determining module, configured to determine a gesture recognition result based on the recognized gesture and a reference gesture; a first output module, configured to output a prompt message for adjusting the handle position if the gesture recognition result indicates that the recognized gesture has a deviation; A repetition module is used to repeatedly obtain the hand images of the hand performing the reference gesture collected by the head mounted device and any of the handles at the same time, until the gesture recognition result indicates that the gesture recognition is accurate.

7. The device according to claim 6, characterized in that The device further comprises: The second output module is configured to output at least one of an image of the reference gesture or a description voice.

8. A head-mounted device, characterized in that: The head-mounted device comprises the apparatus according to claim 6 or 7; or The system comprises a memory and a processor, wherein the memory is used to store computer instructions, and the processor is used to call the computer instructions from the memory to execute the gesture recognition method according to any one of claims 1 to 5.

9. A head-mounted system, characterized in that: The head-mounted system includes the head-mounted device according to claim 8 and at least one handle.

10. A computer-readable storage medium, characterized in that A computer program is stored thereon, and when the computer program is executed by a processor, the gesture recognition method according to any one of claims 1 to 5 is implemented.

Images